WorksAgricultural and Biological Sciences
Optimization principles, revisited
Abstract
The brain solves a wide range of problems, and some of these solutions are very good indeed. It is an old idea that, in some cases, these solutions may be as good as possible given the relevant constraints. If this is true we can promote this notion of optimization to a principle from which aspects of function and mechanism can be derived, ideally with no free parameters. Optimization principles are powerful and have had many successes, but they remain controversial. We start with a reminder of a classic example—diffraction limited optics in insect eyes—and then a summary of why you should worry about optimization as a more general principle. Next we turn to recent progress on the theory of optimal motion estimation in fly vision, and finally summarize recent results on the landscape for optimization, arguing that (contrary to widely held intuitions) optimal performance need not require the underlying mechanisms to have finely tuned parameters. I hope to convince you that this is a moment to take optimization seriously as the basis for theories of neural coding and computation. This is based on a rather informal lecture at the summer school LASCON X in Sao Paulo, and I have tried to preserve that tone here; pointers are given to more detailed and technical discussions.
Introduction
In order to survive in the world organisms have to solve a wide range of problems. We know a great deal about the microscopic components that make these solutions possible, but much less about how the parts fit together into a functional whole. What we do know is that random collections of interacting parts typically are not functional.1 Thus the systems that we see today—the amino acid sequences of proteins, the networks of interacting genes in a developing embryo, the connections among neurons in a brain—have been selected from the set of possible mechanisms to achieve some level of functional performance. Might this selection be sufficiently strong that it has predictive power? Could natural systems be the ones that maximize performance subject to some sensible constraints? Even if organisms don’t maximize performance, could a theory of optimal systems be a useful idealization?
More than three centuries ago, Leibniz gave us tools for finding the maxima and minima of functions. He also articulated the idea that we live in the best of all possible worlds, taking maximization to an extreme [1]. Leibniz’ concerns were theological and not scientific, and his extraordinary optimism has been criticized and parodied, repeatedly. Perhaps the most famous parody is Voltaire’s Professor Pangloss, who gives voice to Leibniz’ claim in Candide, despite the accumulation of disasters around him [2]. Not long after Darwin, du Bois–Reymond suggested that evolution might lead to essentially perfect adaptation, echoing at least the spirit of Leibniz’ original claim [3]. A century later, Gould and Lewontin described this focus on adaptation as the “Panglossian paradigm,” and emphasized instead the roles of randomness, contingency, and constraint in shaping the organisms that we see today [4].
On a dark night, the arrival of single photons at our retina makes a measurable contribution to our perception.2 The reliability with which we can count these single photon events is limited by the dark noise in the rod photoreceptors, and this is possible only because the retinal circuitry implements filters and nonlinearities that are matched to the structure of signals and noise generated by these cells. In a precise sense, it would be very hard for the system to do better: photon counting in vision provides a concrete example of optimization in biological function. Is Panglossian to think that optimization may be a more general principle?
In keeping with Leibniz, many view the optimization of biological function as a philosophical (or perhaps even theological) idea. Some believe that optimization is obvious because evolution has had a billion years to get things right. Others believe that it’s impossible, not least because evolution in a dynamic environment is not guaranteed to reach an optimum. My experience is that, for many people, these general views largely are unaffected by evidence for or against optimization in specific cases.
More seriously, it is not clear what evolution has to say about the question of optimization. In order to conclude, for example, that evolution favors the emergence of single photon counting we need to understand both the benefits and the costs. The benefits seem obvious: a more sensitive visual system allows organisms to search for food and mates under a wider range of conditions. But the costs are substantial, since the receptors that count single photons are among the most metabolically active cells in our body. While we might be able to estimate this metabolic cost, we have no idea how to compare this with the benefits. Absent such a comparison, it is not clear how we could even start to understand what evolutionary theory predicts.
Optimization, used well, is a quantitative theory. Contrary to being a matter of belief, we can use the idea of optimization to discipline our thinking. If we want to say that something is the best of all the possibilities, we need to be clear what we mean by best and what we consider to be possible. If we suggest that “biological systems maximize ” is a principle, then we are making explicit our intuition that matters in the life of the organism, or more precisely that measures performance at a function essential for life. These intuitive claims could simply be wrong. To make progress we will need a mathematically precise definition of , and there may be many different formalizations of our intuition that we need to sort through.
If maximizing is going to serve as a principle, we have to understand why can’t be arbitrarily large: we need a theory that defines the relevant constraints and the resulting physical limits to performance. Ideally we should actually be able to measure and compare its value with this theoretical maximum; often this suggests new experiments or new ways of looking at existing data. Finally, maximizing should generate some definite predictions about the structure and dynamics of the system, predictions that can be tested in independent quantitative experiments.
The idea of deriving quantitative predictions from general principles is of course familiar in physics, but in the context of living systems it often feels that the best we can do is to build models with many parameters and fit them to the data. Optimization provides a way of cutting through this complexity, and in favorable cases results in quantitative predictions that have no free parameters [6]. Replacing parameters with principles, allowing us to derive rather than fit the behavior that we see, is a grand ambition. It would be wonderful if we succeeded, but it is a serious program precisely because it has the possibility of failure.
In one short lecture I could not possibly give a full survey of optimization in neural coding and computation. Instead I sketched two examples, one classical and one modern, with the latter including work in progress. Interleaved between these examples is some discussion of the reasons to be suspicious of optimality as a general principle, and then at the end I look at recent developments which respond directly to these objections. In writing these notes I have tried to keep the same informal tone of the original lecture, including a few personal notes.
A classical example
In the nineteenth century, exploration of vision and hearing was entwined with the emerging understanding of light and sound. Major figures of classical physics—Helmholtz, Maxwell, Ohm, and Rayleigh—all moved freely between subjects that for much of the twentieth century we might have distinguished as physics and neuroscience or psychology. Their search for a physicist’s understanding of perception provides, in many ways, a model for the modern version of biological physics as a branch of physics [7].
Not long after it was understood that diffraction sets a limit to the sharpness of images it was realized that this is relevant for our own eyes, and the limits would be even more severe in the compound eyes of insects with their many small lenses [8]. Fifty years later Balow realized that these limitations lead naturally to an optimization problem [9].
The physicist’s preference for simple models leads to jokes about spherical cows and horses [10]. Barlow considered the case of an insect with a spherical head (Figure 1A), which is a much better approximation.3 In the compound eye each receptor cell (roughly) looks out at the world through its own small lens, and in many cases these lenses form a regular lattice; not just the receptors but some layers of processing circuitry also follow this lattice structure. If lenses of size are arrayed on a head of radius , then each pixel has an angular size , and this limits the resolution with which the visual system can form an image. On the other hand, when lenses become small diffraction blurs the image so that each pixel sees a region of angular size , where is the wavelength of light.

Figure 1. Optics of the insect eye [5]. (A) A schematic, in the approximation of a spherical head with radius . Lenses of size subtend an angle . (B) Limits to image resolution from geometry () and diffraction (). Solid line is a schematic of the combination of limits. (C) The size of lenses in compound eyes as a function of head size, across many species of insect, redrawn from Barlow [9].
As shown in Figure 1B, the geometrical limit pushes for smaller lenses and hence more pixels. The diffraction limit, on the other hand, pushes for larger lenses. The compromise is at
where resolution will be optimized. Barlow went into the drawers of the natural history museum at Cambridge University to find a variety of insects with varying head sizes, and he verified that the pixel size really does scale with the square root of the head radius, as shown in Figure 1C. This same argument reappears a decade later in Feynman’s lectures on introductory physics, and he compares just with a single species of bee, noting that the lens size with accuracy [12].
This is a very rough theory, but illustrates many important features. First, at least one piece of the eye’s structure is being derived from a physical principle, with no free parameters, and it works reasonably well. Second, the way we test this idea is by looking across multiple organisms, exploiting rather than avoiding the diversity of organisms. Finally, this optimization principle carries the hints of its own limitations.
Maximizing angular resolution makes sense as a design principle for the eye only if other features are not limiting. As light levels go down, for example, it is advantageous to use larger pixels each of which captures more photons, trading angular resolution for intensity resolution. With photodetectors that have finite time resolution, acrobatic flight will lead to motion blur and making smaller pixels doesn’t help. These observations suggest that big slow insects flying in the noonday sun should reach the diffraction limit, while others should not, and this is correct in outline. There have been efforts to make these ideas more precise by asking about the ability of the eye to gather information rather than form an image [13, 14]. The fact that we have at least a rough theory for the “design” of the regular photoreceptor lattice in insect compound eyes raises a question about eyes more like our own, where the arrangement of photoreceptors in much less regular. We will return to this in § 5, below.
Reasons for doubt
There are many good reasons to be skeptical of optimization as a basis for theories of living systems. In some ways the strongest criticism was by Gould and Lewontin [4], who raised doubt about any claim that some feature or mechanism serves a particular function. Their poetic example was an architectural feature in the Cathedral of San Marco, in Venice, the “spandrels” that anchor the dome to the rounded arches below. These spandrels are covered in illustrations that fit beautifully into the biblical narrative of the whole space. Gould and Lewontin assert that an “adaptationist” would view this artistic unity as a cause for the construction of the spandrels, embodying the sentiment that everything we see in living organisms has a function. But this is backwards, since the artists took advantage of what was necessary to support the structure. Gould and Lewontin describe this as an “architectural constraint” and extrapolate to claim that biological mechanisms are similarly constrained by body plans and their development rather than being necessarily functional. Their paper has been cited 12,000 times and “spandrels” entered the biologists’ vocabulary as a pointer to the misattribution of function.
It seems worth pointing out that there no equations and almost no numbers in Gould and Lewontin’s discussion. There certainly is no calculation to support their cartoon of the adaptationist position that spandrels are somehow necessitated for the artistry; indeed it is difficult to imagine what such a calculation would look like. When we try to articulate optimization principles, as noted above, it is not enough to say that maximizing is important for the organism. We need to understand something about what limits this maximization and in some sense whether this is a hard problem. It is possible, for example, that random mechanisms lead to quite large values for , in which case maximizing is unlikely to have much predictive power. It is not enough to identify a problem that the organism is solving, we need to identify a problem that is hard for the organism to solve. For the spandrels, many different shapes would be effective in presenting the artist’s vision but only a limited number of shapes solve the hard problem of supporting the dome. Rather than being an architectural constraint, the spandrels are a good (perhaps even optimal) solution to a different and more challenging problem.
To give this discussion a more precise mathematical form, it is useful to think about the performance of the organism as a function of the parameters that describe the underlying mechanisms. These parameters will take different forms in different examples: the amino acid sequences of the relevant proteins and the DNA sequences of the enhancer regions that regulate their expression; the number of copies of each protein that a cell expresses, including for example the number of each type of ion channel in a neuron; the strengths of all the synapses among neurons in a network; and so on. These parameter spaces typically are very high dimensional—amino acid sequences are long, there are many different kinds of proteins, and many synapses.
It is challenging to build intuition for the behavior of functions that depend on very many parameters. One possibility is that complex systems with many degrees of freedom can be approximated as random. This idea goes back at least to Wigner’s discussion of energy levels in heavy nuclei, which provided a crucial stimulus for the development of random matrix theory [15]. Random pairwise interactions among binary variables defines the energy function of the mean–field spin glass, and the solution of this model provided a prototype for complex systems more generally [16]. The energy landscape in this model is rough, with exponentially many local optima organized in a hierarchy and separated by barriers that grow with the number of variables; reasonable local dynamics never really find the true optima, and this connects to “aging” and other slow phenomena seen experimentally in real glasses and spin glasses.
We can make a one dimensional schematic of a rough landscape, as in the left panel of Figure 2. Here we are plotting not the energy of a system but the performance at some task, so optima are maxima rather than minima. We imagine that evolution, learning, and adaptation typically proceed by making small changes in parameters, with guidance or selection from the resulting performance. In such a rough landscape finding true optima via such local moves obviously is hard, as with the dynamics of the mean–field spin glass. A related but more subtle point is that even if the system could find the optimum, small changes in parameters would have a large impact on performance. This is important because we have many examples in which the parameters of biological systems, from the copy numbers of proteins in single cells to the strengths of synapses between neurons, are highly variable [17, 18, 19]. If the landscape is rough then variability is direct evidence against the idea that parameters are tuned to optimal values.

Figure 2. Landscapes, the possibility of optimization, and significance of variability. At left, a schematic “rough” landscape in which the true maximum is hard to find and small variations in parameters around this maximum produce catastrophic loss of performance. At right, the opposite.
We don’t really know what the landscape of performance vs parameters looks like in real biological systems. It could be simple, as in the schematic in the right panel of Figure 2. This may be overly optimistic, but it’s not impossible. Obviously if this picture were correct then it would be easy for systems to find their optima, and they could tolerate significant parameter variations.
In summary, the argument of Gould and Lewontin should been seen as a caution not an objection. As noted at the outset, if we claim that “maximize X” is a principle we should be able to measure X and show that it is close to its optimum. Any claim that some underlying mechanisms are relevant to maximizing X must be supported by honest calculations. The feasibility of optimization and the implications of variability depend on the structure of the landscape, to which we return explicitly in § 5.
A modern example
Visual experience is constructed by the brain from image data collected by the retina. Artificial intelligence draws attention to complex percepts such as objects and faces, but simpler ones are central to survival and still pose open questions: arguably there is no case in which we understand quantitatively what algorithm the brain uses to synthesize a percept, how it is implemented in neural hardware, or why it is selected among many possible algorithms.
An accessible example is the visual perception of motion. For more than a century we have known that spatiotemporal patterns with particular correlation structure elicit the percept of motion even without moving objects [20], and for seventy years that the same structures drive insect behavior [21]; humans and flies even experience analogous illusions [22]. In the fly, a small group of neurons at the output of the visual brain responds to different components of motion [23, 24] and drives steering in walking and flying flies [25, 26]; the receptors act as nearly ideal photon counters over a wide dynamic range [27, 28]; and experiments on the intervening layers now culminate in a complete wiring diagram or “connectome” in fruit flies [29].
When I finished my PhD I didn’t know anything about fly vision. As a postdoc in Groningen, I had the tremendous luck to find myself in the office next door to Rob de Ruyter van Steveninck. He showed me how he could make long, stable recordings from motion sensitive neurons at the output of visual computation and also give a fairly complete analysis of the signal and noise properties of the receptors at the input.4 I didn’t know it then but the data rob could collect was on a scale that would not be matched in other systems for a decade. We saw an opportunity to use this system as a testing ground for a range of theoretical ideas, especially notions of optimality. This effort continues to the present.
Among the neurons at the output of the fly’s visual brain, H1 is sensitive to wide field motion, and if we show the fly a step displacement it responds with a brief volley of action potentials (Figure 3). These few extra spikes occur in a window of msec, consistent with the behavioral response times of the fly [30]. We can characterize the response to a step displacement by counting the spikes, by measuring their arrival times relative to the moment of the step, by measuring interspike intervals, or by discretizing time into small bins and thinking of spikes and silences as a binary string. Because there is some randomness, any of these measures comes from a probability distribution even if the visual stimulus is fixed. But this distribution shifts when we change the size of the step: large steps produce more spikes with shorter latency and smaller intervals. These shifts are large enough that we can reach reliability in distinguishing between steps of and using just one sample of the response, as schematized at right in Figure 3A [31, 32].

Figure 3. Precision of motion estimates encoded by the fly’s H1 neuron. At left, the response to a step displacement is described by the time until the first spike after a latency, the interval from the first spike to the second , … Alternatively we can count the spikes, or discretize time to turn the response into a binary word. In each case we can estimate the distribution of responses from many repetitions of the same steps. At right, an example with the single interspike interval . From the cumulative distributions in the top panel we can read off the probabilities of correct identification of each stimulus [32].
In this simple step experiment, the precision of motion encoding is . This is an order of magnitude smaller than the angular spacing between photoreceptors on the retina, which is comparable to width of the diffraction blur through the lenses of the compound eye. Analogous “hyperacuity” phenomena have been known in human vision for more than a century [33]. It seems impressive that the fly can detect motion with a precision better than the diffraction limit, but our colleagues who track small particles or even single molecules under the microscope do this routinely.5 Should we really be surprised by ? What is the physical limit?
Suppose that we look at a pattern of typical contrast and it moves by an angle . A single photodetector element will see a change in contrast of roughly , where is the angular scale of blurring due to diffraction. If we can measure for a time we will count an average number of photons , with the counting rate per detector, and shot noise in the photon arrival gives a fractional precision in intensity of . But fractional intensity is what we mean by contrast, so is the contrast noise in one photodetector. To get the signal-to-noise ratio, we should compare the signal and noise in each of the detectors, then add the squares if we assume (as for photon shot noise) that noise is independent in each detector while the signal is coherent:
The experiments in Figure 3 were done with displays that lead to and random spatial patterns that have a contrast when viewed through the lenses of the compound eye. With integration times – msec, the combination . H1 integrates over receptors, so we expect to reach at . This is very rough, but gets within 4× of what we see in the data. We can do a more realistic calculation using the measured noise properties of the photoreceptors, which deviate from ideal photon counters at higher frequencies, and looking at the evolution of SNR with time after the step and comparing with a more detailed analysis of discriminability in the neural responses; the result is that neural performance is within 2× of the optimum for the first msec of the response. Rather than asking about discrimination among discrete stimuli we can “decode” the sequence of spikes to reconstruct a continuously varying velocity signal, then compare the spectral density of the errors in reconstruction with the limits from photoreceptor noise and diffraction blur; again the neuron is within 2× of the optimum over a reasonable range of frequencies [34, 35].
It would be nice to make a tighter comparison of performance with the optimum. But we saw in those early data was enough to motivate a theory problem: What computation does the fly’s brain need to do in order to make the best possible estimate of motion? This problem of optimal estimation is much more general: we collect sense data and we are interested in estimating some feature of the world. The key point is that data and features are related only probabilistically, so we can define their joint distribution . The computation we want is a mapping which we can call
The goal is to get as close as possible to the right answer, and we’ll measure this by the mean square error
Although very schematic the notation is meant to remind us that the data typically live in a high dimensional space.
In principle we need to search through all possible functions to find the one that minimizes . This not only seems like a complex problem but also a different problem for every task. In fact we can solve the problem formally while staying completely general:
In words, the best estimator is the conditional mean. If we want to be a bit more careful we can think about in the outside world and receptor signals (e.g., photoreceptor voltages) that provide the primary representation of these data at the input to the brain.
For the problem of motion estimation in fly vision, the are the spatiotemporal patterns of light intensity falling on the retina or the resulting photoreceptor voltages , and the feature of interest in the angular velocity . While direct measurements on the photoreceptors provide a good characterization of the distribution , we know much less about the distribution of the input data . In early work we made reasonable guesses about the form of this distribution, and were able to make analytic progress in calculating the optimal estimator [36]. To get control over the calculations it was useful to think about different regimes in the overall signal–to–noise ratio of the system.
At low SNR we could write the optimal estimator as a function power series in the receptor cell voltages:
The kernel is dominated by having n and m be neighbors in the lattice of receptors. The idea of a series expansion for neural computation goes back at least to the work of Reichardt and Poggio on the analysis of fly visuomotor behavior [37]. reaching back even further, in the mid 1950s Hassenstein and Reichardt argued from experiments on beetle behavior that insects detect motion by computing correlations between neighboring pixels in the compound eye [21], and this correlator model continues to guide the community’s thinking seventy years later [38]. It is attractive to see how the correlator can emerge as one regime of the optimal estimate.
At intermediate SNR the higher order terms represented as in Eq (7) become important, the time scales inside the kernel become shorter, and the restriction that n and m be neighbors becomes tighter. This crosses over smoothly to the behavior at high SNR, where we can forget about the difference between receptor voltages and light intensities, allowing us to write the estimator directly as a mapping from visual inputs. Schematically we find
where is an approximation to the spatial gradient of the image computed from neighboring points on the photoreceptor lattice, denotes an average over all the receptors, and is a constant that depends on the distribution .
If I can be a little more schematic and not worry about the spatial averaging, then when spatial gradients are large enough the high SNR estimator in Eq (8) becomes
This “ratio of gradients” computation makes sense. Suppose we have a spatial pattern that moves rigidly at some velocity , and for simplicity we can think just in one dimension so that
So Eq (9) seems like the right answer. The correlator model, on the other hand, is weird; in particular if we double the image contrast we quadruple the estimated velocity. You might be willing to accept that flies are weird, but we were trying to derive the optimal estimator and in some regime we found the correlator. What’s going on? Why isn’t the ratio of gradients always the right answer?
Experimentalists know that there are two things you shouldn’t do with noisy data. You shouldn’t differentiate, because there usually is more noise than signal at high frequencies so that differentiating lowers the SNR. You shouldn’t divide, because there is a danger of diving by zero and more generally random errors in the denominator become systematic errors in the ratio. The ratio of gradients in Eq (9) commits both of these sins, one of them twice.
Optimal estimation always is a compromise between systematic and random errors. In the absence of noise the ratio of gradients is correct, with zero systematic error (at least in simple settings), but it is very sensitive to noise. This not about biology but about math. Spatial and temporal gradients are correlated in a way that depends on velocity, and if we can trust each gradient completely then we can tae the ratio, which includes all orders of correlations. But when we add noise the higher order correlations are destroyed first, and if we add enough noise all that remains are the second order correlations, precisely what is picked up in the Reichardt correlator model.
The idea that the (systematically wrong) Reichardt model can emerge from a theory of optimal estimation is very attractive. Can we take thus take optimization seriously as a theory of the computation that the fly is doing? The problem is that the correlator emerges as optimal only in some regime. Is the fly in this regime? The answer depends on the statistics of the inputs, , and in the initial work we were only guessing at this. Related ideas have been used to think about motion perception in human and primate vision [39, 40, 41, 42], part of a larger project of treating perception as Bayesian inference [43, 44]. All of this work holds out the promise of a principled theory, but if we don’t know the distribution of sensory inputs then we are back in the world of parameterized models.
My colleague Rob de Ruyter van Steveninck set himself the task of building a special purpose camera that captures a “fly’s eye view” of the world. He was joined by Shiva Sinha and Charlie Edelson, in two generations of this project [45, 46]. The current camera has 187 pixels in a hexagonal array that mimics the compound eye; the spectral sensitivity and optical blur also match those in the fly, and the system samples at msec resolution. Importantly the camera is bigger than a fly, so it collects orders of magnitude more light, providing an essentially noiseless view of the inputs to the visual system. The camera is instrumented with motion sensors so that one also has a recording of its angular velocity as we move the camera through a natural environment, e.g. on a walk through the woods.
There is much to be learned just from the problem of local motion estimation, where we focus on signals from a single pixel and its immediate neighbors. It is plausible that the only relevant information is contained in the spatial and temporal gradients computed from these signals and perhaps filtered in time. The data then give us many samples from which we can estimate the distribution ; we focus on the log of the intensity to remove the absolute brightness of the image, which shouldn’t matter for the motion estimate. Again we think in one dimension, corresponding to the horizontal or yaw coordinate . We will also take the raw data, which as noted are essentially noiseless compared with the fly. Figure 4 shows the optimal estimator, making Eq. (6) explicit:
Figure 4A gives a global view of the estimator. The form is remarkably similar to what we would expect from Eq. (8). At large values of the spatial gradient the contours of constant estimated velocity are straight lines, as with a ratio of gradients computation. But most of the data that we encounter on this walk in the woods is at smaller gradients, where the contours bend in way that corresponds to the correlator model. We can see this more explicitly by taking slices through the estimator (Figure 4B). For the bulk of the data the estimator is linear in both the spatial and temporal gradients, which is what happens if the estimator is proportional to their product. In the tails of the distribution of spatial gradients we see hints of the ratio of gradients.

Figure 4. Optimal local motion estimation in a natural environment, from Eq. (11). (A) Heat map of the estimator [46]. The black line is 70 deg/s in the ratio of gradient computation, and the white square encloses 90% of the data on a walk through the woods. (B) Slices through the optimal estimator at constant temporal (left: ) or spatial (right: ) derivatives. Blue points are optimal estimates, error bars are standard deviations across random quarters of an earlier data set [45]. Blue lines are the predictions of Eq. (9) and red lines show the distributions of derivatives along the slices (scaled for clarity).
The important conclusion is that the correlator computation is optimal under conditions that are relevant in the natural environment. The results give us a quantitative picture of how the estimator departs from the pure correlator predictions, which should be tested in detail against the responses of motion sensitive neurons.6 There is much more to be done:
We can add noise to the image intensities, simulating the signals in photoreceptors at lower light intensities, then recompute the optimal estimator. At lower light levels we see longer filtering times and an expansion of the dynamic range over which we see pure correlator behavior, both of which are in agreement with neural responses.
Perhaps surprisingly, the optimal estimate of yaw velocity depends on the orthogonal spatial gradient of intensity. This dependence is not sign specific, and can be interpreted by saying that orthogonal gradients are a source of noise, which again is consistent with neural responses [47].
We can divide the walk in the woods into epochs of high and low contrast, and search for adaptation to this longer term statistical structure.
Full, quantitative exploration of these ideas still is a work in progress. The results will provide a rich, parameter free theory of what the fly should compute and how this computation should adapt to the statistics of the environment. The availability of data on these statistics allows us to finally realize the promise of optimization as a complete theory.
The landscape for optimization
Biological systems seem to be highly variable. Even in invertebrate neural circuits, where cells can be named and numbered and hence identified across individuals, we see that the selection of ion channels in each cell membrane and the strengths of synapses between cells are highly variable [18, 19]. This seems like prima facie evidence against the idea that parameters have been tuned to optimize anything.
On the other hand, biological systems can exhibit striking order and reproducibility. An example is the insect eye that we discussion in § 2: not only are the lenses the right size to reach the diffraction limit, but in many species these lenses form a very regular array (Figure 5A). The receptor cells and processing circuitry mirror this regular arrangement, and Cajal referred to the insect optic lobes as a “neural crystal.” In contrast, vertebrate and especially mammalian neural circuitry is both disordered within individuals and variable across individuals, and this starts in the retina (Figure 5A).

Figure 5. Sampling the visual world [48]. (A) A segment of the fly’s eye showing the regular array of lenses, which is echoed by a regular array of receptors and processing circuitry [49]. (B) Photoreceptor array in a human retina, with cells colored by their spectral sensitivity [50]. (C) A regular lattice of receptors. (D) The same number of receptors scattered at random. (e) Information per cell as a function of the signal–to–noise ratio for the regular (blue) or random (red) arrangements. The range of SNR is chosen to match estimates for human cones in moderate daylight [51]. Inset shows the spectrum of eigenvalues of the Hessian, from Eqs (17), (18).
Some years ago, at the start of her PhD, Caroline Holmes emphasized to me the problem posed by this contrast between vertebrate and invertebrate brains. If there were no examples of ordered, reproducible circuitry we might conclude the developmental constraints make such precision impossible (perhaps in the spirit of Gould and Lewontin). But since order is possible, how can we and other vertebrates “get away” with such a mess? Are we solving different problems? Does the disorder leave more room for learning a wide variety of tasks? We were joined by a large group of collaborators to address parallel questions in multiple systems [48], but let’s start with the photoreceptor array.
We can imagine an array of receptor cells at positions . They respond to a contrast signal that is blurred by a lens and corrupted by added noise,
We’ll approximate the blurring function as Gaussian with a width that matches the mean spacing between receptors, so we have the highest resolution possible without significant aliasing, and take the noise as Gaussian, independent in each cell with variance . There are measurements of the correlation function for the contrast signal in natural environments [51],
and . If we approximate the fluctuations in as Gaussian then the information that the receptor array captures about the visual world is
where we emphasize that the information depends on the placement of the receptors. With our assumption of matched Gaussian blur we can choose the mean spacing between receptors as our unit of distance and then the only free combination of parameters is the signal-to-noise ratio in a single cell, . Results are shown in Figure 5E.
As far as we can tell the maximum information transmission always is achieved by a regular array of receptors. The surprise is that, over a wide range of conditions, a completely random arrangement of receptors is only a few percent worse. If we consider patterns such as random close packed, this difference is even smaller. In this case, then, we vertebrates pay hardly any penalty for being less ordered than our invertebrate cousins. Said another way, the set of possible arrangements that are within some tiny distance of the optimum performance is enormous.
It will be useful to give a local analysis of these deviations from optimality, and to use a slightly more general language. Let’s replace the positions of the receptors by some more abstract set of parameters , and instead of the information we’ll write for some general measure of functional performance. There is an optimal setting of the parameters where the performance takes its maximum value, and in the neighborhood of this optimum we have
where is the Hessian matrix. As usual we can define eigenvectors and eigenvalues,
The eigenvectors described combinations of parameters that have (to leading order) independence effects on the performance , and the eigenvalues describe the strength of these effects. If we think of the performance as a (negative) energy, then the eigenvalues are the stiffnesses of springs that hold the parameters close to their optimal values.
In the inset to Figure 5E we see the Hessian spectrum for the receptor array. The key result is that there are many very small eigenvalues, so there are directions in parameter space where we can move very far with very little impact on performance. If we look at the eigenvectors associated with these eigenvalues, they correspond roughly to density waves in the array, with longer wavelengths being connected to smaller eigenvalues, as with soft modes in condensed matter physics [52].
We have explored the dependence of performance on parameters in a wide of examples beyond the photoreceptor array [48]: readout of position information from transcription factor concentrations in a developing fly embryo, the electrical dynamics of the stomatogastric ganglion in crabs, filtering in recurrent linear neural networks, and image classification in deep (artificial) neural networks. More recently we have been able to make analytic progress on the landscape for optimization in simple models information transmission and coding [53]. In each case the measure of functional performance is different and the parameters have different meaning, but we always see the emergence of soft modes.
Since it is an example that is relatively easy to explore on your own, let’s take a quick look at the linear recurrent network, as in Figure 6A. We have neurons in which the activity of each neuron is described by a variable . This activity relaxes to its resting state on time scale , and measures the strength of the connection from neuron j to neuron i. A single input that drives all the neurons, each neuron i with a fixed weight , so the full dynamics are

Figure 6. A linear recurrent network with neurons [48]. (A) Network architecture. (B) The effective filter that relates the input to the output , as in Eq. (20). (C) Hessian eigenvalues of the cost function for a random target filter when perturbing the readout weights ; different values of in colors. (D) As in (C) when perturbing the internal weights .
The output of the network is some linear combination of activities,
for some filter that depends on all the parameters . We measure functional performance as the similarity of this filter to some target,
We can make a generic choice of the target filter by setting all the parameters to random values. Note that as we increase the variance of the synaptic strengths, , the network has access to a wider range of time scales and these will be reflected in . If we vary only the readout weights we find the Hessian eigenvalues in Figure 6B. Strikingly these eigenvalues are spread over many decades. At larger , and hence a wider range of time scales, a larger number of eigenvalues are significantly nonzero, and as the spectrum becomes almost perfectly uniform on a logarithmic scale.
When we perturb the synaptic strengths we find the eigenvalues Figure 6C. To understand this result we should think about the eigenvalues and eigenvectors of this matrix. If perturbations change only the eigenvectors then we can compensate by adjusting the readout weights. But then parameters of the synaptic matrix are truly redundant, so we see only significantly nonzero eigenvalues and these are widely distributed (Figure 6C). They span roughly five orders of magnitude, and except for a few they are distributed uniformly on a logarithmic axis,
with some slope .
The existence of soft modes in parameter space connects to the pioneering work of Sethna and colleagues on “sloppy models” [54]. Qualitatively they showed that when fitting multiparameter models to data, some combinations of parameters are determined precisely, and others are not [55, 56]. They argued that this is generic [57]; that the smoothness of models implies that sensitivity to parameters is distributed uniformly on a logarithmic scale [58]; and that this gives a path to model simplification [59].
What we have seen is very much in the spirit of sloppy models, but there are two important differences. First, we are trying to focus not on fitting data but on performance at tasks relevant for the organism’s survival. In place of finding the best fit we are searching for optimal function. Second, the parameters are not constructs of some model, but rather physical properties of the underlying mechanisms—the affinity of a transcription factor for its binding sites along the genome, the number copies of different proteins in a cell, the positions of cells in the retina, the strengths of synaptic connections between neurons. There is no sense in which we can simplify the system by projecting out directions in parameter space that have little effect on the performance, not least because these are combinations of parameters. If there are soft modes in the parameter space of a genetic network, for example, this does not mean, for example, that there is an equivalent network with fewer genes. The observation of a sloppy spectrum does mean that systems with widely varying parameters can be very close to optimal.
To make these idea precise let’s imagine a population of cells, networks, or organisms that choose their underlying parameters at random from a distribution . This distribution supports an average performance
In the absence of any other constraints we expect that the parameters will be as variable as possible while obeying the mean value in Eq (23). To make the distribution as variable as possible we maximize its entropy at fixed , and the result is a Boltzmann distribution
where the “temperature” must be set so that Eq (23) is satisfied and enforces the normalization of the distribution. We can also think of this Boltzmann distribution in several other ways: as a compromise between performance and the information cost of controlling parameters,7 as an approximation to the distributions that emerge in evolutionary dynamics [60, 61],8 as a generalization of maximal growth rates in flux balance analysis of bacterial metabolism [65, 66], or as a Bayesian prior on the space of parameters to be used in fitting data [67]. Here we focus on its implications.
If we can neglect higher order terms in Eq (17) then the distribution of parameters is like the equilibrium statistical mechanics of particles held to the origin by springs, and we can immediately calculate the mean “energy” and the entropy:
We can put these together to give
where is the geometric mean of the Hessian eigenvalues,
The “functional ensemble” that we have defined by is a statistical mechanics problem with degrees of freedom. We expect that the entropy will be extensive so it is reassuring that it is the entropy per parameter, , that enters Eq (27). We also expect that energy will be extensive, and correspondingly the deviation of from its maximum value also is proportional to . But the coefficient of proportionality is . If we have the “exactly sloppy” eigenvalue spectrum from Eq (22) then
This vanishes exponentially as the number of parameters becomes large, so that as . This means that when we have many parameters with a sloppy spectrum we can have an ensemble of organisms with finite entropy per parameter yet the average performance is abitrarily close to optimal. Variability and optimality coexist.
There is more to say, some of which is highlighted in [48]:
Our main result is more general than for the exactly sloppy spectrum. As an example, if we have the slower decay , then at large if .
This seems not to be a generic consequence of being in high dimensional parameter spaces. As an example, if we train a deep network to classify images with (unnatural) randomized labels then the Hessian spectrum fails to be sloppy.
If learning or evolution corresponds to something like Brownian motion on the performance landscape, then there is an entropic force that favors optima with have soft modes [68].
There is considerable interest in the Hessian spectrum as a more general diagnostic in machine learning problems, with hints that soft modes are connected to generalization ability. For a recent view see [69].
We have discussed continuous parameterizations but one can find analogs in discrete spaces. Thus many amino acid sequences map into the same protein structure [70, 71] and many different sequential decision strategies provide optimal solutions to foraging problems [72].
The summary is that, contrary to a widely held intuition, near–optimal performance does not require fine tuning of parameters. Importantly, this tolerance for parameter variation is not a retreat from optimality; rather it is a feature of the landscape for optimization itself. Optimization principles thus predict that there should be substantial variation in the underlying parameters.
Coda
More than a century after the first hints that we can count single photons, optimization principles continue to be productive. I am especially excited that ideas which started in the context of neural coding are so fruitful in thinking about the genetic networks that control early embryonic development in the fruit fly [6]; connections between theory and experiment in this system are especially detailed, quantitative, and parameter free.
The optimization problems relevant for biological systems are challenging. Often we solve much simpler problems—linear receptive fields, coding in populations of highly idealized neurons, ... . Again I think recent work on genetic networks provides an example of how more realistic calculations make closer contact with experiment [73]. In the fly motion estimation problem (§ 4), new data on the statistics of the input to the computation are driving a more detailed analysis with the goal of predicting quantitative features of the optimal estimator and its adaptation that can be compared with neural responses. Importantly the predictions are parameter free.
Exploration of the landscape for optimization is just starting. There are precursors in experiments on protein evolution [74, 75] and connections to machine learning. Indeed the success of modern machine learning is dramatic evidence that optimization in high dimensional parameter spaces works, despite many reasons why it should not. These successes should embolden us to revisit our ambitions for theories of natural systems.
Acknowledgments
The ideas in this lecture have been shaped by 40+ years of conversation, collaboration, and friendship with Rob de Ruyter van Steveninck. In addition to Rob, examples I discussed are based on work with M Bauer, CJ Edelson, CW Goddard, CM Holmes, K Krishnamurthy, SE Palmer, R Pang, M Potters, S Setayeshgar, DJ Schwab, SR Sinha, and L Susman. This work was supported in part by the US National Science Foundation and the Simons Foundation. Special thanks to Antonio Roque who has brought insight, energy, and warmth to the LASCON series, and to the students for making it all so much fun.
References
- [59]KN Quinn, MC Abbott, MK Transtrum, BB Machta, and JP Sethna, Information geometry for multiparameter models: New perspectives on the origin of simplicity. Reports on Progress in Physics 86, 035901 (2022).DOI
- [60]K Bod’ová, G Tkačík, and NH Barton, A general approximation for the dynamics of quantitative traits. Genetics 202, 1523–1548 (2016)DOI
- [61]K Bod’ová, E Szép, and NH Barton, Dynamic maximum entropy provides accurate approximation of structured population dynamics. PLoS Computational Biology 17, e1009661 (2021).DOI
- [62]PH von Hippel and OG Berg. On the specificity of DNA–protein interactions. Proceedings of the National Academy of Sciences (USA) 83, 1608–1612 (1986).DOI
- [63]OG Berg and PH von Hippel, Selection of DNA binding sites by regulatory proteins. Statistical-mechanical theory and application to operators and promoters. Journal of Molecular Biology 193, 723–750 (1988).pubmed.ncbi.nlm.nih.gov/3612791
- [64]OG Berg and PH von Hippel, Selection of DNA binding sites by regulatory proteins II. The binding specificity of cyclic AMP receptor protein to recognition sites. Journal of Molecular Biology 200, 709–723 (1988).
- [65]D De Martino, F Capuani, and A De Martino, Quantifying the entropic cost of cellular growth control. Physical Review E 96, 010401 (2017).DOI
- [66]D De Martino, AMC Andersson, T Bergmiller, CC Guet, and G Tkačík, Gašper Statistical mechanics for metabolic networks during steady state growth. Nature Communications 9, 2988 (2018).DOI
- [67]W Młynarski, M Hledík, TR Sokolowski, and G Tkačík, Statistical analysis and optimality of neural systems. Neuron 109, 1227–1241 (2021).DOI
- [68]L Di Carlo, C Goddard, and DJ Schwab, Entropic confinement and mode connectivity in overparameterized neural networks. arXiv:2512.06297 [cs.LG] (2025).arxiv.org/abs/2512.06297
- [69]DT Bernstein, C Goddard, L Di Carlo, V Ngampruetikorn, and DJ Schwab, Fisher information and the geometry of memorization in neural networks. arXiv:2610.16161 [cs.LG] (2026).arxiv.org/abs/2610.16161
- [70]H Li, R Helling, C Tang, and N Wingreen, Emergence of preferred structures in a simple model of protein folding. Science 273, 666–669 (1996).pubmed.ncbi.nlm.nih.gov/8662562
- [71]H Li, C Tang, and NS Wingreen, Are protein folds atypical? Proceedings of the National Academy of Sciences (USA) 95, 4987–4990 (1998).pubmed.ncbi.nlm.nih.gov/9560215
- [72]T Ma and AM Hermundstad, A vast space of compact strategies for effective decisions. Science Advances 10, eadj4064 (2024).DOI
- [73]TR Sokolowski, T Gregor, W Bialek, and G Tkačik, Deriving a genetic regulatory network from an optimization principle. Proceedings of the National Academy of Sciences (USA) 122, e2402925121 (2025).DOI
- [74]M Carneiro and DL Hartl, Adaptive landscapes and protein evolution. Proceedings of the National Academy of Sciences (USA) 107, 1747–1751 (2010).DOI
- [75]A Papkou, L Garcia–Pastor, JA Escudero, and A Wagner, A rugged yet easily navigable fitness landscape. Science 382, eadh3860 (2023).DOI