WorksMathematics
Neural ODEs - Theory and Application: From Residual Networks to Continuous-Depth Dynamics
Abstract
A Neural Ordinary Differential Equation (Neural ODE) uses a neural network to represent a system’s rate of change. This project investigates how well such a model can learn logistic growth from samples of a single solution. We first derive the exact solution using the Bernoulli substitution, then use it to generate 100 noise-free training points. A small neural network represents the growth rate, and a fourth-order Runge–Kutta solver produces the predicted population. In the reference run, the learned trajectory has a root-mean-square error of approximately 0.20 population units over the training points, but eventually exceeds the true carrying capacity. The same overshoot occurs in ten runs with different random initializations. Examining the learned growth rate shows that the models do not recover the equilibrium at the carrying capacity. For this experiment, agreement with the observed growth curve does not guarantee agreement with the system’s long-term behavior.
Introduction
Residual networks update a hidden state by adding a learned correction to its current value [2]. An update of the form
resembles a forward Euler step for a differential equation. This connection motivates Neural ODEs, which describe the evolution of a state through a neural network representing its derivative [1]:
The prediction at a later time is obtained by solving this initial value problem numerically. Rather than directly predicting a population from a time input, the network in this project learns a rule for how the population changes.
The main question is how accurately this approach reproduces a known growth model. Logistic growth is a useful example because the solution can be found analytically with techniques from an introductory differential equations course. We can therefore compare the learned trajectory with an exact solution, and also compare the learned rate of change with the differential equation that generated the data.
We first derive the logistic solution using the Bernoulli substitution. We then train a Neural ODE on samples from this solution and examine both its prediction error and its behavior near the carrying capacity. The data are synthetic; bacterial growth provides the motivation for the example, rather than a source of experimental measurements.
The logistic growth model
Model and parameters
Let denote population size in arbitrary population units, with time measured in hours. With unlimited resources, the equation describes exponential growth. The logistic model adds a factor that reduces the growth rate as the population approaches the carrying capacity :
Here is the intrinsic growth rate and is the population that the environment can sustain. When is small relative to , the growth is approximately exponential. As approaches , the growth rate approaches zero. In this example we use
These are scaled population units, so the initial value does not mean a fraction of an individual bacterium.
Solution using the Bernoulli substitution
A Bernoulli equation has the form
For , the substitution reduces the equation to a linear equation on an interval where the powers and the substitution are defined. Expanding Equation (2) gives
so , , and . For the positive population considered here, we set . Since , multiplying the equation by gives
This is a first-order linear equation with integrating factor . Multiplication by this factor gives
Integrating both sides, we obtain
Substituting back for gives
Writing and applying yields
For the chosen parameters, , so the exact solution is
For example, the predicted population after ten hours is
Behavior to compare with the learned model
Equation (2) has equilibrium solutions and . For , the derivative is positive; for , it is negative. Thus a population starting between and grows toward . Equation (3) also shows that this population remains strictly below at every finite time. In particular, .
The growth rate is greatest at . To see this, differentiate the right-hand side :
The maximum occurs at , where population units per hour. The trajectory reaches this value at hours. These properties let us assess more than whether two curves look similar.
Neural ODE construction and training
Data and network
The training data consist of 100 equally spaced samples from Equation (3) over :
The data contain no added noise. All 100 points are used for training and for the reported trajectory errors.
The network has one input, two hidden layers of 50 units each with hyperbolic tangent activations, and one output:
Its input is the current population, and its output is the predicted rate of change. Although Equation (1) allows explicit time dependence, this implementation does not use time as a network input. The learned equation is therefore autonomous:
The initial condition is supplied, but the network is not given , , or the logistic formula. Those quantities are used to generate the target data.
Numerical solution and optimization
We solve the learned equation using the classical fourth-order Runge–Kutta method (RK4). For this autonomous model, one step is
The step size is hours, matching the sampling interval. Each predicted trajectory requires 99 steps and 396 evaluations of the network.
Training minimizes the mean squared error
We use Adam [3] with a learning rate of 0.01 for 1,000 iterations. Gradients are computed by backpropagating through the RK4 steps. The experiment is repeated with random seeds 0 through 9, using the default PyTorch network initialization. Seed 0 serves as the reference run shown in Figure 1. The supplied implementation uses a single CPU thread; the saved results report PyTorch 2.10.0.
Figure 1. Reference run (seed 0). Left: the exact logistic solution and the learned trajectory over the training interval. The horizontal dotted line marks . Right: the true and learned growth rates. The shaded region shows the range of population values in the training data. The filled and open circles mark the true equilibrium at and the numerically located learned equilibrium near 10.35, respectively.
For evaluation, we report the root-mean-square error (RMSE) and maximum absolute error at the training points. A second integration uses 20 smaller RK4 steps per sampling interval to check sensitivity to the integration step and obtain predictions between samples. We also integrate to hours and search for sign changes of on a grid of 30,001 points over . Linear interpolation between neighboring grid points gives approximate equilibrium locations. This search detects sign-changing roots; it does not rule out every possible zero of the learned field.
Results
Reference run
In the reference run, the final mean squared error is approximately 0.0385, corresponding to an RMSE of 0.196 population units. The maximum absolute error is 0.334. Figure 1(a) shows that the learned trajectory follows the main shape of the exact solution, including its transition from rapid growth toward saturation.
At hours, the learned model predicts , compared with the exact value 5.999, a relative error of approximately 3.9%. Near the end of the interval, the difference is qualitative as well as numerical. The learned trajectory crosses at approximately 19.13 hours and reaches . The exact logistic trajectory never reaches in finite time.
The learned growth rate explains this overshoot. At the true carrying capacity, , so the learned equation still predicts growth. Its growth rate changes from positive to negative near , indicating a stable equilibrium above the true carrying capacity. The integration to approaches this value. At the other true equilibrium, , so the network also fails to recover the zero population equilibrium.
Reducing the integration step changes the reference trajectory at the sample times by at most population units. This comparison suggests that the observed overshoot is not caused by the coarse integration step. It is a numerical consistency check, rather than a rigorous error bound relative to the exact solution of the learned equation.
Different random initializations
Table 1 summarizes the ten runs. Their RMSE values range from 0.118 to 0.348, and every prediction at is within 4% of the exact value. However, every run crosses before , and every learned field is positive at both and .
| Seed | RMSE | Equilibrium found | ||
| 0 | 0.196 | 5.765 | 10.072 | 10.35 |
| 1 | 0.144 | 6.081 | 10.213 | 10.74 |
| 2 | 0.219 | 6.111 | 10.498 | 13.14 |
| 3 | 0.196 | 5.833 | 10.201 | 10.67 |
| 4 | 0.145 | 6.116 | 10.292 | 11.17 |
| 5 | 0.348 | 5.762 | 10.626 | None found |
| 6 | 0.118 | 6.057 | 10.153 | 10.56 |
| 7 | 0.149 | 6.059 | 10.298 | 11.23 |
| 8 | 0.171 | 6.149 | 10.313 | 11.20 |
| 9 | 0.149 | 6.135 | 10.239 | 10.91 |
Table 1. Results across random initializations. Population values and RMSE are in arbitrary population units. The exact values are and . The last column gives the stable equilibrium found by the grid search over .
Nine runs have a detected stable equilibrium between approximately 10.35 and 13.14. In seed 5, no sign-changing zero is detected in the searched interval, and the predicted population reaches approximately 32.1 at hours. This observation does not establish the absence of an equilibrium outside that interval.
The training loss also fluctuates. In the reference run, it is approximately 0.0184 at iteration 800 but 0.0385 after the final update. This behavior suggests that optimization settings deserve further investigation before attributing the size of the errors to the model architecture.
Discussion and limitations
The model follows the observed growth curve reasonably closely. However, it fails to recover the equilibrium at the true carrying capacity. This result shows that fitting a trajectory does not necessarily mean learning the underlying differential equation.
For the logistic equation, the equilibrium at acts as a boundary for solutions that start below it. The uniqueness theorem prevents these solutions from crossing the constant solution . The learned network is smooth, so its solutions also satisfy this uniqueness property. However, the learned growth rate at is positive: . Thus, is not an equilibrium of the learned equation. The learned model continues growing when it reaches the true carrying capacity.
The training populations range from 0.1 to approximately 9.955. Neither true equilibrium appears in the data. Training only penalizes differences between the predicted and observed populations. The network is not required to predict zero growth at 0 or . These features may help explain the misplaced equilibria.
The data also limit what we can conclude. They come from the exact logistic solution and contain no measurement noise. The results therefore do not show how the model would perform on real bacterial growth measurements. Training and evaluation also use the same sample points. The reported errors measure the fit to those points. They do not establish accuracy for new initial conditions.
Each run stops after 1,000 iterations, even though the loss still fluctuates. Longer training or a different learning rate might improve the learned equilibria. The equilibrium search only covers . It can also miss zeros where the growth rate does not change sign.
Future experiments could test new initial conditions. They could also compare training settings. Another approach is to enforce known equilibria. For example, writing guarantees zero growth at . This change would need to be tested for its effect on the overall fit.
Conclusion
This project connects the analytical solution of a Bernoulli equation with a computational model that learns a rate of change from data. The Neural ODE reproduces much of the logistic growth curve, but all ten runs in this experiment exceed the true carrying capacity. Examining the learned vector field reveals why: the growth rate remains positive at , so the learned equation has different equilibrium behavior. The comparison shows why both trajectory errors and qualitative properties are useful when evaluating a learned dynamical model.
The implementation and numerical summaries accompany this paper, with instructions for reproducing the reference run and the ten-seed study.
References
- [1]Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31, 2018. arXiv:1806.07366.arxiv.org/abs/1806.07366
- [2]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.arxiv.org/abs/1512.03385
- [3]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. arXiv:1412.6980.arxiv.org/abs/1412.6980
