Neural ODE’s - Theory and Application: From Residual Networks to Continuous-Depth Dynamics

Los Altos Hills, CA

Introduction

Residual Networks

In 2015, Residual Networks were introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.¹ ResNets allowed for the training neural networks using discrete layers, a process identical to Euler’s Method for solving ODE’s.

Then, in 2018, Chen et al. proposed Neural ODE’s, which took the limit of ResNet layers as step size approached zero. Essentially, this work parameterized the derivative of a hidden state in the neural network, further allowing for continuous-time modeling.²

We will apply ODE techniques, like Bernoulli forms to solve initial value problems, to the field of Machine Learning and Mathematical Biology.

Background Knowledge

The Research Question

Now that we know Neural ODE’s attempt to learn continuous dynamics, it would not be wrong to question the relevance of this. In comparison to ResNets, Neural Networks define the rate of change of output rather than just the output. This means that the Network is learning a vector field rather than a static map. Furthermore, when trying to make a prediction to calculate the next data point $y(t_1)$ from initial input $y(t_0)$, we must solve an Initial Value Problem given by

where $f$ is a Neural Network parameterized by $\theta$.¹

However, instead of solving this equation analytically, Neural Networks computationally solve the ODE using numerical integration methods. Specifically, they use methods like Euler’s Method or Runge Kutta to approximate integration.²

This brings us to the research question: How can we continuously predict successive data points with a Neural ODE, and how accurate will it be?

Chapter 2

Summary of ODE Techniques

Bernoulli Differential Equations

A key to answering the focus question lies in the Bernoulli form. To recap, a non-linear, first-order ODE can be classified as a Bernoulli ODE if it is given in the form

The reason that this is so special is that Bernoulli ODE’s can be solved relatively simply. The steps to solve Bernoulli ODE’s are as follows:

1. Divide by $y^n$:

2. Define variable substitution, let

then

3. Multiply the ODE:

4. Make substitution:

5. Solve for $z(x)$ using integrating factor $\mu(x) = e^{\int(1-nP(x))\,dx}$

6. Substitute back for $z = y^{1-n}$ and find implicit/explicit solutions

This is the algorithm for analytically solving Bernoulli ODE’s. However, in the example next, we will show how Neural ODE’s can forego this process using a training phase.

Example and Model Derivation

Bacteria Population

In Mathematical Biology, states of systems like population size are tracked over time. Here are some field concepts that will be helpful to know:

• Vector Field: A map that assigns some direction and magnitude at every point in the space

  • This is what the Neural ODE will learn

• Carrying Capacity ($K$): The maximum population size that an environment can sustain with limited resources

• Intrinsic Growth Rate ($r$): How fast the population grows given unlimited resources

In simple population models with unlimited resources, exponential growth occurs. However, since resources are finite in the real world, we must model population using a logistic model given by:

When population $y$ is small, the term $\left(1-\frac{y}{K}\right)\approx1$ and the growth of $y$ is exponential. However, as population $y$ approaches carrying capacity $K$, the term approaches $0$, thus stopping growth of $y$.

Problem Setup and Solution

Suppose we aim to find the population function $y(t)$ given the following parameters:

• $r = 0.5$ (growth rate per hour)

• $K = 10$ (carrying capacity of 10 bacteria units)

• $y_0 = 0.1$ (initial bacteria unit population)

Therefore, we can model the initial value problem as such

Compared to the Bernoulli form, we can identify $P(t)=-r$, $Q(t)=-\frac{r}{K}$, and $n=2$.

Let’s now follow the steps to solve Bernoulli ODE's:

Define variable substitution, let

then

Now multiply the ODE by $-y^{-2}$ and substitute:

This is now a linear, first-order ODE! To solve this linear equation, we calculate the Integrating Factor, $\mu(t)$:

Now multiply the entire linear equation by $e^{rt}$:

We recognize the left side as the derivative of the product $(e^{rt}z)$:

Now integrate both sides:

Solving for $z$:

Now we can substitute back for $y$:

Finally, let’s apply our initial conditions $y(0) = 0.1$, knowing that $K = 10$ and $r = 0.5$:

Solving for the arbitrary constant term $A = CK$:

Substituting $A$ back into the general solution gives the specific solution:

And finally, we have constructed our logistic population growth model with a particular solution. This solution and graph will be the benchmark that the Neural ODE will be trained against.

Chapter 4

Predictions

Neural ODE Construction and Limitations

Using our derived model $y(t) = \frac{10}{1+99e^{-0.5t}}$, we predict the population at time $t = 10$ hours:

After 10 hours, the bacteria population will be about 6.01 units. Note: The model assumes that $r$ and $K$ are constant, but in reality, carrying capacity often fluctuates with the environment. We also assumed that the population is large enough to be treated as continuous, which would not hold for small populations (like discrete individuals). Additionally, Neural ODE’s are often computationally expensive, so the limitations of using these networks include the capabilities of computer hardware.

With this in mind, let’s find the end result after training a Neural ODE with 1000 iterations. See Appendix § 10 for code files.

Conclusion

As you can see, at $t = 10$ hours, the Neural ODE path runs through $(10, 6)$, which we predicted with our analytical Bernoulli solution.

By treating the Neural ODE as a learned vector field of an ODE, we successfully modeled the Logistic Growth of a bacteria population with limited resources. Both the Neural ODE and the Bernoulli Differential Equation techniques yielded a sigmoid curve that stabilized once carrying capacity was hit.

An Invitation for Further Exploration

Suppose the carrying capacity $K$ was not constant, but a periodic function that changed with the seasons. Let

How would the solution to the Bernoulli Equation change, and would Neural ODE’s be a superior method to solving this new equation?

Code Files

# Generate "True" training data
true_model = LogisticGrowth(r=0.5, K=10.0)
# We compute true data using the analytical solution (Bernoulli result)
true_y = true_model.analytical_solution(y0, t)

# Now train with 1000 iterations
func = ODEFunc()
optimizer = optim.Adam(func.parameters(), lr=0.01)
n_iters = 1000

print("Training Neural ODE on Logistic Growth...")

for i in range(n_iters):
    optimizer.zero_grad()

    # Forward Pass: Integrate NN using RK4
    pred_y = rk4_solve(func, y0, t)

    # Compute Loss (MSE)
    loss = torch.mean((pred_y - true_y)**2)

    loss.backward()
    optimizer.step()

    if i % 100 == 0:
        print(f"Iter {i:04d} | Loss {loss.item():.6f}")

print("Training Complete!")

# Visualize the data
t_np = t.detach().numpy().flatten()
true_y_np = true_y.detach().numpy().flatten()
pred_y_np = pred_y.detach().numpy().flatten()

plt.figure(figsize=(10, 6))
plt.title("Neural ODE Learning Logistic Growth (Bernoulli Example)")

# Plot True Data (The Analytical Solution)
plt.plot(t_np, true_y_np, 'g-', label='Analytical Solution (Bernoulli)', linewidth=4, alpha=0.4)

# Plot Neural ODE Prediction
plt.plot(t_np, pred_y_np, 'k--', label='Neural ODE (Learned)', linewidth=2)

# Plot Carrying Capacity line
plt.axhline(y=10.0, color='r', linestyle=':', label='Carrying Capacity K=10')

plt.xlabel("Time")
plt.ylabel("Population")
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

# Verification: Predict outside training range (Extrapolation)
print("\nVerifying 'Carrying Capacity' behavior...")
last_val = pred_y_np[-1]
print(f"Final Population at t={t_max}: {last_val:.4f} (Should be close to K=10.0)")