betaFeedback

WorksComputer Science

Recurrent Modular Structure in Spiking Neural Network Reinforcement Learning

Abstract

Biological neural circuits are both modular and recurrent, with dense communities sparsely linked to one another. Spiking neural networks (SNNs) for reinforcement learning are usually feed-forward, and it is unclear how to obtain functional modules in a recurrent SNN or what they buy. In this work, we introduce a class of spiking networks, called Community Recurrent SNNs (CoRe SNNs): recurrent networks partitioned into communities with fixed roles. A controller community receives the stimulus, each action has its own community, and latent communities carry internal state. Spiking activity is then a walk between communities, and the action is the community the walk reaches. All learning is local and three-factor, and the recurrent communities are closer to cortical microcircuits than a feed-forward actor. We compare three regimes for obtaining the communities: dynamic modularity, in which a new learning rule we introduce, Map-STDP, encourages modular structure during learning; partial modularity, in which a stochastic block model (SBM) initialization enforces it partly; and full modularity, in which sealing enforces it fully. On CartPole, across three spiking neuron models, Map-STDP improves learning over the same network without it matches full modularity (sealed communities) on two of the three models, and makes the observed spike walk more compressible. Replacing the feed-forward actor of a TD-STDP actor-critic with recurrent action communities improves on the feed-forward network on CartPole and LunarLander, and matches it on Acrobot. How strongly to enforce modularity depends on the task: full modularity helps most on Acrobot, while on LunarLander the dynamic, partial and full regimes perform alike. Overall, these recurrent modular networks can match or even improve upon feed-forward approaches, with improvements in bio-plausibility and energy efficiency.

Figure 1. Concept overview. (a) A feed-forward network passes information layer by layer and is usually trained by backpropagation. Each action is a single output neuron. (b) A community recurrent SNN (CoRe SNN) organises neurons into communities with distinct roles instead of layers. Each action is an entire community, and information flows through recurrent connections within and between communities. (c–e) Three ways to obtain the communities: (c) dynamic modularity, where Map-STDP encourages modular structure during learning, (d) partial modularity, where a stochastic block model (SBM) initialization imposes it partially, and (e) full modularity, where sealing imposes it fully. (f) The full model with CoRe inserted on a TD-STDP actor-critic backbone. An interactive visualization of the CoRe SNN variants is at https://vikramramavarapu.org/CoRe.

Index Terms—spiking neural networks, reinforcement learning, spike-timing-dependent plasticity, three-factor learning, modular networks, map equation, community detection, neuromorphic computing

Introduction

The cortex is organized into columns [27], and its wiring forms densely connected communities that are sparsely linked to one another [28], [29]. Modularity lets a circuit divide its work among parts that can be specialized and reused. Spiking neural networks (SNNs) are attractive models of such circuits and, on neuromorphic hardware, an energy-efficient way to run them [2], [3]. Yet SNNs for reinforcement learning (RL) are usually feed-forward stacks of layers, trained by backpropagation or by local reward-modulated rules [39]. It is unclear how a recurrent SNN could acquire functional modules using only local plasticity, or what those modules would buy.

A guiding principle comes from compression. Hinton and van Camp argued that a good network is a simple one and trained networks by minimizing the description length of their weights [25]. Efficient-coding theories make a similar claim for neural activity: neural systems should transmit information at low cost [35], and the same accuracy–complexity trade-off underlies the free-energy principle [36]. As such, the map equation gives the description length of a random walk on a graph split into communities [24], and it is shortest when the communities contain the walk. Spiking activity is a walk over synapses, so a network whose activity is compressible has activity confined within modules and sparse traffic between them.

This view recalls the self-organizing map (SOM) [47], [48], the classic example of structure that emerges from local, unsupervised learning. Both approaches replace a global error signal with a local rule whose collective effect is organization. They differ in what is organized and why. A SOM arranges a layer of prototype units on a fixed lattice so that it preserves the topology of the input. Our network organizes its own recurrent connectivity into communities with distinct roles, using spikes, a reward-modulated rule, and the description length of its activity as the organizing objective. We see this as a modern descendant of the SOM idea, brought to recurrent SNNs, reinforcement learning, and information-theoretic compressibility.

We introduce Community Recurrent SNNs (CoRe SNNs), a class of recurrent networks partitioned into communities with fixed roles: a controller community that receives the stimulus, one community per action, and latent communities that carry internal state. We make four contributions.

  • CoRe SNNs. Spiking activity is a walk between communities, and the action is the community the walk reaches.

  • Map-STDP. A local, three-factor learning rule that encourages the communities, and lowers description length during learning.

  • Modularity regimes. A comparison of dynamic, partial and full modularity across four neuron engines.

  • A drop-in actor. CoRe SNNs replace the feed-forward actor of a TD-STDP actor–critic and are evaluated on CartPole, Acrobot and LunarLander, together with a sweep of cross-module connectivity.

Background

Spiking Neural Networks

SNNs communicate through discrete spike events, enabling sparse, temporally-coded computation that maps naturally to neuromorphic hardware [1]. This makes them a closer model of cortical computation than conventional artificial neural networks. Because computation is event-driven, a neuron consumes energy mainly when it spikes or receives a spike, and can therefore run at a fraction of the energy of dense matrix multiplication on digital neuromorphic chips [2], [3] and on analog memristive crossbars. In a crossbar, each synapse is a conductance, and multiply–accumulate happens in place [4]. These savings depend on activity staying sparse [5]. Training SNNs is difficult because the spike function is non-differentiable. Surrogate-gradient backpropagation through time (BPTT) [6] works well but is non-local, needing weight transport and the full activity history, which have no known biological counterpart [7] and are hard to implement with the local plasticity that neuromorphic hardware supports, so such networks are typically trained offline and then deployed [3], [6].

Spike-timing-dependent plasticity (STDP) is a local alternative to BPTT. A synapse is potentiated when a presynaptic spike shortly precedes a postsynaptic spike and depressed when the order is reversed [8], [9]. The update depends only on quantities available at the synapse, and the pairing of pulses across a device maps directly onto memristive hardware [4]. STDP alone is unsupervised, however. STDP alone is unsupervised: it strengthens whatever correlations the input contains. In networks that respect Dale’s law, where each neuron is either excitatory or inhibitory, it can still learn useful features, for example digit prototypes shaped by lateral inhibition [10]. It cannot, however, tell behaviorally useful correlations from irrelevant ones, so on its own it cannot learn a task defined by reward.

Three-factor rules add a global modulatory signal to the two local factors of STDP [30], [31]. Rewards arrive seconds after the spikes that caused them, far outside the tens-of-milliseconds STDP window. Each synapse therefore keeps an eligibility trace eije_{ij}. The trace accumulates pre–post coincidences and decays with a time constant matched to the delay of the outcome. A broadcast third factor M(t)M(t) then turns the trace into a weight change:

e˙ij=−eijτe+STDP⁡ij(t),Δwij∝M(t)eij(t).\dot{e}_{ij} = -\frac{e_{ij}}{\tau_e} + \operatorname{STDP}_{ij}(t), \qquad\Delta w_{ij} \propto M(t)e_{ij}(t).

Chung and Kozma [39] introduced a three-factor rule in which the temporal-difference (TD) reward prediction error δ=r+γV(s′)−V(s)\delta= r + \gamma V(s') - V(s) [32] is the third factor. In a spiking actor–critic network, this TD-modulated STDP was applied to standard control tasks such as CartPole and LunarLander.

Neuron Models

We use three spiking neuron models, which we call engines. In every engine, a spike of presynaptic neuron jj gives postsynaptic neuron ii the output-normalized kick

bij=sgn⁡jw0Wijdj,dj=∑kWkj.(1)b_{ij}=\operatorname{sgn}_j w_0\frac{W_{ij}}{d_j},\qquad d_j=\sum_k W_{kj}. \tag*{(1)}

1) Branching Neuron Model: The branching model is the standard neuron model of the SuperNeuroMAT simulator [11] and follows the branching-process description of cortical activity [12]. The membrane forgets after every step, so one step is one hop of the walk:

ui(t+1)=∑jbijxj(t)+2θμi(t),u_i(t+1)=\sum_j b_{ij}x_j(t)+2\theta\mu_i(t),
xi(t+1)∼Bernoulli⁡(clip⁡(ui(t+1)2θ,0,1)),(2)x_i(t+1)\sim\operatorname{Bernoulli}\left(\operatorname{clip}\left(\frac{u_i(t+1)}{2\theta},0,1\right)\right), \tag*{(2)}

where xi(t)∈{0,1}x_i(t)\in\{0,1\} is the spike indicator, θ\theta the threshold and μi\mu_i the external (stimulus) drive.

2) Leaky Integrate-and-Fire (LIF): The LIF neuron [13] is driven by background noise as in Brunel [14]:

τmdVidt=−(Vi−EL)+μi(t)+σV2τm ξi(t)+τm∑jbij∑fδ(t−tjf),(3)\tau_m\frac{dV_i}{dt}=-(V_i-E_L)+\mu_i(t)+\sigma V\sqrt{2\tau_m}\,\xi_i(t) +\tau_m\sum_j b_{ij}\sum_f\delta\left(t-t_j^f\right), \tag*{(3)}

with a spike and reset to VrV_r when ViV_i reaches VthV_{th}. Here ξi\xi_i is unit white noise standing for background input, and tjft_j^f are the spike times of jj.

3) Quadratic Integrate-and-Fire (QIF): The QIF neuron [15] is written in the normalization of Montbrio et al. [16]:

τdVidt=Vi2+ηi+Iiext(t)+στ ξi(t)+τ∑jbij∑fδ(t−tjf),(4)\tau\frac{dV_i}{dt}=V_i^2+\eta_i+I_i^{\mathrm{ext}}(t)+\sigma\sqrt{\tau}\,\xi_i(t) +\tau\sum_j b_{ij}\sum_f\delta\left(t-t_j^f\right), \tag*{(4)}

with a spike at Vi=+VpeakV_i=+V_{\mathrm{peak}} and reset to −Vpeak-V_{\mathrm{peak}}. Here ηi\eta_i is the excitability.

Random Walks, Information Theory, and Communities

A random walk on a weighted directed graph is a process in which a walker at node jj moves to node ii with probability proportional to the weight of the edge j→ij\to i. It is a Markov chain with transition probabilities Tij=Wij/∑kWkjT_{ij}=W_{ij}/\sum_k W_{kj}, and over time the fraction of steps spent at each node converges to a stationary distribution π\pi that measures how much flow each node carries. This is the idea behind PageRank [17].

In a graph with communities, a walker that has entered a densely connected group tends to stay there for many steps before crossing a sparse link, so the walk’s behavior reveals the community structure. In a spiking network, a natural walk follows spikes forward from each neuron to the neurons it excites.

Info-theoretic entropy can be measured by the description length of a random walk, following the core principle of information theory: the more predictable a process, the shorter its description. The map equation [19] measures the description length of a random walk on a graph partitioned into communities:

L(M)=q↷H(Q)+∑i=1mp↻iH(Pi).(5)L(M)=q_{\curvearrowright}H(Q)+\sum_{i=1}^{m}p_{\circlearrowright}^{i}H(P^{i}). \tag*{(5)}

Each step of the walk is given a binary codeword, shorter for more frequent steps, and the entropy of a codebook is its average codeword length in bits. The map equation uses a two-level code. An index codebook QQ names the module entered at each exit, which happens with probability q↷q_{\curvearrowright} per step. Each module ii has its own codebook PiP^i for naming the nodes inside it, used at rate p↻ip_{\circlearrowright}^i. Leaving a community costs extra bits, so the map equation is shorter for partitions that capture the graph’s community structure. The partition that minimizes L(M)L(M) is the one that best compresses the walk and reveals the communities, hence making the map equation the objective function of the community detection algorithm, Infomap [24]. The map equation has been used to detect communities in networks from social [19], biological [20], and to study the modularity of cortical networks [29].

A basic model that generates networks with community structure is the stochastic block model (SBM) [21]. In an SBM, each node belongs to one of KK communities, and the probability of an edge between two nodes depends on a K×KK\times K matrix of connection probabilities. The SBM can generate networks with strong communities, weak communities, or no communities at all, depending on the choice of the probability matrix.

Other Graph-Theoretic Approaches to SNNs

Understanding how the structure of a network shapes its dynamics in the brain has been a long standing question in the neuroscience community [49]. For threshold-linear rate networks it has a concrete answer. A CTLN takes a directed graph, assigns the weights from it, and its fixed point supports, stable states and dynamic attractors are then determined by the graph [50, 51, 52, 53]. Brief external kicks switch the network from one attractor to the next, and this has been enough to assemble counters, sequences and gait controllers by hand [54, 55]. Our SNN work sits next to this. We ask the same thing of a spiking network, and in both settings the answer takes the same shape, a collection of attractors pinned down by connectivity plus a set of input-triggered jumps between them, which is also how a finite state machine lives inside a recurrent network [61].

Part of the CTLN picture has been shown to survive the move to spikes, but only in a restricted setting. In clustered E-I networks of Hawkes neurons the excitatory rates obey a CTLN in the mean-field limit, and the CTLN fixed points become metastable states of the finite network [59]. CTLNs also come with a modular story, where the attractors of a graph assembled from smaller pieces are inherited from the pieces [56, 57], though no such result yet exists for spiking networks. Whether either property carries over to the SNNs we study is open. If it does, it would suggest a use for these rules in learning. A reward machine turns a history-dependent RL task into one with a finite set of states [60], and if those states could be planted as the attractors of a glued graph, a spiking network started from that graph would have the states in place and would only need to learn the jumps [62]. Some known initialisations of structure are detailed below.

Columnar organization has also been used for supervised learning in spiking networks. CoLaNET [22] gives each target class its own column, built from several layers of neurons and sub-class microcolumns, and trains it with local anti-Hebbian and dopamine-modulated plasticity. It was evaluated on classifying MNIST images [23] and Pong states by their proximity to reward, a task derived from model-based reinforcement learning. Its signal path is feed-forward, with only local winner-take-all competition and gating inside each column. The columns are fixed by design as class-specific classifiers, and the network evaluates states without selecting actions. In contrast, our work learns recurrent modular structure with Map-STDP and studies the effect of modularity on reinforcement learning.

Map-STDP Learning Rule

Community detection solves a forward problem: given a graph, find the partition into communities that minimizes the description length of a random walk on it, the map equation [24]. Map-STDP solves the inverse problem. The partition is fixed in advance (one controller module, one module per action, and a few latent modules), and plasticity shapes the synapses until that partition is the minimum-description-length one. This section states the rule and what each part does. The derivation from the map equation is in Appendix B.

Ingredients

Each synapse from neuron jj to neuron ii has a magnitude Wij≥0W_{ij} \ge0 and the sign of its presynaptic type, sgn⁡j=±1\operatorname{sgn}_j = \pm1. Each neuron belongs to one module m(j)m(j): the controller C\mathcal{C}, an action module in A\mathcal{A}, or a latent module in L\mathcal{L}.

  • The walk. A step of the walk from jj to ii is a spike of ii within a short window after a spike of jj. Its flow JijJ_{ij}, coarse-grained to modules, is a small Markov chain over the network’s own states. The policy is the flow into the action modules, P(a∣of)∝pa(f)βP(a \mid o_f) \propto p_a(f)^\beta, where ofo_f is the stimulus at frame ff, pa(f)p_a(f) is the flow into action module aa in that frame, and β\beta is a temperature parameter.

  • Description length. D(W)=Ef[L(Jf)]D(W) = E_f[L(J_f)] is the code length of the walk averaged over frames. Leaving module mm costs Mm∗=∂L/∂qm≥0M_m^* = \partial L/\partial q_m \ge0 bits per unit of exit flow qmq_m. It is computed from KK module-level statistics and broadcast to module mm: it is the rule’s neuromodulator.

  • Signed counter. cijc_{ij} is the change in ii’s spikes, within the window, caused by jj’s kicks alone. It is the local pre-synaptic activity times post-synaptic sensitivity, and carries sgn⁡j\operatorname{sgn}_j.

The rule

The per-frame update of each synapse is

ΔWij=ηδfeij−λMm(j)∗([Ii≠m(j)]−eˉjobs)cij−ϵh(dj−1)πjWijdj.(6)\Delta W_{ij} = \eta\delta_f e_{ij} - \lambda M_{m(j)}^* \left(\left[\mathbb{I}_i \ne m(j)\right] - \bar{e}_j^{\mathrm{obs}}\right)c_{ij} - \epsilon_h(d_j - 1)\frac{\pi_j W_{ij}}{d_j}. \tag*{(6)}

The rule has three terms:

  • Task. δf\delta_f is a global TD error and eije_{ij} an action-gated eligibility trace built from cijc_{ij} (Eq. (24)).

  • Structure (the map term). eˉjobs\bar{e}_j^{\mathrm{obs}} is the share of jj’s observed transitions that leave its module. Transitions that leave the module are pushed down when the module leaks, and those that stay are pushed up, scaled by Mm(j)∗M_{m(j)}^*.

  • Homeostasis. djd_j is jj’s total output weight, held near 11. It rescales a column uniformly, so it does not change the walk.

λ\lambda sets the strength of the map term. In partial and full modularity, λ=0\lambda= 0 and the map term is not applied. In dynamic modularity, λ>0\lambda> 0 and the map term is applied. Inhibitory synapses reverse the signs of the structural term, through sgn⁡j\operatorname{sgn}_j in cijc_{ij}. On average the structural term is gradient descent on DD, up to a rescaling that cannot change DD (Appendix B-§ 16). The controller’s synapses onto the action and latent modules carry the policy, so they are exempt from the map term (Appendix B-§ 16.1).

Three regimes of modularity

The experiments compare three regimes:

  • partial modularity: λ=0\lambda= 0 on a network initialized with SBM connectivity (pcross=0.05p_{\mathrm{cross}} = 0.05), so communities are only partly enforced and the task term alone shapes the structure;

  • full modularity: λ=0\lambda= 0 on a network built without cross-module synapses (pcross=0p_{\mathrm{cross}} = 0, scaled);

  • dynamic modularity: the complete rule (6), which encourages the communities during learning.

Model Architecture

Standalone Map-STDP Agent

The controller has no recurrent synapses of its own. In each environment frame the network is simulated on the current weights:

  • 100 steps for the branching engine;

  • 100 ms for the LIF and QIF engines;

  • 20 hops of power iteration for the flow engine, which has no spikes.

The spike raster gives the module flow (spike shares), the pair counts and the signed counters. Each spiking agent runs 8 parallel heads on shared weights and pools their statistics.

TD-STDP Actor-Critic Backbone

The backbone is the TD-STDP actor–critic of Chung and Kozma. Its neurons are noisy LIF (τm=20\tau_m = 20 ms) on a 1 ms grid, and each environment step is a 20 ms window. Population sizes follow the authors’ configurations on CartPole, and in Acrobot the critic has 40 excitatory neurons and the actor has 20 per action; on LunarLander each excitatory group is paired with an inhibitory group of equal size (128 critic, 32 per action). CoRe replaces the feed-forward actor with recurrent communities, one per action, among the actor’s excitatory neurons. Actions are read out by a race between the action communities’ excitatory spike counts instead of the original softmax.

SBM Initialization

We initialize the network with an SBM with K=∣C∣+∣A∣+∣L∣K = |\mathcal{C}|+|\mathcal{A}|+|\mathcal{L}| communities. The intra-community connection probability, pinp_{\mathrm{in}} is consistently set to 0.5. In the sealed regime, the inter-community connection probability, pcross=0p_{\mathrm{cross}} = 0 with all-to-all connections only between the controller and the other communities. In most experiments, the partial regime has pcross=0.05p_{\mathrm{cross}} = 0.05, but in the inter-module connectivity study, we vary pcrossp_{\mathrm{cross}} from 0 (full modularity) to 0.5 (no communities).

Implementation Details

Models are implemented in NumPy, with Numba kernels for the LIF and QIF engines. The branching engine uses SuperNeuroMAT’s [11] branching mode (infinite leak, uniform threshold noise), so each simulation step is one hop of the walk. Environments come from Gymnasium [26].

Training

a) Standalone agent.: Agents are trained on CartPole for 1500 episodes, with 20 seeds per condition. Each condition is one engine (flow, branching, LIF, QIF) in one of three regimes: dynamic (λ=0.05\lambda= 0.05), partial (λ=0\lambda= 0), or full (sealed: λ=0\lambda= 0, no cross-module synapses). Map-STDP and the task update are applied once per frame. All conditions use TD-error normalization and an adaptive readout sharpness. In the spiking engines, λ\lambda and the task rate are rescaled to match the flow engine’s update size at initialization.

b) Backbone.: The TD-STDP and Map-STDP updates are applied once per 20 ms window, without Adam. Input scales are calibrated so that every arm starts at the same 40 Hz rate. The arms are the original feed-forward network, the feed-forward network with the race readout, and CoRe SNNs in the partial, dynamic and full regimes. Each arm has 20 seeds. Per task:

  • CartPole: 400 episodes, one environment copy, race bound 16.

  • Acrobot: 500 episodes, one environment copy, race bound 16, one action every 40 ms, plus a firing-rate cost and floor on the actor.

  • LunarLander: 10,000 episodes pooled over 16 environment copies on shared weights, race bound 26, one action every 40 ms, plus the same rate cost and floor.

Results

Minimal spiking policy network

1) Results on CartPole: Figure 2 summarizes the comparison. In panel (a) of the figure, the trained networks of one seed are shown for each regime of the flow engine with a central controller, two latent modules, and two action modules. In the dynamic and full regimes, the modules are clearly apparent, indicating that qualitatively, on this engine, the map term is effective in shaping the network into a modular structure. In the partial regime, the modules are less distinct, indicating that the SBM initialization alone leaves room for inter-module connectivity. In panel (b), the mean return over training is shown for each engine and regime. The dynamic and full regimes outperform the partial regime across all engines, however overall, flow outperforms the other engines, albeit with a larger spread. LIF comes next, followed by QIF and branching. This may be due to the fact that in the base CoRe architecture, the random walk entropy formulation is most closely aligned with the flow engine, and spiking engines approximate walk behavior. In panel (c), the effective synaptic operations per decision (spikes ×\times outgoing synapses above the weight floor, averaged over 50 held-out stimuli with W frozen) is plotted against LobsL_{\mathrm{obs}}, the held-out description length of the observed walk. As there is a direct exponential correlation, this may hint at the fact that the more compressible the walk, the fewer synaptic operations are needed to make a decision, and therefore the more energy efficient the network is. In panel (d), LobsL_{\mathrm{obs}} is shown for each engine and regime at episode 1500. The dynamic and full regimes have lower LobsL_{\mathrm{obs}} than the partial arm, indicating that the modularity is effective in shaping the network into a more compressible structure. Compressibility largely follows from modularity itself: the full regime, which has no map term, has a lower LobsL_{\mathrm{obs}} than the partial regime on all four engines, and on flow and branching it is within 0.2 bits of the dynamic regime (Table II). On QIF and LIF the map term lowers LobsL_{\mathrm{obs}} further. Overall, these results suggest that the map term is effective in shaping the network into a modular structure that is more compressible and energy efficient, and that this leads to improved performance on the CartPole task.

Figure 2. (a) Trained CartPole networks on the flow engine, one seed (2000), from the same initial network. C is the controller, L1–L2 the latent modules and A1–A2 the action modules. Edges of at least 2% of the maximum weight are drawn, with width and opacity proportional to weight. Below each network: the share of weight within modules and across modules (excluding controller routing), and LWL_W, the map equation (bits) of W’s own walk. (b) AUC, the mean return over 1500 training episodes, for each engine and regime; 20 seeds per condition. (c) Effective synaptic operations per decision (spikes × outgoing synapses above the weight floor, averaged over 50 held-out stimuli with W frozen) against LobsL_{\mathrm{obs}}, for the final networks of the three spiking engines. Color: engine; shape: regime. Dashed lines are least-squares fits of log⁡10\log_{10} effective SOPs on LobsL_{\mathrm{obs}}; ρ\rho is Spearman’s. Post-hoc, exploratory analysis. (d) LobsL_{\mathrm{obs}}, the held-out description length (bits) of the engine’s observed walk, at episode 1500. In (b) and (d), boxes show the median and interquartile range, whiskers the full range, and dots individual runs.

EngineRegimeAUCReturnFrozenLobsL_{\mathrm{obs}} (bits)SolvedpAUCp_{\mathrm{AUC}}pLobsp_{L_{\mathrm{obs}}}
FlowDynamic356.9±96.7356.9 \pm96.7442.4±103.5442.4 \pm103.5463.9±94.3463.9 \pm94.34.54±0.014.54 \pm0.0115/20––
Partial335.9±64.9335.9 \pm64.9431.6±105.6431.6 \pm105.6412.8±122.8412.8 \pm122.86.00±0.146.00 \pm0.1420/200.17<10−6< 10^{-6}
Full388.3±68.8388.3 \pm68.8475.9±80.3475.9 \pm80.3481.4±80.1481.4 \pm80.14.34±0.014.34 \pm0.0120/201.0<10−6< 10^{-6}
BranchingDynamic204.1±14.1204.1 \pm14.1213.2±50.4213.2 \pm50.4198.9±90.2198.9 \pm90.26.69±0.036.69 \pm0.030/20––
Partial116.8±37.5116.8 \pm37.5128.7±68.1128.7 \pm68.1126.5±108.8126.5 \pm108.87.06±0.147.06 \pm0.140/20<10−6< 10^{-6}<10−6< 10^{-6}
Full201.4±39.0201.4 \pm39.0176.2±67.9176.2 \pm67.9177.5±78.3177.5 \pm78.36.75±0.046.75 \pm0.043/201.05.9×10−55.9 \times10^{-5}
QIFDynamic168.3±18.8168.3 \pm18.8174.3±39.7174.3 \pm39.7151.6±43.6151.6 \pm43.66.42±0.116.42 \pm0.110/20––
Partial103.4±17.1103.4 \pm17.1124.5±91.1124.5 \pm91.197.5±82.897.5 \pm82.87.05±0.057.05 \pm0.050/20<10−6< 10^{-6}<10−6< 10^{-6}
Full196.0±24.3196.0 \pm24.3198.0±43.2198.0 \pm43.2189.4±30.7189.4 \pm30.76.74±0.066.74 \pm0.060/200.0023<10−6< 10^{-6}
LIFDynamic276.2±13.5276.2 \pm13.5321.6±80.8321.6 \pm80.8310.2±94.0310.2 \pm94.07.44±0.027.44 \pm0.020/20––
Partial194.0±7.8194.0 \pm7.8208.4±40.4208.4 \pm40.4199.7±54.1199.7 \pm54.17.74±0.017.74 \pm0.010/20<10−6< 10^{-6}<10−6< 10^{-6}
Full270.8±12.7270.8 \pm12.7306.5±72.0306.5 \pm72.0294.6±80.6294.6 \pm80.67.66±0.017.66 \pm0.010/200.6<10−6< 10^{-6}

Table II. CartPole, model v4 with the observed-walk rule: dynamic (λ=0.05\lambda= 0.05), partial (λ=0\lambda= 0) and full (sealed: no cross-module synapses, λ=0\lambda= 0) modularity, on four engines. Mean ±\pm S.D. over 20 seeds (2000–2019), 1500 episodes per run. AUC: mean return over all episodes. Return: mean of the last 100 episodes. Frozen: mean return of 20 episodes with WW frozen at the end. LobsL_{\mathrm{obs}}}: held-out description length (bits) of the engine’s observed walk at episode 1500. Solved: runs whose trailing-100 mean reaches 475. pp: two-sided Mann–Whitney, Holm-adjusted over the four engines, of each arm against the MAP-term-on arm of the same engine.

2) Latent Ablation: To find out what the latent modules contribute, we ablated them in the final CartPole models (episode 1500) of every engine and regime (240 models in all) and measured the change in return with plasticity off. The design was pre-registered. Each network has a 36-neuron controller CC, two latent modules L=L1∪L2L = L_1 \cup L_2 and two 16-neuron action modules AA. Because every engine normalizes outgoing weights, Tij=Wij/djT_{ij} = W_{ij}/d_j with dj=∑kWkjd_j = \sum_k W_{kj}, a cut either loses the drive the cut synapses carried or reroutes it, so we test both cases. A lesion cuts the latents’ outgoing synapses and leaves every other neuron’s djd_j unchanged, so the drive into the latents is lost. It tests whether the latents relay anything the policy uses. A removal disconnects the latents entirely, and each remaining neuron’s outflow renormalizes onto its other targets, Tij=Tij/(1−ℓj)T_{ij} = T_{ij}/(1 - \ell_j) with ℓj=∑k∈LTkj\ell_j = \sum_{k \in\mathcal{L}} T_{kj}. It tests whether the latents act as a sink that sets the action modules’ share of the controller’s drive. In the latent path only condition the direct controller-to-action synapses are cut, which tests whether the latents alone can carry the policy. Each model is evaluated intact and under each ablation for 50 episodes with common random numbers, so every model is its own control. QIF’s warm state is not saved, so its intact returns differ slightly from training.

The policy is carried by the direct routing, not by the latents. With it cut, every group fell to about random play (returns of 10–32 against 22.8 for a random policy; Table I, last column). Removing the latents lowered return most in flow (−206 to −287), branching in the dynamic regime (−118) and LIF in the dynamic and full regimes (−112, −80), and by −31 to −65 in the other groups. Without the latents the controller’s whole outflow reaches the action modules, and their share of the drive rose from 0.08–0.49 to 0.57–0.80. The latents therefore act as a sink that sets how much of the controller’s drive reaches the actions, and the trained policy depends on that setting.

Their output is not a relay the policy needs. In full-modularity models the lesion left returns identical to intact, since they have no latent-to-action synapses, and in the dynamic regime it changed almost nothing on QIF and LIF. Where the latents still drive the actions, cutting their output improved return (branching dynamic +47, LIF partial +38), so it acted as interference. Flow in the dynamic regime is

the exception: lesioning one latent module cost return (L2\mathcal{L}_2: −117, L1\mathcal{L}_1: −77), but lesioning both did not (−3), which suggests that the two modules’ small outputs balance each other. Differences between regimes were small. Only LIF (partial against dynamic) and branching (full against dynamic) differed, and the remaining contrasts did not.

When proceeding with the TD-STDP backbone, we test models without latents.

TD-STDP actor-critic backbone

Performance

In the previous subsection, we showed that the minimal spiking policy network can be significantly improved with CoRe. However, in order to show that advancements can be made with this abstraction, we now demonstrate how CoRe can be applied to an existing TD-STDP actor-critic backbone to improve its performance. Figure 3 shows the mean return over training episodes for each environment and arm. The results indicate that when CoRe is applied to the TD-STDP backbone, the performance of the network improves cleanly in CartPole and LunarLander. However, the results on Acrobot are a bit more complicated, when the race readout first replaced the softmax, the performance dropped, but when CoRe was applied, the performance improved and slightly exceeded the original softmax performance in the full regime. Overall, these results suggest that CoRe can be effectively applied to existing TD-STDP actor-critic backbones to improve their performance on reinforcement learning tasks.

Figure 3. TD-STDP actor-critic backbone. The actor network generates actions based on the current state, while the critic network evaluates the actions. The TD-STDP learning rule updates the weights based on the prediction error and the eligibility trace. In panels (a-c), the mean return over training episodes is shown for each environment and arm; 20 seeds per arm. Lunar Lander shows mean return for the last 100 episodes.

EngineRegimeIntactL\mathcal{L}L1\mathcal{L}_1L2\mathcal{L}_2L\mathcal{L}path only
Flowdynamic462−3−77−117−206−455
partial414−18+10−287−418
full4810†0^\dagger0†0^\dagger0†0^\dagger−254−478
Branchingdynamic199+47+18+16−118−167
partial124+18+12+8−65−105
full1780†0^\dagger0†0^\dagger0†0^\dagger−41−137
QIFdynamic1500+1+1−31−140
partial96+13+6+13−50−87
full1920†0^\dagger0†0^\dagger0†0^\dagger−32−179
LIFdynamic327+2+1−2−112−311
partial199+38+18+17−42−181
full3050†0^\dagger0†0^\dagger0†0^\dagger−80−281

Table 1. CHANGE IN RETURN FROM THE INTACT BASELINE WHEN THE LATENTS ARE ABLATED (MEDIAN OVER THE 20 MODELS IN EACH ROW). †\daggerIDENTICAL TO INTACT ON EVERY MODEL. INTACT: MEAN RETURN OF THE INTACT MODELS.

2) Inter-module Connectivity Effects: In the previous sections, we have explored comparing the three regimes of modularity: partial (SBM with pcross=0.05p_{\mathrm{cross}} = 0.05), full (sealed, pcross=0p_{\mathrm{cross}} = 0), and dynamic (learned with the map term). However, it has been left open how much cross-module connectivity is optimal for performance, and what happens when pcrossp_{\mathrm{cross}} is varied. To address this, we conducted a sweep of pcrossp_{\mathrm{cross}} values from 0 (sealed) to 0.5 (no communities) in the TD-STDP actor-critic backbone, with both the map term on and off. When pcross=0p_{\mathrm{cross}} = 0, while the map-term is constrained to act upon intra-module connectivity, this still yields a difference in performance compared to when the map term is off.

The results, shown in Figure 4, indicate that optimal inter-module connectivity is task-dependent, and setting-dependent. Across all three environments, the most optimal pcrossp_{\mathrm{cross}} lies in its lower range. In CartPole, the performance is highest at pcross=0p_{\mathrm{cross}} = 0 with the map term off. In Acrobot, the performance is highest at pcross=0p_{\mathrm{cross}} = 0 with the map term on. In LunarLander, the performance is highest when pcross=0.025p_{\mathrm{cross}} = 0.025 with the map term on. Overall, this suggests that in some tasks, actions may be dependent on each other, and therefore some cross-module connectivity is beneficial, while in other tasks, actions may be independent of each other, and therefore cross-module connectivity is detrimental.

Figure 4. Inter-module connectivity effects in the TD-STDP actor-critic backbone. The figure illustrates how the connectivity between different modules influences the performance and learning dynamics of the network. The Spearman coefficients are shown to the right of each plot, indicating the strength and direction of the relationship between inter-module connectivity and performance.

The trends in the results are also task and setting dependent. While most results show a downward trend in performance as pcrossp_{\mathrm{cross}} increases, there are several exceptions and nuances. Many of the correlation coefficients are brought down by non-monotonicity in the trends, with many trends following a drop off, then a rise, then a drop again, and sometimes another rise. This is especially apparent with CartPole with the map term on, and LunarLander with the map term off.

Overall, these results suggest that the optimal inter-module connectivity is task-dependent, and setting-dependent, and that the map term can have a significant impact on performance. Future work should explore dynamically setting the K×KK \times K inter-module connectivity matrix, rather than a single pcrossp_{\mathrm{cross}} value, to allow for more flexibility in the network’s structure and performance.

Discussion: Biological Plausibility

CoRe SNNs share two features of cortex. First, as motivated in the Introduction, their connectivity is modular: Map-STDP learns weights that are dense within communities and sparse between them, and partial and full modularity impose the same structure directly. Second, all CoRe models learn with three-factor rules, in which a local eligibility changes a weight only when a neuromodulatory signal arrives. Such rules have direct experimental support [30], [31], our task term follows the dopaminergic temporal-difference account of reward learning [32], and the structural modulator is a single scalar per module, in line with the regional specificity of neuromodulation [33]. Two parts remain abstractions. The structural modulator is indexed by the presynaptic module, which requires a roughly uniform or projection-specific signal, and the per-neuron baseline eˉjobs\bar{e}_{j}^{\mathrm{obs}} has no established biological counterpart.

Map-STDP also fits the efficient-coding view of the Introduction. Communication accounts for much of the brain’s energy budget [34], and compressing the network’s activity prunes cross-module traffic. In exploratory measurements, a lower description length coincided with fewer effective synaptic operations per decision (Spearman ρ=0.61\rho= 0.61–0.950.95 across engines). The task term keeps the network useful for acting, and the structural term keeps its activity compressible. Finally, our TD-STDP actor-critic [39] is more biologically plausible than the original. Its actors are recurrent communities, closer to cortical microcircuits and to the channel organization of the basal ganglia [37], [38], it runs in a noisy leaky integrate-and-fire regime at lower firing rates, and it selects actions by a race between action communities, which is equivalent to softmax sampling under constant hazards [40].

Limitations and Future Work

Limitations. Map-STDP relies on a random-walk abstraction of inference. Random walks, and branching processes in particular, are inherently sequential, so estimating the walk statistics per frame becomes a bottleneck as networks scale; a more parallelizable abstraction of inference is needed. Second, the map equation is only one principle that can drive module formation. Alternatives such as the Constant Potts Model [41], modularity maximization [42], attractor models [43], and the dynamic SBM matrices discussed earlier remain unexplored here. Our results suggest that such alternatives are worth testing, because compressibility largely accompanied modularity itself rather than requiring an explicit information-theoretic objective. Objectives that reward modularity directly could therefore yield similar structure without the map equation. Third, as Table I shows, the latent communities act largely as sinks and, at best, routers; we do not yet train them to perform useful computation of their own.

CoRe as a starting point. We view CoRe as a base architecture that opens many directions. Its modular structure decomposes functionality into smaller, separately trainable pieces, and the TD-STDP backbone experiment shows that a CoRe module can be inserted into and coupled with existing, performant architectures and improve them. This opens several directions. Latents could be trained to carry out subtasks or store memory, through persistent activity across frames or through coupling with memory-oriented architectures such as Hopfield networks [44] or Boltzmann machines [45]. Coupling with architectures built for binding and unbinding [46] in the controller could let different latents attend to different objects in the environment. Finally, the same modular interface could host other backbones and other module-forming objectives, so that CoRe serves as a common substrate for these extensions.

Acknowledgment

We thank Dr. Ramakrishnan (Ramki) Kannan at Oak Ridge National Laboratory for providing access to his local cluster. We also thank the AI Interpretability @ Illinois group for providing the collaborative network to make this work possible. The authors used Claude Code (Anthropic) with the Claude Sonnet 5.5, Opus 5.5, and Fable 5.1 models during this work. The AI generated software and experimental scripts used for the results in Section VII; assisted with the derivation in Appendix B; assisted in the iterative design of the figures; and provided language editing throughout the manuscript. All code was reviewed, tested, and validated by the authors against the source data; all derivations were independently checked by the authors; and all text was reviewed and revised by the authors. The authors take full responsibility for the content of this paper, including the integrity of the analysis and results.

Full Results

Derivation of MAP-STDP

This appendix derives the rule in § 5. All logarithms are base 2 (description lengths in bits).

Setup and notation

Each synapse from presynaptic neuron jj to postsynaptic neuron ii has a magnitude Wij≥0W_{ij} \ge0. The type of jj fixes its sign, sgn⁡j=+1\operatorname{sgn}_j = +1 (excitatory) or −1-1 (inhibitory), as in Dale’s law. We write

dj=∑kWkj,Tij=Wijdj,(7)d_j = \sum_k W_{kj}, \qquad T_{ij} = \frac{W_{ij}}{d_j}, \tag*{(7)}

for jj’s total output weight and the share of it that goes to ii. Each neuron belongs to one module m(j)∈{1,…,K}m(j) \in\{1, \ldots, K\}. The module types are the controller C\mathcal{C}, the action modules A\mathcal{A} (one per action) and the latent modules L\mathcal{L}. Environment frames are indexed by ff, and the stimulus of frame ff is ofo_f.

TT is column-stochastic: column jj says where jj’s output goes. A random walk that follows spikes forward moves from jj to ii with probability TijT_{ij}.

The observed walk

A step of the walk from jj to ii is a spike of ii within a short window ww after a spike of jj. The expected number of such spikes splits into three parts:

Eij=ai⏟chance+Dij⏟direct+Xij⏟indirect≥0,(8)E_{ij} = \underbrace{a_i}_{\text{chance}} + \underbrace{D_{ij}}_{\text{direct}} + \underbrace{X_{ij}}_{\text{indirect}} \ge0, \tag*{(8)}
ai=riw,Dij=sgn⁡jλ~ij.a_i = r_i w, \qquad D_{ij} = \operatorname{sgn}_j \widetilde{\lambda}_{ij}.

Here rir_i is ii’s firing rate. λ~ij=hiTij\widetilde{\lambda}_{ij} = h_i T_{ij} is the direct effect of jj’s kick on ii, where hih_i is ii’s windowed sensitivity (gain times kick size). XijX_{ij} collects chains j→k→ij \to k \to i and common input. The observed transition probabilities and link flows are

Ej=∑iEij,Tijobs=EijEj,Jij=πjTijobs,(9)E_j = \sum_i E_{ij}, \qquad T_{ij}^{\mathrm{obs}} = \frac{E_{ij}}{E_j}, \qquad J_{ij} = \pi_j T_{ij}^{\mathrm{obs}}, \tag*{(9)}

where πj\pi_j is jj’s share of all spikes in the frame.

a) Intuition. EijE_{ij} is the mean of a count, so it is non-negative whatever the signs of the synapses. Inhibition does not create negative transitions; it pushes EijE_{ij} below chance.

b) Pair counts measure the walk.: With κij=∑txj(t)Ni(t,t+w]\kappa_{ij}=\sum_t x_j(t)N_i(t,t+w], where xj(t)x_j(t) marks a spike of jj at step tt and NiN_i counts spikes of ii, stationarity within the frame gives E[κij]=nˉjEij\mathbb{E}[\kappa_{ij}]=\bar{n}_jE_{ij}. So

T^ijobs=κij∑kκkj,eˉjobs=∑i∉m(j)κij∑iκij.(10)\widehat{T}^{\mathrm{obs}}_{ij}=\frac{\kappa_{ij}}{\sum_k\kappa_{kj}},\qquad\bar{e}^{\mathrm{obs}}_j=\frac{\sum_{i\notin m(j)}\kappa_{ij}}{\sum_i\kappa_{ij}}. \tag*{(10)}

eˉjobs\bar{e}^{\mathrm{obs}}_j is the share of jj’s observed transitions that leave its module. A neuron can estimate it from its own spike times and two population signals: activity in its own module, and in the whole network.

c) The flow engine is the special case.: Without spikes there is no chance (a=0a=0) and no indirect term (X=0X=0), and all gains are equal, so Tobs=TT^{\mathrm{obs}}=T. The stimulus v(o)v(\mathrm{o}) re-injects a share α\alpha of the activity at each step:

π=(1−α)Tπ+αv(o),∑iπi=1.(11)\pi=(1-\alpha)T\pi+\alpha v(\mathrm{o}),\qquad\sum_i\pi_i=1. \tag*{(11)}

For α>0\alpha>0 the solution is unique. Starting from x0=vx_0=v, nn power-iteration hops give ∥xn−π∥1≤2(1−α)n\lVert x_n-\pi\rVert_1\le2(1-\alpha)^n, whatever the network size.

Communities as states and the policy

Coarse-graining the frame’s walk to modules gives a KK-state Markov chain, with tables indexed [post, pre]:

pa=∑j∈aπj,Jba=∑i∈b∑j∈aJij,TbaK(of)=Jbapa.(12)p_a=\sum_{j\in a}\pi_j,\qquad J_{ba}=\sum_{i\in b}\sum_{j\in a}J_{ij},\qquad T^K_{ba}(\mathrm{o_f})=\frac{J_{ba}}{p_a}. \tag*{(12)}

Because the stimulus is concentrated in the controller, the controller’s column of TKT^K is the stimulus-dependent one. The action is read from the flow into the action modules,

P(a∣of)=pa(f)β∑a′∈Apa′(f)β,a∈A,(13)P(a\mid\mathrm{o_f})=\frac{p_a(f)^\beta}{\sum_{a'\in A}p_{a'}(f)^\beta},\qquad a\in A, \tag*{(13)}

with β=1\beta=1 the plain readout. Model v4 adapts β\beta to the trend of episode return. In spiking engines the readout is a race between the action modules’ spikes, which samples (13).

The map equation

Describe the walk with a two-level code. Inside a module, name the next neuron, or “exit”, with a module-local codeword. On exit, name the next module from a global index. The average code length per step is [24]

L=q↷log⁡q↷−2∑mqmlog⁡qm−∑jπjlog⁡πj+∑m(pm+qm)log⁡(pm+qm),(14)L=q_{\curvearrowright}\log q_{\curvearrowright}-2\sum_m q_m\log q_m -\sum_j\pi_j\log\pi_j+\sum_m(p_m+q_m)\log(p_m+q_m), \tag*{(14)}

where qm=∑j∈m∑i∉mJijq_m=\sum_{j\in m}\sum_{i\notin m}J_{ij} is module mm’s exit flow and q↷=∑mqmq_{\curvearrowright}=\sum_m q_m. With π\pi held fixed, the weights enter LL only through the exit flows, and

Mm∗:=∂L∂qm=log⁡q↷(pm+qm)qm2≥0.(15)M_m^*:=\frac{\partial L}{\partial q_m}=\log\frac{q_{\curvearrowright}(p_m+q_m)}{q_m^2}\ge0. \tag*{(15)}

a) Intuition.: Leakage out of a module always costs bits. Mm∗M_m^* is the marginal number of bits per unit of exit flow from mm. It is computed from KK module-level statistics and broadcast to module mm: this is the rule’s neuromodulator.

b) The inverse problem.: Map-STDP pins the partition and minimizes over the weights, averaged over the frames the environment produces:

min⁡WD(W),D(W)=Ef[L(Jf)].(16)\min_W D(W),\qquad D(W)=\mathbb{E}_f[L(J_f)]. \tag*{(16)}

c) Check.: Differentiate (14) term by term. The derivative of xlog⁡xx\log x is log⁡x+c\log x+c for a constant cc, and the constants cancel (c−2c+c=0c-2c+c=0). The result is log⁡q↷−2log⁡qm+log⁡(pm+qm)\log q_{\curvearrowright}-2\log q_m+\log(p_m+q_m). It is non-negative because q↷≥qmq_{\curvearrowright}\ge q_m and $p_m+q_m\ge q_m.

The structural rule

Let cijc_{ij} be the signed counter: the change in ii’s spikes within the window caused by jj’s kicks alone, with E[cij]=nˉjDij\mathbb{E}[c_{ij}]=\bar{n}_jD_{ij}. The structural update is

ΔWij=−λ(gij−gˉjobs)cij,\Delta W_{ij}=-\lambda\left(g_{ij}-\bar{g}^{\mathrm{obs}}_j\right)c_{ij},
gij=∂D∂Jij,gˉjobs=∑kTkjobsgkj.(17)g_{ij}=\frac{\partial D}{\partial J_{ij}},\qquad\bar{g}^{\mathrm{obs}}_j=\sum_k T^{\mathrm{obs}}_{kj}g_{kj}. \tag*{(17)}

For the map equation, qmq_m is a sum of the JijJ_{ij} with j∈mj\in m and i∉mi\notin m, so

gij=Mm(j)∗I[i∉m(j)],gˉjobs=Mm(j)∗eˉjobs,(18)g_{ij}=M^*_{m(j)}\mathbb{I}[i\notin m(j)],\qquad\bar{g}^{\mathrm{obs}}_j=M^*_{m(j)}\bar{e}^{\mathrm{obs}}_j, \tag*{(18)}

and the rule becomes

ΔWij=−λMm(j)∗(I[i∉m(j)]−eˉjobs)cij.(19)\Delta W_{ij}=-\lambda M^*_{m(j)}\left(\mathbb{I}[i\notin m(j)]-\bar{e}^{\mathrm{obs}}_j\right)c_{ij}. \tag*{(19)}

a) Two statistics, two jobs.: The baseline eˉjobs\bar{e}^{\mathrm{obs}}_j describes the walk, so it comes from pair counts, chance included. The step must say how the walk changes when WijW_{ij} changes, which is the direct effect, so it comes from the signed counter. Using pair counts for the step fails: they are mostly chance, and renormalization cancels them.

b) Sign pattern.: The factor I[i∉m(j)]−eˉjobs\mathbb{I}[i\notin m(j)]-\bar{e}^{\mathrm{obs}}_j is positive across modules and negative within. The counter carries sgn⁡j\operatorname{sgn}_j. Hence:

SynapseAcross modulesWithin a module
Excitatoryweakenedstrengthened
Inhibitorystrengthenedweakened

Table 1 (PDF p. 13).

Why the rule descends the description length

a) Local split.: While differentiating with respect to one weight, hold fixed everything a single synapse does not directly touch: the flow π\pi, chance aa, the indirect term XX and the gains. Then WijW_{ij} changes only column jj’s direct part, Dkj=sgn⁡jhkWkj/djD_{kj}=\operatorname{sgn}_j h_kW_{kj}/d_j (through WijW_{ij} itself and through djd_j).

Lemma 1 (Lemma E). If DD depends on WW only through the link flows JJ, then under the local split

Wij∂D∂Wij=πjsgn⁡jEj[Λ~ij(gij−gˉjobs)−TijCj],W_{ij}\frac{\partial D}{\partial W_{ij}}=\frac{\pi_j\operatorname{sgn}_j}{E_j}\left[\widetilde{\Lambda}_{ij}\left(g_{ij}-\bar{g}^{\mathrm{obs}}_j\right)-T_{ij}C_j\right],
Cj=∑kΛ^kj(gkj−gˉjobs).(20)C_j=\sum_k\widehat{\Lambda}_{kj}\left(g_{kj}-\bar{g}^{\mathrm{obs}}_j\right). \tag*{(20)}

Proof sketch. Under the split, ∂Ekj/∂Wij=sgn⁡j(hk/dj)(δki−Tkj)\partial E_{kj}/\partial W_{ij}=\operatorname{sgn}_j(h_k/d_j)(\delta_{ki}-T_{kj}), and Jkj=πjEkj/EjJ_{kj}=\pi_jE_{kj}/E_j. By the quotient rule,

∂Jkj∂Wij=πjEj(∂Ekj∂Wij−Tkjobs∂Ej∂Wij).\frac{\partial J_{kj}}{\partial W_{ij}}=\frac{\pi_j}{E_j}\left(\frac{\partial E_{kj}}{\partial W_{ij}}-T^{\mathrm{obs}}_{kj}\frac{\partial E_j}{\partial W_{ij}}\right).

Summing against gkjg_{kj}, and using ∂Ej/∂Wij=∑k∂Ekj/∂Wij\partial E_j/\partial W_{ij}=\sum_k\partial E_{kj}/\partial W_{ij},

∂D∂Wij=πjEj∑k(gkj−gˉjobs)∂Ekj∂Wij=πjsgn⁡jEjdj[hi(gij−gˉjobs)−∑khkTkj(gkj−gˉjobs)].\begin{aligned} \frac{\partial D}{\partial W_{ij}}=\frac{\pi_j}{E_j}\sum_k\left(g_{kj}-\bar{g}^{\mathrm{obs}}_j\right)\frac{\partial E_{kj}}{\partial W_{ij}} \\ &=\frac{\pi_j\operatorname{sgn}_j}{E_jd_j}\left[h_i\left(g_{ij}-\bar{g}^{\mathrm{obs}}_j\right)-\sum_k h_kT_{kj}\left(g_{kj}-\bar{g}^{\mathrm{obs}}_j\right)\right]. \end{aligned}

Multiplying by WijW_{ij}, with Wij/dj=TijW_{ij}/d_j=T_{ij} and hiTij=Λ~ijh_iT_{ij}=\widetilde{\Lambda}_{ij}, gives (20).

b) Intuition.: CjC_j is the same for every target ii of jj, so TijCjT_{ij}C_j rescales column jj uniformly. That cannot change TT, so it cannot change DD; hence ∑iWij∂D/∂Wij=0\sum_iW_{ij}\partial D/\partial W_{ij}=0. The other factors read as follows:

  • πj/Ej\pi_j/E_j is a per-neuron step size;

  • sgn⁡j\operatorname{sgn}_j flips the direction for inhibitory synapses;

  • Λ~ij\widetilde{\Lambda}_{ij} scales the step by how much jj actually moves ii;

  • gij−gˉjobsg_{ij}-\bar{g}^{\mathrm{obs}}_j compares this transition’s marginal cost with the average over jj’s observed transitions.

Theorem 1 (Theorem E). If the counter is unbiased, E[cij]=Tfrjsgn⁡jΛ~ij\mathbb{E}[c_{ij}]=T_f r_j\operatorname{sgn}_j\tilde{\Lambda}_{ij} over a frame of duration TfT_f, then

E[ΔWij]=−λζEjWij∂D∂Wij−λTfrjsgn⁡jCjTij,\mathbb{E}[\Delta W_{ij}]=-\lambda\zeta E_jW_{ij}\frac{\partial D}{\partial W_{ij}}-\lambda T_f r_j\operatorname{sgn}_j C_jT_{ij},
ΔD=−λζ∑ijEjWij(∂D∂Wij)2≤0,(21)\Delta D=-\lambda\zeta\sum_{ij}E_jW_{ij}\left(\frac{\partial D}{\partial W_{ij}}\right)^2\leq0, \tag*{(21)}

with ζ=Tf∑krk\zeta=T_f\sum_k r_k.

Proof sketch. Solve (20) for sgn⁡jΛ~ij(gij−gˉjobs)\operatorname{sgn}_j\tilde{\Lambda}_{ij}(g_{ij}-\bar{g}^{\mathrm{obs}}_j), substitute it into −λ(gij−gˉjobs)E[cij]-\lambda(g_{ij}-\bar{g}^{\mathrm{obs}}_j)\mathbb{E}[c_{ij}], and use πj=rj/∑krk\pi_j=r_j/\sum_k r_k. To first order, ΔD=∑ij(∂D/∂Wij)ΔWij\Delta D=\sum_{ij}(\partial D/\partial W_{ij})\Delta W_{ij}. The column-rescaling term contributes ∝∑iWij∂D/∂Wij=0\propto\sum_i W_{ij}\partial D/\partial W_{ij}=0.

c) Intuition.: On average the rule is gradient descent on DD, preconditioned by the positive diagonal factor EjWijE_jW_{ij} (which never has to be computed), plus a column rescaling that cannot change DD.

d) Flow-engine case.: With a=X=0a=X=0, excitation only and equal gains, Tobs=TT^{\mathrm{obs}}=T and Cj=0C_j=0, and Lemma E reduces to

∂D∂Wij=πjdj(gij−gˉj),gˉj=∑kTkjgkj.(22)\frac{\partial D}{\partial W_{ij}}=\frac{\pi_j}{d_j}(g_{ij}-\bar{g}_j),\qquad\bar{g}_j=\sum_k T_{kj}g_{kj}. \tag*{(22)}

The flow engine replaces the counter by its expectation JijJ_{ij}, giving E[ΔWij]=−λWij∂D/∂Wij\mathbb{E}[\Delta W_{ij}]=-\lambda W_{ij}\partial D/\partial W_{ij}: gradient descent with each coordinate scaled by its own weight.

Routing exemption

The policy is carried by cross-module flow: controller →\to action and controller →\to latent. Applied to those synapses, the map term would drive the controller’s exit flow to zero and starve the action modules. The routing pairs are therefore exempt: gba=0g_{ba}=0 for a=Ca=C and b∈A∪Lb\in\mathcal{A}\cup\mathcal{L}. The controller then acts as an input layer, and the map term seals the latent and action modules.

Task term

The task is learned by a TD-modulated, action-gated eligibility. Take the surrogate policy P~(a)=Jain/∑a′∈AJa′in\tilde{P}(a)=J^{\mathrm{in}}_a/\sum_{a'\in\mathcal{A}}J^{\mathrm{in}}_{a'}, with Jbin=∑i∈b∑jJijJ^{\mathrm{in}}_b=\sum_{i\in b}\sum_j J_{ij}. Its score with respect to JijJ_{ij} depends only on the module of ii:

hb=I[b=a∗]Ja∗in−I[b∈A]∑a′∈AJa′in,hˉj=∑kTkjhm(k),(23)h_b=\frac{\mathbb{I}[b=a^*]}{J^{\mathrm{in}}_{a^*}}-\frac{\mathbb{I}[b\in\mathcal{A}]}{\sum_{a'\in\mathcal{A}}J^{\mathrm{in}}_{a'}},\qquad\bar{h}_j=\sum_k T_{kj}h_{m(k)}, \tag*{(23)}

where a∗a^* is the chosen action. The eligibility trace and the update are

eij←(1−1τe)eij+cij(hm(i)−hˉj),ΔWij=ηδfeij,(24)e_{ij}\leftarrow\left(1-\frac{1}{\tau_e}\right)e_{ij}+c_{ij}\left(h_{m(i)}-\bar{h}_j\right),\qquad\Delta W_{ij}=\eta\delta_f e_{ij}, \tag*{(24)}

with the TD error and a linear critic trained by normalized LMS:

δf=rf+γVf+1−Vf,Vf=u⋅xf,\delta_f=r_f+\gamma V_{f+1}-V_f,\qquad V_f=u\cdot x_f,
u←u+ηVδfxf∥xf∥2.(25)u\leftarrow u+\eta_V\delta_f\frac{x_f}{\lVert x_f\rVert^2}. \tag*{(25)}

In the flow engine, by the same Lemma, the increment of (24) is Wij∂log⁡P~(a∗)/∂WijW_{ij}\partial\log\tilde{P}(a^*)/\partial W_{ij}: a one-hop policy-gradient score.

Homeostasis

A presynaptic homeostasis term holds each neuron’s output weight near 1:

ΔWij=−ϵh(dj−1)πjWijdj.(26)\Delta W_{ij}=-\epsilon_h(d_j-1)\frac{\pi_jW_{ij}}{d_j}. \tag*{(26)}

It rescales column jj uniformly, so it leaves TT, the walk and DD unchanged. The full per-frame update, Eq. (6), is the sum of the task, structure and homeostasis terms.

Computing the counter in each engine

Each engine uses the same rule. What differs is how cijc_{ij} is computed.

a) Branching.: The neuron is ui(t+1)=∑jbijxj(t)+noiseu_i(t+1)=\sum_j b_{ij}x_j(t)+\text{noise}, which spikes with probability p(u)=clip⁡(u/2θ,0,1)p(u)=\operatorname{clip}(u/2\theta,0,1). The kick bij=sgn⁡jw0Tijb_{ij}=\operatorname{sgn}_j w_0T_{ij} is output-normalized. Removing jj’s kick changes ii’s firing probability by a known amount, so the counter is exact:

cij=∑txj(t)[p(ui(t+1))−p(ui(t+1)−bij)].(27)c_{ij}=\sum_t x_j(t)\left[p\left(u_i(t+1)\right)-p\left(u_i(t+1)-b_{ij}\right)\right]. \tag*{(27)}

b) LIF and QIF. A kick of size bb is an input pulse of area τb\tau b. In linear response it changes ii’s spike count by G^iτb\hat{G}_i\tau b, where G^i\hat{G}_i is ii’s gain (the slope of its rate transfer). So

cij=τG^ibijnj,(28)c_{ij}=\tau\hat{G}_i b_{ij}n_j, \tag*{(28)}

an outer product of the presynaptic spike count and the postsynaptic gain. For LIF the gain comes from the Siegert rate formula, or from the exact Markov transfer of the discretized neuron. For QIF it comes from regressing the rate on the injected noise.

Stability

The walk always exists, but the counters rely on linear response around a stable operating point. With Λ\Lambda the signed offspring matrix (extra spikes of ii per spike of jj), the conditions are

branching (map):max⁡k∣λk(Λ)∣<1,LIF, QIF (continuous):max⁡kRe⁡λk(Λ)<1.(29)\begin{aligned} \text{branching (map)}:\quad&\max_k\lvert\lambda_k(\Lambda)\rvert< 1,\\ \text{LIF, QIF (continuous)}:\quad&\max_k \operatorname{Re}\lambda_k(\Lambda) < 1. \tag*{(29)} \end{aligned}

The network monitors them through a module-level offspring table estimated from the counters.

Map-STDP communities in the TD-STDP backbone

In the hybrid model, the actor of the TD-STDP actor-critic of Chung and Kozma [39] is organized into modules:

  • Controller. The controller CC is the Fourier input population.

  • Action communities. Each action’s excitatory actor population is one module.

  • Recurrence. Recurrent synapses RR connect all actor excitatory neurons, with output-normalized kicks bij=χrecRij/djb_{ij}=\chi_{\mathrm{rec}}R_{ij}/d_j.

  • Routing. The input →\to actor weights are exempt routing, learned by the TD-STDP rule alone.

  • Where the map term acts. The structural term acts on RR only, with the LIF counter cij=τG^ibijnjc_{ij}=\tau\hat{G}_i b_{ij}n_j accumulated per window.

The action is sampled from a softmax over community rates, pk∝exp⁡(αaρˉk)p_k\propto\exp(\alpha_a\bar{\rho}_k). This is equivalent to a race of escape-noise decision units with constant hazards exp⁡(αaρˉk)\exp(\alpha_a\bar{\rho}_k), in which the first unit to spike wins [40]. Each unit’s local score I[k=a]−Λk(t∗)\mathbb{I}[k=a]-\Lambda_k(t^*), with Λk\Lambda_k its integrated hazard up to the decision time t∗t^*, has expectation Ak−pkA_k-p_k given the winner, where AA is the one-hot chosen action. So the community credit hb∝Ab−pbh_b\propto A_b-p_b of the backbone is available locally. The task term on RR uses the one-hop credit hm(i)−hˉjh_{m(i)}-\bar{h}_j of (24).

Random Walk Entropy and SNN Energy Usage

In this section we discuss in more depth how entropy of a random walk abstraction of SNN inference relates to the energy usage of the network (Figure 5 and Table III).

Figure 5. Energy use and description length across the inter-module connectivity sweep in the TD-STDP actor-critic backbone, for (a) CartPole, (b) Acrobot and (c) LunarLander. Each point is one final network (77 values of pcross×2p_{\mathrm{cross}} \times2 map-term settings ×20\times20 seeds, n=140n = 140); color gives pcrossp_{\mathrm{cross}} and shape the map-term setting (dynamic with the term on, partial with it off, full at pcross=0p_{\mathrm{cross}} = 0 with it off). Columns 1–3: effective synaptic operations (SOPs) per decision against LobsL_{\mathrm{obs}} and LWL_W, and LobsL_{\mathrm{obs}} against LWL_W, with the Spearman ρ\rho over all points above each panel. Column 4: Spearman ρ\rho within each pcrossp_{\mathrm{cross}} value (n=20n = 20 per setting), with bootstrap intervals.

GroupnnSOPs vs LobsL_{\mathrm{obs}}SOPs vs LWL_WLobsL_{\mathrm{obs}} vs LWL_WReturn vs LobsL_{\mathrm{obs}}Return vs LWL_W
Map-STDP v4, CartPole
Branching60+0.61* [+0.35, +0.77]+0.71* [+0.52, +0.81]+0.49* [+0.21, +0.67]−0.49* [−0.70, −0.24]−0.19 [−0.44, +0.08]
QIF60+0.95* [+0.90, +0.97]+0.93* [+0.87, +0.96]+0.92* [+0.86, +0.95]−0.09 [−0.35, +0.18]−0.11 [−0.37, +0.17]
LIF60+0.91* [+0.86, +0.94]+0.91* [+0.86, +0.93]+0.92* [+0.86, +0.95]−0.50* [−0.68, −0.28]−0.45* [−0.64, −0.23]
TD-STDP, CartPole
All regimes60+0.88* [+0.78, +0.94]+0.72* [+0.53, +0.84]+0.71* [+0.52, +0.84]——
Dynamic20+0.83* [+0.56, +0.95]+0.55* [+0.11, +0.85]+0.66* [+0.20, +0.93]——
Partial20+0.65* [+0.24, +0.87]−0.23 [−0.61, +0.21]−0.29 [−0.73, +0.24]——
Full20+0.65* [+0.24, +0.88]+0.25 [−0.22, +0.65]+0.02 [−0.52, +0.55]——
TD-STDP, Acrobot
All regimes60+0.02 [−0.24, +0.28]+0.38* [+0.11, +0.63]+0.07 [−0.19, +0.31]−0.04 [−0.33, +0.24]−0.10 [−0.34, +0.14]
Dynamic20+0.32 [−0.21, +0.77]+0.46* [+0.03, +0.80]+0.61* [+0.24, +0.82]+0.02 [−0.44, +0.45]+0.26 [−0.24, +0.66]
Partial20+0.18 [−0.32, +0.61]−0.15 [−0.59, +0.41]−0.64* [−0.87, −0.21]+0.13 [−0.36, +0.59]−0.30 [−0.71, +0.20]
Full20−0.13 [−0.59, +0.40]+0.18 [−0.34, +0.60]+0.06 [−0.41, +0.51]−0.39 [−0.75, +0.08]−0.02 [−0.52, +0.41]
TD-STDP, LunarLander
All regimes60+0.49* [+0.25, +0.69]−0.47* [−0.69, −0.17]−0.01 [−0.30, +0.30]+0.19 [−0.09, +0.43]−0.12 [−0.37, +0.16]
Dynamic20+0.11 [−0.44, +0.55]+0.29 [−0.14, +0.65]+0.26 [−0.27, +0.71]+0.07 [−0.50, +0.54]−0.09 [−0.52, +0.41]
Partial20+0.58* [+0.13, +0.79]−0.69* [−0.87, −0.35]−0.39 [−0.72, +0.12]+0.65* [+0.22, +0.88]−0.79* [−0.93, −0.50]
Full20+0.26 [−0.24, +0.69]+0.49* [+0.03, +0.79]−0.14 [−0.60, +0.37]+0.18 [−0.30, +0.60]−0.08 [−0.54, +0.41]
All settings360+0.64* [+0.56, +0.71]+0.53* [+0.45, +0.60]+0.52* [+0.43, +0.59]−0.19* [−0.30, −0.07]−0.19* [−0.30, −0.08]

Table III. Spearman correlation between energy use, random-walk description length and performance of the final networks: effective synaptic operations (SOPs) per decision, the map equation of the observed walk (LobsL_{\mathrm{obs}}) and of the walk on the trained weights (LWL_W), and the mean return of the frozen final network. ρ\rho with its bootstrap 95% interval (2000 resamples over seeds) below; * p<0.05p < 0.05. Map-STDP v4: per spiking engine, pooled over the three modularity regimes. TD-STDP: all regimes pooled, then within each regime. All settings: each variable ranked within its setting (three v4 engines, three TD-STDP tasks) and pooled; bootstrap stratified by setting. —: Every frozen TD-STDP CartPole network reaches the 500 ceiling, so its return is constant; that setting is left out of the all-settings return columns (n=300n = 300).

Energy Usage

In this supplementary study, we measure energy usage through spiking operations (SOPs). An SOP is one spike delivered across one synapse. Each spike is weighted by the number of outgoing synapses of the neuron that fired it:

SOPs=∑jnj⋅∣{i:Wij>2wmin⁡}∣\mathrm{SOPs}=\sum_j n_j\cdot\left|\{i:W_{ij}>2w_{\min}\}\right|

Here nin_i is the spike count of neuron ii. The count only includes synapses above twice the weight floor (wmin⁡=10−3w_{\min}=10^{-3}), the “effective” synapses. Synapses the rule has pushed down to the floor are treated as pruned, so an SOP is only counted where a synapse effectively exists. SOPs are computed on frozen weight models without any continual learning.

LWL_W and LobsL_{\mathrm{obs}}

There are two measurements of info-theoretic entropy in the random-walk abstraction: the map equation of the observed walk (LobsL_{\mathrm{obs}}) and the map equation of the walk on the trained weights (LWL_W). Both description lengths are the map equation of a walk with visit rates π\pi and transitions TT over the modules mm (see Equation (5)). They differ in the walk they use:

LW=L(πW,TW),TijW=Wij/∑kWkj,πW=(1−α)TWπW+α1N1,α=0.2.(30)\begin{aligned} L_W &= L(\pi^W,T^W),\\ T^W_{ij} &= W_{ij}/\sum_k W_{kj},\\ \pi^W &= (1-\alpha)T^W\pi^W+\alpha\frac{1}{N}\mathbf{1},\quad\alpha=0.2. \tag*{(30)} \end{aligned}
Lobs=⟨L(πf,Tfobs)⟩f,Tijobs=κij/∑kκkj,πf,j=nj/∑knk.(31)\begin{aligned} L_{\mathrm{obs}} &= \left\langle L(\pi_f,T_f^{\mathrm{obs}})\right\rangle_f,\\ T_{ij}^{\mathrm{obs}} &= \kappa_{ij}/\sum_k\kappa_{kj},\\ \pi_{f,j} &= n_j/\sum_k n_k. \tag*{(31)} \end{aligned}

LWL_W is computed from the weights alone (the recurrent weights RR for TD-STDP) under uniform teleportation. LobsL_{\mathrm{obs}} is computed from spikes and averaged over frames ff. In it, κij\kappa_{ij} counts spikes of ii within the engine’s window after a spike of jj, and njn_j is the spike count of neuron jj in the frame.

Info-theoretic entropy and downstream energy use

No experiment was run on the flow model as there are no spiking operations in the flow model. The full correlation table can be found in Table III. Overall, there is general agreement between energy usage and description length (Spearman correlations: +0.64 overall SOPs vs LobsL_{\mathrm{obs}}, +0.53 overall SOPs vs LWL_W), showing strong evidence for lower description length yielding lower energy usage. Moreover, there is even a weak negative correlation between performance and the two description length quantities (-0.19 Return vs LobsL_{\mathrm{obs}} and vs LWL_W), providing a small amount of evidence for Hinton’s compressibility argument. However, the agreement between info-theoretic entropy via description length and downstream energy usage, as well as the two measures of description length, depends on the model and the task.

The correlation between SOPs and LobsL_{\mathrm{obs}} becomes weak and even goes slightly negative in the full modularity regime in the TD-STDP backbone on Acrobot (-0.13 Spearman correlation). The correlation between SOPs and LWL_W, and between LobsL_{\mathrm{obs}} and LWL_W drops heavily on LunarLander, indicating that preemptive info-theoretic analysis holds weaker on this task.

In Figure 5, showing the correlation between SOPs and the two description lengths across pcrossp_{\mathrm{cross}} values, we see a similar trend. There is a positive correlation between SOPs and LobsL_{\mathrm{obs}} across all three tasks (CartPole: +0.93, Acrobot: +0.43, LunarLander: +0.41). The correlation between all three quantities are strongest in CartPole (SOPs vs LobsL_{\mathrm{obs}}: +0.93, SOPs vs LWL_W: +0.83, LobsL_{\mathrm{obs}} vs LWL_W: +0.78). And LunarLander holds the weakest correlation, with theoretical description length analysis even serving counterintuitively (SOPs vs LobsL_{\mathrm{obs}}: +0.41, SOPs vs LWL_W: -0.43, LobsL_{\mathrm{obs}} vs LWL_W: -0.24). Across the pcrossp_{\mathrm{cross}} values, in the plots on the rightmost column, correlations hold positive consistently in CartPole and Acrobot, and while SOPs and LobsL_{\mathrm{obs}} hold positive, the correlations with LWL_W drop negative consistently. This opens the question of what attributes the LunarLander task has for preemptive info-theoretic analysis to hold so counterintuitively. This question is explored in the following subsection.

The Special Case of LunarLander

Synaptic operations (SOPs) here track the number of surviving synapses, not activity. Within each task, column spikes per decision barely change across regimes: about 5 in CartPole, 92 in Acrobot and 211–223 in LunarLander, since the rate-matching settings of these studies hold firing rates fixed. SOPs therefore vary only through the effective fan-out, the number of a neuron’s recurrent synapses still above threshold (Table IV).

TaskDynamicFullPartial
CartPole3.39.59.5
Acrobot7.19.68.3
LunarLander15.615.66.2

Table IV. Effective fan-out (recurrent synapses per neuron above 2ωmin⁡2\omega_{\min}) of the final networks, mean over seeds.

LWL_W, in contrast, measures how much of the weight crosses between columns. The two usually move together because one process drives both: the map term removes cross-column synapses, lowering the synapse count and LWL_W at once. Without the map term, each task treats the cross-column synapses differently (Table V).

TaskMean cross-column weightWithin-column synapsesWithin-column share
CartPole0.094 → 0.0019.5 → 9.50.91 → 1.00
Acrobot0.084 → 0.1879.5 → 6.90.83 → 0.62
LunarLander0.048 → 0.14015.6 → 4.50.77 → 0.31

Table V. Recurrent weights of the partial ARM (MAP term off) from the start to the end of training, mean over seeds. Within-column synapses: per neuron, above 2wmin⁡2w_{\min}. Within-column share: fraction of the total recurrent weight inside columns.

In CartPole the task rule removes the cross-column synapses by itself within about 200 episodes, so the partial arm ends up equal to the full one. In Acrobot the cross-column weights double, but the columns survive. In LunarLander the cross-column weights triple and most within-column synapses fall below threshold, mainly between episodes 2,500 and 5,500. The final within-column share of 0.31 is below the 0.77 of the random initial wiring, so the network ends up less modular than chance.

In LunarLander’s partial arm, then, the pruned synapses are the within-column ones, and fewer synapses go with more cross-column flow. This makes SOPs and LWL_W negatively correlated (ρ=−0.47\rho=-0.47 over all regimes, −0.69-0.69 within the partial arm). With the map term on, the cross-column synapses go and all 15.6 within-column synapses stay, which is why the dynamic and full arms end up with the same SOPs and LWL_W.

Why the within-column synapses decay is not yet established. Candidates are the rate cost, homeostasis, or the task rule over LunarLander’s much longer training (10,000 episodes, against 400 for CartPole and 500 for Acrobot).

A fixed partition thus appears too rigid for this task, and unconstrained coupling too loose. The map term alone has limitations expressing a middle ground: its modulator gba=Ma∗I[b≠a]g_{ba}=M_a^*\mathbb{I}[b\ne a] charges every exit from column aa the same marginal cost, whatever the target column bb. A natural extension is an adaptive K×KK \times K module-pair table learned for instance from the task’s credit, so that useful directed links between action columns are kept while the rest are pruned. We leave this to future work.

References

  1. [1]K. Malcolm and J. Casco-Rodriguez, “A comprehensive review of spiking neural networks: Interpretation, optimization, efficiency, and best practices,” arXiv:2303.10780 [cs.NE], 2023.arxiv.org/abs/2303.10780
  2. [2]P. A. Morella et al., “A memristor spiking-neuron integrated circuit with a scalable communication network and interface,” Science, vol. 345, no. 6197, pp. 668–673, 2014, doi: 10.1126/science.1254642.
  3. [3]M. Davies et al., “Loihi: A neuromorphic manycore processor with on-chip learning,” IEEE Micro, vol. 38, pp. 82–99, 2018.DOI
  4. [4]M. Prezioso et al., “Training and operation of an integrated neuromorphic network based on metal-oxide memristors,” Nature, vol. 521, pp. 61–64, 2015, doi: 10.1038/nature14441.
  5. [5]M. Davies et al., “Advancing neuromorphic computing with Loihi: A survey of results and outlook,” Proc. IEEE, vol. 109, no. 5, pp. 911–934, 2021.DOI
  6. [6]E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks,” arXiv:1901.09948 [cs.NE], 2019.arxiv.org/abs/1901.09948
  7. [7]T. P. Lillicrap, A. Santoro, L. Marris, C. J. Akerman, and G. E. Hinton, “Backpropagation and the brain,” Nat. Rev. Neurosci., vol. 21, pp. 335–346, 2020.
  8. [8]H. Markram, J. Lübke, M. Frotscher, and B. Sakmann, “Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs,” Science, vol. 275, no. 5297, pp. 213–215, 1997, doi: 10.1126/science.275.5297.213.DOI
  9. [9]G. Bi and M. Poo, “Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type,” J. Neurosci., vol. 18, no. 24, pp. 10464–10472, 1998.
  10. [10]P. U. Diehl and M. Cook, “Unsupervised learning of digit recognition using spike-timing-dependent plasticity,” Front. Comput. Neurosci., vol. 9, p. 99, 2015, doi: 10.3389/fncom.2015.00099.DOI
  11. [11]P. Date et al., “SuperNeuronMAT: An efficient matrix-based simulator for spiking neural networks,” arXiv:2608.008479 [cs.NE], 2026.
  12. [12]J. M. Beggs and D. Plenz, “Neuronal avalanches in neocortical circuits,” J. Neurosci., vol. 23, no. 35, pp. 11167–11177, 2003.
  13. [13]L. Lapicque, “Recherches quantitatives sur l’excitation électrique des nerfs traitée comme une polarisation,” J. Physiol. Pathol. Gen., vol. 9, pp. 620–635, 1907.
  14. [14]N. Brunel, “Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons,” J. Comput. Neurosci., vol. 8, no. 3, pp. 183–208, 2000.
  15. [15]G. B. Ermentrout and N. Kopell, “Parabolic bursting in an excitable system coupled with a slow oscillation,” SIAM J. Appl. Math., vol. 46, no. 2, pp. 233–253, 1986.
  16. [16]E. Montbrió, D. Pazó, and A. Roxin, “Macroscopic description for networks of spiking neurons,” Phys. Rev. X, vol. 5, no. 2, p. 021028, 2015.
  17. [17]S. Brin and L. Page, “The anatomy of a large-scale hypertextual web search engine,” Comput. Netw., vol. 30, pp. 107–117, 1998.
  18. [19]M. Rosvall, D. Axelsson, and C. T. Bergstrom, “The map equation,” Eur. Phys. J. Spec. Top., vol. 178, no. 1, pp. 13–23, 2009, doi: 10.1140/epjst/e2010-01179-1.DOI
  19. [20]M. Rosvall and C. T. Bergstrom, “Mapping change in large networks,” PLoS ONE, vol. 5, no. 1, p. e8694, 2010, doi: 10.1371/journal.pone.0008694.DOI
  20. [21]P. W. Holland, K. B. Laskey, and S. Leinhardt, “Stochastic blockmodels: First steps,” Soc. Netw., vol. 5, no. 2, pp. 109–137, 1983.
  21. [22]M. Kiselev, “CoLaNET – A spiking neural network with columnar layered architecture for classification,” arXiv:2409.01230 [cs.NE], 2025.arxiv.org/abs/2409.01230
  22. [23]M. Kiselev, “Classifying images with CoLaNET spiking neural network – the MNIST example,” arXiv:2409.07833 [cs.NE], 2024.arxiv.org/abs/2409.07833
  23. [24]M. Rosvall and C. T. Bergstrom, “Maps of random walks on complex networks reveal community structure,” Proc. Natl. Acad. Sci. USA, vol. 105, no. 4, pp. 1118–1123, 2008.
  24. [25]G. E. Hinton and D. van Camp, “Keeping the neural networks simple by minimizing the description length of the weights,” in Proc. 6th Annu. Conf. Comput. Learn. Theory (COLT), Santa Cruz, CA, USA, 1993, pp. 5–13, doi: 10.1145/168304.168306.DOI
  25. [26]M. Towers et al., “Gymnasium: A standard interface for reinforcement learning environments,” arXiv:2407.17032 [cs.LG], 2024.arxiv.org/abs/2407.17032
  26. [27]V. B. Mountcastle, “The columnar organization of the neocortex,” Brain, vol. 120, no. 4, pp. 701–722, 1997, doi: 10.1093/brain/120.4.701.DOI
  27. [28]D. Meunier, R. Lambiotte, and E. T. Bullmore, “Modular and hierarchically modular organization of brain networks,” Front. Neurosci., vol. 4, p. 200, 2010, doi: 10.3389/fnins.2010.00200.DOI
  28. [29]O. Sporns and R. F. Betzel, “Modular brain networks,” Annu. Rev. Psychol., vol. 67, pp. 613–640, 2016, doi: 10.1146/annurev-psych-122414-033634.DOI
  29. [30]W. Gerstner, M. Lehmann, V. Liakoni, D. Corneil, and J. Brea, “Eligibility traces and plasticity on behavioral time scales: Experimental support of neoHebbian three-factor learning rules,” Front. Neural Circuits, vol. 12, p. 53, 2018, doi: 10.3389/fncir.2018.00053.DOI
  30. [31]N. Frémaux and W. Gerstner, “Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules,” Front. Neural Circuits, vol. 9, p. 85, 2016, doi: 10.3389/fncir.2015.00085.DOI
  31. [32]W. Schultz, P. Dayan, and P. R. Montague, “A neural substrate of prediction and reward,” Science, vol. 275, no. 5306, pp. 1593–1599, 1997, doi: 10.1126/science.275.5306.1593.DOI
  32. [33]B. Engelhard et al., “Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons,” Nature, vol. 570, no. 7762, pp. 509–513, 2019, doi: 10.1038/s41586-019-1261-9.DOI
  33. [34]D. Attwell and S. B. Laughlin, “An energy budget for signaling in the grey matter of the brain,” J. Cereb. Blood Flow Metab., vol. 21, no. 10, pp. 1133–1145, 2001, doi: 10.1097/00004647-200110000-00001.DOI
  34. [35]H. B. Barlow, “Possible principles underlying the transformations of sensory messages,” in Sensory Communication, W. A. Rosenblith, Ed. Cambridge, MA, USA: MIT Press, 1961, pp. 217–234.
  35. [36]K. Friston, “The free-energy principle: A unified brain theory?” Nat. Rev. Neurosci., vol. 11, no. 2, pp. 127–138, 2010, doi: 10.1038/nrn2787.DOI
  36. [37]R. J. Douglas and K. A. C. Martin, “Neuronal circuits of the neocortex,” Annu. Rev. Neurosci., vol. 27, pp. 419–451, 2004, doi: 10.1146/annurev.neuro.27.070203.144152.DOI
  37. [38]P. Redgrave, T. J. Prescott, and K. Gurney, “The basal ganglia: A vertebrate solution to the selection problem?” Neuroscience, vol. 89, no. 4, pp. 1009–1023, 1999, doi: 10.1016/S0306-4522(98)00319-4.DOI
  38. [39]S. Chung and R. Kozma, “Reinforcement learning with feedback-modulated TD-STDP,” arXiv:2008.13044 [cs.LG], 2020.arxiv.org/abs/2008.13044
  39. [40]J. I. Yellott, “The relationship between Luce’s choice axiom, Thurstone’s theory of comparative judgment, and the double exponential distribution,” J. Math. Psychol., vol. 15, no. 2, pp. 109–144, 1977.
  40. [41]V. A. Traag, P. Van Dooren, and Y. Nesterov, “Narrow scope for resolution-limit-free community detection,” Phys. Rev. E, vol. 84, no. 1, p. 016114, Jul. 2011.
  41. [42]V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre, “Fast unfolding of communities in large networks,” J. Stat. Mech.: Theory Exp., vol. 2008, no. 10, p. P10008, Oct. 2008.
  42. [43]C. Curto, J. Geneson, and K. Morrison, “Fixed points of competitive threshold-linear networks,” arXiv:1804.00794, 2018.arxiv.org/abs/1804.00794
  43. [44]J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proc. Natl. Acad. Sci. USA, vol. 79, no. 8, pp. 2554–2558, 1982.
  44. [45]D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A learning algorithm for Boltzmann machines,” Cogn. Sci., vol. 9, no. 1, pp. 147–169, 1985.
  45. [46]E. P. Frady, S. J. Kent, B. A. Olshausen, and F. T. Sommer, “Resonator networks, 1: An efficient solution for factoring high-dimensional, distributed representations of data structures,” Neural Comput., vol. 32, no. 12, pp. 2311–2331, 2020.
  46. [47]T. Kohonen, “Self-organized formation of topologically correct feature maps,” Biol. Cybern., vol. 43, no. 1, pp. 59–69, 1982.
  47. [48]T. Kohonen, “The self-organizing map,” Proc. IEEE, vol. 78, no. 9, pp. 1464–1480, 1990.
  48. [49]C. Curto and K. Morrison. Relating network connectivity to dynamics: opportunities and challenges for theoretical neuroscience. Current Opinion in Neurobiology, 58:11–20, 2019. doi:10.1016/j.conb.2019.06.003.DOI
  49. [50]C. Curto, J. Geneson, and K. Morrison. Fixed points of competitive threshold-linear networks. Neural Computation, 31(9):94–155, 2019. doi:10.1162/neco_a_01151.DOI
  50. [51]C. Curto and K. Morrison. Graph rules for recurrent neural network dynamics. Notices of the American Mathematical Society, 70(4):536–551, 2023. doi:10.1090/noti2661.DOI
  51. [52]C. Curto, J. Geneson, and K. Morrison. Stable fixed points of combinatorial threshold-linear networks. Advances in Applied Mathematics, 154:102652, 2024. doi:10.1016/j.aam.2023.102652.DOI
  52. [53]C. Parmelee, S. Moore, K. Morrison, and C. Curto. Core motifs predict dynamic attractors in combinatorial threshold-linear networks. PLOS ONE, 17(3):e0264456, 2022. doi:10.1371/journal.pone.0264456.DOI
  53. [54]C. Parmelee, J. Londono Alvarez, C. Curto, and K. Morrison. Sequential attractors in combinatorial threshold-linear networks. SIAM Journal on Applied Dynamical Systems, 21(2):1597–1630, 2022. doi:10.1137/21M1445120.DOI
  54. [55]J. Londono Alvarez, K. Morrison, and C. Curto. Attractor-based models for sequences and pattern generation in neural circuits. Neural Computation, 38(3):257, 2026. Preprint: bioRxiv, doi:10.1101/2025.03.07.642121.DOI
  55. [56]D. Egas Santander, S. Ebli, A. Patania, N. Sanderson, F. Burtscher, K. Morrison, and C. Curto. Never theorems for fixed points of neural networks. In Research in Computational Topology 2, Association for Women in Mathematics Series, vol. 30. Springer, 2022. doi:10.1007/978-3-030-95519-9_6.
  56. [57]J. Londono Alvarez. Fixed point compositionality via low-rank gluing rules in inhibition-dominated threshold-linear networks. arXiv:2606.07336, 2026.arxiv.org/abs/2606.07336
  57. [59]C. Lienkaemper and G. K. Ocker. Diverse mean-field dynamics of clustered, inhibition-stabilized Hawkes networks via combinatorial threshold-linear networks. PRX Life, 2025. doi:10.1103/m8hf-q5pl. arXiv:2506.06234.DOI
  58. [60]R. Toro Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith. Reward machines: exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73:173–208, 2022. doi:10.1613/jair.1.12440.DOI
  59. [61]C. W. Omlin and C. L. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6):937–972, 1996. doi:10.1145/235809.235811.DOI
  60. [62]M. Rigotti, D. D. Ben Dayan Rubin, X.-J. Wang, and S. Fusi. Internal representation of task rules by recurrent dynamics: the importance of the diversity of neural responses. Frontiers in Computational Neuroscience, 4:24, 2010. doi:10.3389/fncom.2010.00024.DOI

Paper details

vlnc:W_588q5ae6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Sonnet 5.5, Opus 5.5, Claude Fable 5.1 (bare)

View statement
Looking for related papers…
Contents