WorksComputer Science
Recurrent Modular Structure in Spiking Neural Network Reinforcement Learning
Abstract
Biological neural circuits are both modular and recurrent, with dense communities sparsely linked to one another. Spiking neural networks (SNNs) for reinforcement learning are usually feed-forward, and it is unclear how to obtain functional modules in a recurrent SNN or what they buy. In this work, we introduce a class of spiking networks, called Community Recurrent SNNs (CoRe SNNs): recurrent networks partitioned into communities with fixed roles. A controller community receives the stimulus, each action has its own community, and latent communities carry internal state. Spiking activity is then a walk between communities, and the action is the community the walk reaches. All learning is local and three-factor, and the recurrent communities are closer to cortical microcircuits than a feed-forward actor. We compare three regimes for obtaining the communities: dynamic modularity, in which a new learning rule we introduce, Map-STDP, encourages modular structure during learning; partial modularity, in which a stochastic block model (SBM) initialization enforces it partly; and full modularity, in which sealing enforces it fully. On CartPole, across three spiking neuron models, Map-STDP improves learning over the same network without it matches full modularity (sealed communities) on two of the three models, and makes the observed spike walk more compressible. Replacing the feed-forward actor of a TD-STDP actor-critic with recurrent action communities improves on the feed-forward network on CartPole and LunarLander, and matches it on Acrobot. How strongly to enforce modularity depends on the task: full modularity helps most on Acrobot, while on LunarLander the dynamic, partial and full regimes perform alike. Overall, these recurrent modular networks can match or even improve upon feed-forward approaches, with improvements in bio-plausibility and energy efficiency.
Figure 1. Concept overview. (a) A feed-forward network passes information layer by layer and is usually trained by backpropagation. Each action is a single output neuron. (b) A community recurrent SNN (CoRe SNN) organises neurons into communities with distinct roles instead of layers. Each action is an entire community, and information flows through recurrent connections within and between communities. (c–e) Three ways to obtain the communities: (c) dynamic modularity, where Map-STDP encourages modular structure during learning, (d) partial modularity, where a stochastic block model (SBM) initialization imposes it partially, and (e) full modularity, where sealing imposes it fully. (f) The full model with CoRe inserted on a TD-STDP actor-critic backbone. An interactive visualization of the CoRe SNN variants is at https://vikramramavarapu.org/CoRe.
Index Terms—spiking neural networks, reinforcement learning, spike-timing-dependent plasticity, three-factor learning, modular networks, map equation, community detection, neuromorphic computing
Introduction
The cortex is organized into columns [27], and its wiring forms densely connected communities that are sparsely linked to one another [28], [29]. Modularity lets a circuit divide its work among parts that can be specialized and reused. Spiking neural networks (SNNs) are attractive models of such circuits and, on neuromorphic hardware, an energy-efficient way to run them [2], [3]. Yet SNNs for reinforcement learning (RL) are usually feed-forward stacks of layers, trained by backpropagation or by local reward-modulated rules [39]. It is unclear how a recurrent SNN could acquire functional modules using only local plasticity, or what those modules would buy.
A guiding principle comes from compression. Hinton and van Camp argued that a good network is a simple one and trained networks by minimizing the description length of their weights [25]. Efficient-coding theories make a similar claim for neural activity: neural systems should transmit information at low cost [35], and the same accuracy–complexity trade-off underlies the free-energy principle [36]. As such, the map equation gives the description length of a random walk on a graph split into communities [24], and it is shortest when the communities contain the walk. Spiking activity is a walk over synapses, so a network whose activity is compressible has activity confined within modules and sparse traffic between them.
This view recalls the self-organizing map (SOM) [47], [48], the classic example of structure that emerges from local, unsupervised learning. Both approaches replace a global error signal with a local rule whose collective effect is organization. They differ in what is organized and why. A SOM arranges a layer of prototype units on a fixed lattice so that it preserves the topology of the input. Our network organizes its own recurrent connectivity into communities with distinct roles, using spikes, a reward-modulated rule, and the description length of its activity as the organizing objective. We see this as a modern descendant of the SOM idea, brought to recurrent SNNs, reinforcement learning, and information-theoretic compressibility.
We introduce Community Recurrent SNNs (CoRe SNNs), a class of recurrent networks partitioned into communities with fixed roles: a controller community that receives the stimulus, one community per action, and latent communities that carry internal state. We make four contributions.
CoRe SNNs. Spiking activity is a walk between communities, and the action is the community the walk reaches.
Map-STDP. A local, three-factor learning rule that encourages the communities, and lowers description length during learning.
Modularity regimes. A comparison of dynamic, partial and full modularity across four neuron engines.
A drop-in actor. CoRe SNNs replace the feed-forward actor of a TD-STDP actor–critic and are evaluated on CartPole, Acrobot and LunarLander, together with a sweep of cross-module connectivity.
Background
Spiking Neural Networks
SNNs communicate through discrete spike events, enabling sparse, temporally-coded computation that maps naturally to neuromorphic hardware [1]. This makes them a closer model of cortical computation than conventional artificial neural networks. Because computation is event-driven, a neuron consumes energy mainly when it spikes or receives a spike, and can therefore run at a fraction of the energy of dense matrix multiplication on digital neuromorphic chips [2], [3] and on analog memristive crossbars. In a crossbar, each synapse is a conductance, and multiply–accumulate happens in place [4]. These savings depend on activity staying sparse [5]. Training SNNs is difficult because the spike function is non-differentiable. Surrogate-gradient backpropagation through time (BPTT) [6] works well but is non-local, needing weight transport and the full activity history, which have no known biological counterpart [7] and are hard to implement with the local plasticity that neuromorphic hardware supports, so such networks are typically trained offline and then deployed [3], [6].
Spike-timing-dependent plasticity (STDP) is a local alternative to BPTT. A synapse is potentiated when a presynaptic spike shortly precedes a postsynaptic spike and depressed when the order is reversed [8], [9]. The update depends only on quantities available at the synapse, and the pairing of pulses across a device maps directly onto memristive hardware [4]. STDP alone is unsupervised, however. STDP alone is unsupervised: it strengthens whatever correlations the input contains. In networks that respect Dale’s law, where each neuron is either excitatory or inhibitory, it can still learn useful features, for example digit prototypes shaped by lateral inhibition [10]. It cannot, however, tell behaviorally useful correlations from irrelevant ones, so on its own it cannot learn a task defined by reward.
Three-factor rules add a global modulatory signal to the two local factors of STDP [30], [31]. Rewards arrive seconds after the spikes that caused them, far outside the tens-of-milliseconds STDP window. Each synapse therefore keeps an eligibility trace . The trace accumulates pre–post coincidences and decays with a time constant matched to the delay of the outcome. A broadcast third factor then turns the trace into a weight change:
Chung and Kozma [39] introduced a three-factor rule in which the temporal-difference (TD) reward prediction error [32] is the third factor. In a spiking actor–critic network, this TD-modulated STDP was applied to standard control tasks such as CartPole and LunarLander.
Neuron Models
We use three spiking neuron models, which we call engines. In every engine, a spike of presynaptic neuron gives postsynaptic neuron the output-normalized kick
1) Branching Neuron Model: The branching model is the standard neuron model of the SuperNeuroMAT simulator [11] and follows the branching-process description of cortical activity [12]. The membrane forgets after every step, so one step is one hop of the walk:
where is the spike indicator, the threshold and the external (stimulus) drive.
2) Leaky Integrate-and-Fire (LIF): The LIF neuron [13] is driven by background noise as in Brunel [14]:
with a spike and reset to when reaches . Here is unit white noise standing for background input, and are the spike times of .
3) Quadratic Integrate-and-Fire (QIF): The QIF neuron [15] is written in the normalization of Montbrio et al. [16]:
with a spike at and reset to . Here is the excitability.
Random Walks, Information Theory, and Communities
A random walk on a weighted directed graph is a process in which a walker at node moves to node with probability proportional to the weight of the edge . It is a Markov chain with transition probabilities , and over time the fraction of steps spent at each node converges to a stationary distribution that measures how much flow each node carries. This is the idea behind PageRank [17].
In a graph with communities, a walker that has entered a densely connected group tends to stay there for many steps before crossing a sparse link, so the walk’s behavior reveals the community structure. In a spiking network, a natural walk follows spikes forward from each neuron to the neurons it excites.
Info-theoretic entropy can be measured by the description length of a random walk, following the core principle of information theory: the more predictable a process, the shorter its description. The map equation [19] measures the description length of a random walk on a graph partitioned into communities:
Each step of the walk is given a binary codeword, shorter for more frequent steps, and the entropy of a codebook is its average codeword length in bits. The map equation uses a two-level code. An index codebook names the module entered at each exit, which happens with probability per step. Each module has its own codebook for naming the nodes inside it, used at rate . Leaving a community costs extra bits, so the map equation is shorter for partitions that capture the graph’s community structure. The partition that minimizes is the one that best compresses the walk and reveals the communities, hence making the map equation the objective function of the community detection algorithm, Infomap [24]. The map equation has been used to detect communities in networks from social [19], biological [20], and to study the modularity of cortical networks [29].
A basic model that generates networks with community structure is the stochastic block model (SBM) [21]. In an SBM, each node belongs to one of communities, and the probability of an edge between two nodes depends on a matrix of connection probabilities. The SBM can generate networks with strong communities, weak communities, or no communities at all, depending on the choice of the probability matrix.
Other Graph-Theoretic Approaches to SNNs
Understanding how the structure of a network shapes its dynamics in the brain has been a long standing question in the neuroscience community [49]. For threshold-linear rate networks it has a concrete answer. A CTLN takes a directed graph, assigns the weights from it, and its fixed point supports, stable states and dynamic attractors are then determined by the graph [50, 51, 52, 53]. Brief external kicks switch the network from one attractor to the next, and this has been enough to assemble counters, sequences and gait controllers by hand [54, 55]. Our SNN work sits next to this. We ask the same thing of a spiking network, and in both settings the answer takes the same shape, a collection of attractors pinned down by connectivity plus a set of input-triggered jumps between them, which is also how a finite state machine lives inside a recurrent network [61].
Part of the CTLN picture has been shown to survive the move to spikes, but only in a restricted setting. In clustered E-I networks of Hawkes neurons the excitatory rates obey a CTLN in the mean-field limit, and the CTLN fixed points become metastable states of the finite network [59]. CTLNs also come with a modular story, where the attractors of a graph assembled from smaller pieces are inherited from the pieces [56, 57], though no such result yet exists for spiking networks. Whether either property carries over to the SNNs we study is open. If it does, it would suggest a use for these rules in learning. A reward machine turns a history-dependent RL task into one with a finite set of states [60], and if those states could be planted as the attractors of a glued graph, a spiking network started from that graph would have the states in place and would only need to learn the jumps [62]. Some known initialisations of structure are detailed below.
Columnar organization has also been used for supervised learning in spiking networks. CoLaNET [22] gives each target class its own column, built from several layers of neurons and sub-class microcolumns, and trains it with local anti-Hebbian and dopamine-modulated plasticity. It was evaluated on classifying MNIST images [23] and Pong states by their proximity to reward, a task derived from model-based reinforcement learning. Its signal path is feed-forward, with only local winner-take-all competition and gating inside each column. The columns are fixed by design as class-specific classifiers, and the network evaluates states without selecting actions. In contrast, our work learns recurrent modular structure with Map-STDP and studies the effect of modularity on reinforcement learning.
Map-STDP Learning Rule
Community detection solves a forward problem: given a graph, find the partition into communities that minimizes the description length of a random walk on it, the map equation [24]. Map-STDP solves the inverse problem. The partition is fixed in advance (one controller module, one module per action, and a few latent modules), and plasticity shapes the synapses until that partition is the minimum-description-length one. This section states the rule and what each part does. The derivation from the map equation is in Appendix B.
Ingredients
Each synapse from neuron to neuron has a magnitude and the sign of its presynaptic type, . Each neuron belongs to one module : the controller , an action module in , or a latent module in .
The walk. A step of the walk from to is a spike of within a short window after a spike of . Its flow , coarse-grained to modules, is a small Markov chain over the network’s own states. The policy is the flow into the action modules, , where is the stimulus at frame , is the flow into action module in that frame, and is a temperature parameter.
Description length. is the code length of the walk averaged over frames. Leaving module costs bits per unit of exit flow . It is computed from module-level statistics and broadcast to module : it is the rule’s neuromodulator.
Signed counter. is the change in ’s spikes, within the window, caused by ’s kicks alone. It is the local pre-synaptic activity times post-synaptic sensitivity, and carries .
The rule
The per-frame update of each synapse is
The rule has three terms:
Task. is a global TD error and an action-gated eligibility trace built from (Eq. (24)).
Structure (the map term). is the share of ’s observed transitions that leave its module. Transitions that leave the module are pushed down when the module leaks, and those that stay are pushed up, scaled by .
Homeostasis. is ’s total output weight, held near . It rescales a column uniformly, so it does not change the walk.
sets the strength of the map term. In partial and full modularity, and the map term is not applied. In dynamic modularity, and the map term is applied. Inhibitory synapses reverse the signs of the structural term, through in . On average the structural term is gradient descent on , up to a rescaling that cannot change (Appendix B-§ 16). The controller’s synapses onto the action and latent modules carry the policy, so they are exempt from the map term (Appendix B-§ 16.1).
Three regimes of modularity
The experiments compare three regimes:
partial modularity: on a network initialized with SBM connectivity (), so communities are only partly enforced and the task term alone shapes the structure;
full modularity: on a network built without cross-module synapses (, scaled);
dynamic modularity: the complete rule (6), which encourages the communities during learning.
Model Architecture
Standalone Map-STDP Agent
The controller has no recurrent synapses of its own. In each environment frame the network is simulated on the current weights:
100 steps for the branching engine;
100 ms for the LIF and QIF engines;
20 hops of power iteration for the flow engine, which has no spikes.
The spike raster gives the module flow (spike shares), the pair counts and the signed counters. Each spiking agent runs 8 parallel heads on shared weights and pools their statistics.
TD-STDP Actor-Critic Backbone
The backbone is the TD-STDP actor–critic of Chung and Kozma. Its neurons are noisy LIF ( ms) on a 1 ms grid, and each environment step is a 20 ms window. Population sizes follow the authors’ configurations on CartPole, and in Acrobot the critic has 40 excitatory neurons and the actor has 20 per action; on LunarLander each excitatory group is paired with an inhibitory group of equal size (128 critic, 32 per action). CoRe replaces the feed-forward actor with recurrent communities, one per action, among the actor’s excitatory neurons. Actions are read out by a race between the action communities’ excitatory spike counts instead of the original softmax.
SBM Initialization
We initialize the network with an SBM with communities. The intra-community connection probability, is consistently set to 0.5. In the sealed regime, the inter-community connection probability, with all-to-all connections only between the controller and the other communities. In most experiments, the partial regime has , but in the inter-module connectivity study, we vary from 0 (full modularity) to 0.5 (no communities).
Implementation Details
Models are implemented in NumPy, with Numba kernels for the LIF and QIF engines. The branching engine uses SuperNeuroMAT’s [11] branching mode (infinite leak, uniform threshold noise), so each simulation step is one hop of the walk. Environments come from Gymnasium [26].
Training
a) Standalone agent.: Agents are trained on CartPole for 1500 episodes, with 20 seeds per condition. Each condition is one engine (flow, branching, LIF, QIF) in one of three regimes: dynamic (), partial (), or full (sealed: , no cross-module synapses). Map-STDP and the task update are applied once per frame. All conditions use TD-error normalization and an adaptive readout sharpness. In the spiking engines, and the task rate are rescaled to match the flow engine’s update size at initialization.
b) Backbone.: The TD-STDP and Map-STDP updates are applied once per 20 ms window, without Adam. Input scales are calibrated so that every arm starts at the same 40 Hz rate. The arms are the original feed-forward network, the feed-forward network with the race readout, and CoRe SNNs in the partial, dynamic and full regimes. Each arm has 20 seeds. Per task:
CartPole: 400 episodes, one environment copy, race bound 16.
Acrobot: 500 episodes, one environment copy, race bound 16, one action every 40 ms, plus a firing-rate cost and floor on the actor.
LunarLander: 10,000 episodes pooled over 16 environment copies on shared weights, race bound 26, one action every 40 ms, plus the same rate cost and floor.
Results
Minimal spiking policy network
1) Results on CartPole: Figure 2 summarizes the comparison. In panel (a) of the figure, the trained networks of one seed are shown for each regime of the flow engine with a central controller, two latent modules, and two action modules. In the dynamic and full regimes, the modules are clearly apparent, indicating that qualitatively, on this engine, the map term is effective in shaping the network into a modular structure. In the partial regime, the modules are less distinct, indicating that the SBM initialization alone leaves room for inter-module connectivity. In panel (b), the mean return over training is shown for each engine and regime. The dynamic and full regimes outperform the partial regime across all engines, however overall, flow outperforms the other engines, albeit with a larger spread. LIF comes next, followed by QIF and branching. This may be due to the fact that in the base CoRe architecture, the random walk entropy formulation is most closely aligned with the flow engine, and spiking engines approximate walk behavior. In panel (c), the effective synaptic operations per decision (spikes outgoing synapses above the weight floor, averaged over 50 held-out stimuli with W frozen) is plotted against , the held-out description length of the observed walk. As there is a direct exponential correlation, this may hint at the fact that the more compressible the walk, the fewer synaptic operations are needed to make a decision, and therefore the more energy efficient the network is. In panel (d), is shown for each engine and regime at episode 1500. The dynamic and full regimes have lower than the partial arm, indicating that the modularity is effective in shaping the network into a more compressible structure. Compressibility largely follows from modularity itself: the full regime, which has no map term, has a lower than the partial regime on all four engines, and on flow and branching it is within 0.2 bits of the dynamic regime (Table II). On QIF and LIF the map term lowers further. Overall, these results suggest that the map term is effective in shaping the network into a modular structure that is more compressible and energy efficient, and that this leads to improved performance on the CartPole task.
Figure 2. (a) Trained CartPole networks on the flow engine, one seed (2000), from the same initial network. C is the controller, L1–L2 the latent modules and A1–A2 the action modules. Edges of at least 2% of the maximum weight are drawn, with width and opacity proportional to weight. Below each network: the share of weight within modules and across modules (excluding controller routing), and , the map equation (bits) of W’s own walk. (b) AUC, the mean return over 1500 training episodes, for each engine and regime; 20 seeds per condition. (c) Effective synaptic operations per decision (spikes × outgoing synapses above the weight floor, averaged over 50 held-out stimuli with W frozen) against , for the final networks of the three spiking engines. Color: engine; shape: regime. Dashed lines are least-squares fits of effective SOPs on ; is Spearman’s. Post-hoc, exploratory analysis. (d) , the held-out description length (bits) of the engine’s observed walk, at episode 1500. In (b) and (d), boxes show the median and interquartile range, whiskers the full range, and dots individual runs.
| Engine | Regime | AUC | Return | Frozen | (bits) | Solved | ||
| Flow | Dynamic | 15/20 | – | – | ||||
| Partial | 20/20 | 0.17 | ||||||
| Full | 20/20 | 1.0 | ||||||
| Branching | Dynamic | 0/20 | – | – | ||||
| Partial | 0/20 | |||||||
| Full | 3/20 | 1.0 | ||||||
| QIF | Dynamic | 0/20 | – | – | ||||
| Partial | 0/20 | |||||||
| Full | 0/20 | 0.0023 | ||||||
| LIF | Dynamic | 0/20 | – | – | ||||
| Partial | 0/20 | |||||||
| Full | 0/20 | 0.6 |
Table II. CartPole, model v4 with the observed-walk rule: dynamic (), partial () and full (sealed: no cross-module synapses, ) modularity, on four engines. Mean S.D. over 20 seeds (2000–2019), 1500 episodes per run. AUC: mean return over all episodes. Return: mean of the last 100 episodes. Frozen: mean return of 20 episodes with frozen at the end. }: held-out description length (bits) of the engine’s observed walk at episode 1500. Solved: runs whose trailing-100 mean reaches 475. : two-sided Mann–Whitney, Holm-adjusted over the four engines, of each arm against the MAP-term-on arm of the same engine.
2) Latent Ablation: To find out what the latent modules contribute, we ablated them in the final CartPole models (episode 1500) of every engine and regime (240 models in all) and measured the change in return with plasticity off. The design was pre-registered. Each network has a 36-neuron controller , two latent modules and two 16-neuron action modules . Because every engine normalizes outgoing weights, with , a cut either loses the drive the cut synapses carried or reroutes it, so we test both cases. A lesion cuts the latents’ outgoing synapses and leaves every other neuron’s unchanged, so the drive into the latents is lost. It tests whether the latents relay anything the policy uses. A removal disconnects the latents entirely, and each remaining neuron’s outflow renormalizes onto its other targets, with . It tests whether the latents act as a sink that sets the action modules’ share of the controller’s drive. In the latent path only condition the direct controller-to-action synapses are cut, which tests whether the latents alone can carry the policy. Each model is evaluated intact and under each ablation for 50 episodes with common random numbers, so every model is its own control. QIF’s warm state is not saved, so its intact returns differ slightly from training.
The policy is carried by the direct routing, not by the latents. With it cut, every group fell to about random play (returns of 10–32 against 22.8 for a random policy; Table I, last column). Removing the latents lowered return most in flow (−206 to −287), branching in the dynamic regime (−118) and LIF in the dynamic and full regimes (−112, −80), and by −31 to −65 in the other groups. Without the latents the controller’s whole outflow reaches the action modules, and their share of the drive rose from 0.08–0.49 to 0.57–0.80. The latents therefore act as a sink that sets how much of the controller’s drive reaches the actions, and the trained policy depends on that setting.
Their output is not a relay the policy needs. In full-modularity models the lesion left returns identical to intact, since they have no latent-to-action synapses, and in the dynamic regime it changed almost nothing on QIF and LIF. Where the latents still drive the actions, cutting their output improved return (branching dynamic +47, LIF partial +38), so it acted as interference. Flow in the dynamic regime is
the exception: lesioning one latent module cost return (: −117, : −77), but lesioning both did not (−3), which suggests that the two modules’ small outputs balance each other. Differences between regimes were small. Only LIF (partial against dynamic) and branching (full against dynamic) differed, and the remaining contrasts did not.
When proceeding with the TD-STDP backbone, we test models without latents.
TD-STDP actor-critic backbone
Performance
In the previous subsection, we showed that the minimal spiking policy network can be significantly improved with CoRe. However, in order to show that advancements can be made with this abstraction, we now demonstrate how CoRe can be applied to an existing TD-STDP actor-critic backbone to improve its performance. Figure 3 shows the mean return over training episodes for each environment and arm. The results indicate that when CoRe is applied to the TD-STDP backbone, the performance of the network improves cleanly in CartPole and LunarLander. However, the results on Acrobot are a bit more complicated, when the race readout first replaced the softmax, the performance dropped, but when CoRe was applied, the performance improved and slightly exceeded the original softmax performance in the full regime. Overall, these results suggest that CoRe can be effectively applied to existing TD-STDP actor-critic backbones to improve their performance on reinforcement learning tasks.
Figure 3. TD-STDP actor-critic backbone. The actor network generates actions based on the current state, while the critic network evaluates the actions. The TD-STDP learning rule updates the weights based on the prediction error and the eligibility trace. In panels (a-c), the mean return over training episodes is shown for each environment and arm; 20 seeds per arm. Lunar Lander shows mean return for the last 100 episodes.
| Engine | Regime | Intact | path only | ||||
| Flow | dynamic | 462 | −3 | −77 | −117 | −206 | −455 |
| partial | 414 | −18 | +1 | 0 | −287 | −418 | |
| full | 481 | −254 | −478 | ||||
| Branching | dynamic | 199 | +47 | +18 | +16 | −118 | −167 |
| partial | 124 | +18 | +12 | +8 | −65 | −105 | |
| full | 178 | −41 | −137 | ||||
| QIF | dynamic | 150 | 0 | +1 | +1 | −31 | −140 |
| partial | 96 | +13 | +6 | +13 | −50 | −87 | |
| full | 192 | −32 | −179 | ||||
| LIF | dynamic | 327 | +2 | +1 | −2 | −112 | −311 |
| partial | 199 | +38 | +18 | +17 | −42 | −181 | |
| full | 305 | −80 | −281 |
Table 1. CHANGE IN RETURN FROM THE INTACT BASELINE WHEN THE LATENTS ARE ABLATED (MEDIAN OVER THE 20 MODELS IN EACH ROW). IDENTICAL TO INTACT ON EVERY MODEL. INTACT: MEAN RETURN OF THE INTACT MODELS.
2) Inter-module Connectivity Effects: In the previous sections, we have explored comparing the three regimes of modularity: partial (SBM with ), full (sealed, ), and dynamic (learned with the map term). However, it has been left open how much cross-module connectivity is optimal for performance, and what happens when is varied. To address this, we conducted a sweep of values from 0 (sealed) to 0.5 (no communities) in the TD-STDP actor-critic backbone, with both the map term on and off. When , while the map-term is constrained to act upon intra-module connectivity, this still yields a difference in performance compared to when the map term is off.
The results, shown in Figure 4, indicate that optimal inter-module connectivity is task-dependent, and setting-dependent. Across all three environments, the most optimal lies in its lower range. In CartPole, the performance is highest at with the map term off. In Acrobot, the performance is highest at with the map term on. In LunarLander, the performance is highest when with the map term on. Overall, this suggests that in some tasks, actions may be dependent on each other, and therefore some cross-module connectivity is beneficial, while in other tasks, actions may be independent of each other, and therefore cross-module connectivity is detrimental.
Figure 4. Inter-module connectivity effects in the TD-STDP actor-critic backbone. The figure illustrates how the connectivity between different modules influences the performance and learning dynamics of the network. The Spearman coefficients are shown to the right of each plot, indicating the strength and direction of the relationship between inter-module connectivity and performance.
The trends in the results are also task and setting dependent. While most results show a downward trend in performance as increases, there are several exceptions and nuances. Many of the correlation coefficients are brought down by non-monotonicity in the trends, with many trends following a drop off, then a rise, then a drop again, and sometimes another rise. This is especially apparent with CartPole with the map term on, and LunarLander with the map term off.
Overall, these results suggest that the optimal inter-module connectivity is task-dependent, and setting-dependent, and that the map term can have a significant impact on performance. Future work should explore dynamically setting the inter-module connectivity matrix, rather than a single value, to allow for more flexibility in the network’s structure and performance.
Discussion: Biological Plausibility
CoRe SNNs share two features of cortex. First, as motivated in the Introduction, their connectivity is modular: Map-STDP learns weights that are dense within communities and sparse between them, and partial and full modularity impose the same structure directly. Second, all CoRe models learn with three-factor rules, in which a local eligibility changes a weight only when a neuromodulatory signal arrives. Such rules have direct experimental support [30], [31], our task term follows the dopaminergic temporal-difference account of reward learning [32], and the structural modulator is a single scalar per module, in line with the regional specificity of neuromodulation [33]. Two parts remain abstractions. The structural modulator is indexed by the presynaptic module, which requires a roughly uniform or projection-specific signal, and the per-neuron baseline has no established biological counterpart.
Map-STDP also fits the efficient-coding view of the Introduction. Communication accounts for much of the brain’s energy budget [34], and compressing the network’s activity prunes cross-module traffic. In exploratory measurements, a lower description length coincided with fewer effective synaptic operations per decision (Spearman – across engines). The task term keeps the network useful for acting, and the structural term keeps its activity compressible. Finally, our TD-STDP actor-critic [39] is more biologically plausible than the original. Its actors are recurrent communities, closer to cortical microcircuits and to the channel organization of the basal ganglia [37], [38], it runs in a noisy leaky integrate-and-fire regime at lower firing rates, and it selects actions by a race between action communities, which is equivalent to softmax sampling under constant hazards [40].
Limitations and Future Work
Limitations. Map-STDP relies on a random-walk abstraction of inference. Random walks, and branching processes in particular, are inherently sequential, so estimating the walk statistics per frame becomes a bottleneck as networks scale; a more parallelizable abstraction of inference is needed. Second, the map equation is only one principle that can drive module formation. Alternatives such as the Constant Potts Model [41], modularity maximization [42], attractor models [43], and the dynamic SBM matrices discussed earlier remain unexplored here. Our results suggest that such alternatives are worth testing, because compressibility largely accompanied modularity itself rather than requiring an explicit information-theoretic objective. Objectives that reward modularity directly could therefore yield similar structure without the map equation. Third, as Table I shows, the latent communities act largely as sinks and, at best, routers; we do not yet train them to perform useful computation of their own.
CoRe as a starting point. We view CoRe as a base architecture that opens many directions. Its modular structure decomposes functionality into smaller, separately trainable pieces, and the TD-STDP backbone experiment shows that a CoRe module can be inserted into and coupled with existing, performant architectures and improve them. This opens several directions. Latents could be trained to carry out subtasks or store memory, through persistent activity across frames or through coupling with memory-oriented architectures such as Hopfield networks [44] or Boltzmann machines [45]. Coupling with architectures built for binding and unbinding [46] in the controller could let different latents attend to different objects in the environment. Finally, the same modular interface could host other backbones and other module-forming objectives, so that CoRe serves as a common substrate for these extensions.
Acknowledgment
We thank Dr. Ramakrishnan (Ramki) Kannan at Oak Ridge National Laboratory for providing access to his local cluster. We also thank the AI Interpretability @ Illinois group for providing the collaborative network to make this work possible. The authors used Claude Code (Anthropic) with the Claude Sonnet 5.5, Opus 5.5, and Fable 5.1 models during this work. The AI generated software and experimental scripts used for the results in Section VII; assisted with the derivation in Appendix B; assisted in the iterative design of the figures; and provided language editing throughout the manuscript. All code was reviewed, tested, and validated by the authors against the source data; all derivations were independently checked by the authors; and all text was reviewed and revised by the authors. The authors take full responsibility for the content of this paper, including the integrity of the analysis and results.
Full Results
Derivation of MAP-STDP
This appendix derives the rule in § 5. All logarithms are base 2 (description lengths in bits).
Setup and notation
Each synapse from presynaptic neuron to postsynaptic neuron has a magnitude . The type of fixes its sign, (excitatory) or (inhibitory), as in Dale’s law. We write
for ’s total output weight and the share of it that goes to . Each neuron belongs to one module . The module types are the controller , the action modules (one per action) and the latent modules . Environment frames are indexed by , and the stimulus of frame is .
is column-stochastic: column says where ’s output goes. A random walk that follows spikes forward moves from to with probability .
The observed walk
A step of the walk from to is a spike of within a short window after a spike of . The expected number of such spikes splits into three parts:
Here is ’s firing rate. is the direct effect of ’s kick on , where is ’s windowed sensitivity (gain times kick size). collects chains and common input. The observed transition probabilities and link flows are
where is ’s share of all spikes in the frame.
a) Intuition. is the mean of a count, so it is non-negative whatever the signs of the synapses. Inhibition does not create negative transitions; it pushes below chance.
b) Pair counts measure the walk.: With , where marks a spike of at step and counts spikes of , stationarity within the frame gives . So
is the share of ’s observed transitions that leave its module. A neuron can estimate it from its own spike times and two population signals: activity in its own module, and in the whole network.
c) The flow engine is the special case.: Without spikes there is no chance () and no indirect term (), and all gains are equal, so . The stimulus re-injects a share of the activity at each step:
For the solution is unique. Starting from , power-iteration hops give , whatever the network size.
Communities as states and the policy
Coarse-graining the frame’s walk to modules gives a -state Markov chain, with tables indexed [post, pre]:
Because the stimulus is concentrated in the controller, the controller’s column of is the stimulus-dependent one. The action is read from the flow into the action modules,
with the plain readout. Model v4 adapts to the trend of episode return. In spiking engines the readout is a race between the action modules’ spikes, which samples (13).
The map equation
Describe the walk with a two-level code. Inside a module, name the next neuron, or “exit”, with a module-local codeword. On exit, name the next module from a global index. The average code length per step is [24]
where is module ’s exit flow and . With held fixed, the weights enter only through the exit flows, and
a) Intuition.: Leakage out of a module always costs bits. is the marginal number of bits per unit of exit flow from . It is computed from module-level statistics and broadcast to module : this is the rule’s neuromodulator.
b) The inverse problem.: Map-STDP pins the partition and minimizes over the weights, averaged over the frames the environment produces:
c) Check.: Differentiate (14) term by term. The derivative of is for a constant , and the constants cancel (). The result is . It is non-negative because and $p_m+q_m\ge q_m.
The structural rule
Let be the signed counter: the change in ’s spikes within the window caused by ’s kicks alone, with . The structural update is
For the map equation, is a sum of the with and , so
and the rule becomes
a) Two statistics, two jobs.: The baseline describes the walk, so it comes from pair counts, chance included. The step must say how the walk changes when changes, which is the direct effect, so it comes from the signed counter. Using pair counts for the step fails: they are mostly chance, and renormalization cancels them.
b) Sign pattern.: The factor is positive across modules and negative within. The counter carries . Hence:
| Synapse | Across modules | Within a module |
| Excitatory | weakened | strengthened |
| Inhibitory | strengthened | weakened |
Table 1 (PDF p. 13).
Why the rule descends the description length
a) Local split.: While differentiating with respect to one weight, hold fixed everything a single synapse does not directly touch: the flow , chance , the indirect term and the gains. Then changes only column ’s direct part, (through itself and through ).
Lemma 1 (Lemma E). If depends on only through the link flows , then under the local split
Proof sketch. Under the split, , and . By the quotient rule,
Summing against , and using ,
Multiplying by , with and , gives (20).
b) Intuition.: is the same for every target of , so rescales column uniformly. That cannot change , so it cannot change ; hence . The other factors read as follows:
is a per-neuron step size;
flips the direction for inhibitory synapses;
scales the step by how much actually moves ;
compares this transition’s marginal cost with the average over ’s observed transitions.
Theorem 1 (Theorem E). If the counter is unbiased, over a frame of duration , then
with .
Proof sketch. Solve (20) for , substitute it into , and use . To first order, . The column-rescaling term contributes .
c) Intuition.: On average the rule is gradient descent on , preconditioned by the positive diagonal factor (which never has to be computed), plus a column rescaling that cannot change .
d) Flow-engine case.: With , excitation only and equal gains, and , and Lemma E reduces to
The flow engine replaces the counter by its expectation , giving : gradient descent with each coordinate scaled by its own weight.
Routing exemption
The policy is carried by cross-module flow: controller action and controller latent. Applied to those synapses, the map term would drive the controller’s exit flow to zero and starve the action modules. The routing pairs are therefore exempt: for and . The controller then acts as an input layer, and the map term seals the latent and action modules.
Task term
The task is learned by a TD-modulated, action-gated eligibility. Take the surrogate policy , with . Its score with respect to depends only on the module of :
where is the chosen action. The eligibility trace and the update are
with the TD error and a linear critic trained by normalized LMS:
In the flow engine, by the same Lemma, the increment of (24) is : a one-hop policy-gradient score.
Homeostasis
A presynaptic homeostasis term holds each neuron’s output weight near 1:
It rescales column uniformly, so it leaves , the walk and unchanged. The full per-frame update, Eq. (6), is the sum of the task, structure and homeostasis terms.
Computing the counter in each engine
Each engine uses the same rule. What differs is how is computed.
a) Branching.: The neuron is , which spikes with probability . The kick is output-normalized. Removing ’s kick changes ’s firing probability by a known amount, so the counter is exact:
b) LIF and QIF. A kick of size is an input pulse of area . In linear response it changes ’s spike count by , where is ’s gain (the slope of its rate transfer). So
an outer product of the presynaptic spike count and the postsynaptic gain. For LIF the gain comes from the Siegert rate formula, or from the exact Markov transfer of the discretized neuron. For QIF it comes from regressing the rate on the injected noise.
Stability
The walk always exists, but the counters rely on linear response around a stable operating point. With the signed offspring matrix (extra spikes of per spike of ), the conditions are
The network monitors them through a module-level offspring table estimated from the counters.
Map-STDP communities in the TD-STDP backbone
In the hybrid model, the actor of the TD-STDP actor-critic of Chung and Kozma [39] is organized into modules:
Controller. The controller is the Fourier input population.
Action communities. Each action’s excitatory actor population is one module.
Recurrence. Recurrent synapses connect all actor excitatory neurons, with output-normalized kicks .
Routing. The input actor weights are exempt routing, learned by the TD-STDP rule alone.
Where the map term acts. The structural term acts on only, with the LIF counter accumulated per window.
The action is sampled from a softmax over community rates, . This is equivalent to a race of escape-noise decision units with constant hazards , in which the first unit to spike wins [40]. Each unit’s local score , with its integrated hazard up to the decision time , has expectation given the winner, where is the one-hot chosen action. So the community credit of the backbone is available locally. The task term on uses the one-hop credit of (24).
Random Walk Entropy and SNN Energy Usage
In this section we discuss in more depth how entropy of a random walk abstraction of SNN inference relates to the energy usage of the network (Figure 5 and Table III).
Figure 5. Energy use and description length across the inter-module connectivity sweep in the TD-STDP actor-critic backbone, for (a) CartPole, (b) Acrobot and (c) LunarLander. Each point is one final network ( values of map-term settings seeds, ); color gives and shape the map-term setting (dynamic with the term on, partial with it off, full at with it off). Columns 1–3: effective synaptic operations (SOPs) per decision against and , and against , with the Spearman over all points above each panel. Column 4: Spearman within each value ( per setting), with bootstrap intervals.
| Group | SOPs vs | SOPs vs | vs | Return vs | Return vs | |
| Map-STDP v4, CartPole | ||||||
| Branching | 60 | +0.61* [+0.35, +0.77] | +0.71* [+0.52, +0.81] | +0.49* [+0.21, +0.67] | −0.49* [−0.70, −0.24] | −0.19 [−0.44, +0.08] |
| QIF | 60 | +0.95* [+0.90, +0.97] | +0.93* [+0.87, +0.96] | +0.92* [+0.86, +0.95] | −0.09 [−0.35, +0.18] | −0.11 [−0.37, +0.17] |
| LIF | 60 | +0.91* [+0.86, +0.94] | +0.91* [+0.86, +0.93] | +0.92* [+0.86, +0.95] | −0.50* [−0.68, −0.28] | −0.45* [−0.64, −0.23] |
| TD-STDP, CartPole | ||||||
| All regimes | 60 | +0.88* [+0.78, +0.94] | +0.72* [+0.53, +0.84] | +0.71* [+0.52, +0.84] | — | — |
| Dynamic | 20 | +0.83* [+0.56, +0.95] | +0.55* [+0.11, +0.85] | +0.66* [+0.20, +0.93] | — | — |
| Partial | 20 | +0.65* [+0.24, +0.87] | −0.23 [−0.61, +0.21] | −0.29 [−0.73, +0.24] | — | — |
| Full | 20 | +0.65* [+0.24, +0.88] | +0.25 [−0.22, +0.65] | +0.02 [−0.52, +0.55] | — | — |
| TD-STDP, Acrobot | ||||||
| All regimes | 60 | +0.02 [−0.24, +0.28] | +0.38* [+0.11, +0.63] | +0.07 [−0.19, +0.31] | −0.04 [−0.33, +0.24] | −0.10 [−0.34, +0.14] |
| Dynamic | 20 | +0.32 [−0.21, +0.77] | +0.46* [+0.03, +0.80] | +0.61* [+0.24, +0.82] | +0.02 [−0.44, +0.45] | +0.26 [−0.24, +0.66] |
| Partial | 20 | +0.18 [−0.32, +0.61] | −0.15 [−0.59, +0.41] | −0.64* [−0.87, −0.21] | +0.13 [−0.36, +0.59] | −0.30 [−0.71, +0.20] |
| Full | 20 | −0.13 [−0.59, +0.40] | +0.18 [−0.34, +0.60] | +0.06 [−0.41, +0.51] | −0.39 [−0.75, +0.08] | −0.02 [−0.52, +0.41] |
| TD-STDP, LunarLander | ||||||
| All regimes | 60 | +0.49* [+0.25, +0.69] | −0.47* [−0.69, −0.17] | −0.01 [−0.30, +0.30] | +0.19 [−0.09, +0.43] | −0.12 [−0.37, +0.16] |
| Dynamic | 20 | +0.11 [−0.44, +0.55] | +0.29 [−0.14, +0.65] | +0.26 [−0.27, +0.71] | +0.07 [−0.50, +0.54] | −0.09 [−0.52, +0.41] |
| Partial | 20 | +0.58* [+0.13, +0.79] | −0.69* [−0.87, −0.35] | −0.39 [−0.72, +0.12] | +0.65* [+0.22, +0.88] | −0.79* [−0.93, −0.50] |
| Full | 20 | +0.26 [−0.24, +0.69] | +0.49* [+0.03, +0.79] | −0.14 [−0.60, +0.37] | +0.18 [−0.30, +0.60] | −0.08 [−0.54, +0.41] |
| All settings | 360 | +0.64* [+0.56, +0.71] | +0.53* [+0.45, +0.60] | +0.52* [+0.43, +0.59] | −0.19* [−0.30, −0.07] | −0.19* [−0.30, −0.08] |
Table III. Spearman correlation between energy use, random-walk description length and performance of the final networks: effective synaptic operations (SOPs) per decision, the map equation of the observed walk () and of the walk on the trained weights (), and the mean return of the frozen final network. with its bootstrap 95% interval (2000 resamples over seeds) below; * . Map-STDP v4: per spiking engine, pooled over the three modularity regimes. TD-STDP: all regimes pooled, then within each regime. All settings: each variable ranked within its setting (three v4 engines, three TD-STDP tasks) and pooled; bootstrap stratified by setting. —: Every frozen TD-STDP CartPole network reaches the 500 ceiling, so its return is constant; that setting is left out of the all-settings return columns ().
Energy Usage
In this supplementary study, we measure energy usage through spiking operations (SOPs). An SOP is one spike delivered across one synapse. Each spike is weighted by the number of outgoing synapses of the neuron that fired it:
Here is the spike count of neuron . The count only includes synapses above twice the weight floor (), the “effective” synapses. Synapses the rule has pushed down to the floor are treated as pruned, so an SOP is only counted where a synapse effectively exists. SOPs are computed on frozen weight models without any continual learning.
and
There are two measurements of info-theoretic entropy in the random-walk abstraction: the map equation of the observed walk () and the map equation of the walk on the trained weights (). Both description lengths are the map equation of a walk with visit rates and transitions over the modules (see Equation (5)). They differ in the walk they use:
is computed from the weights alone (the recurrent weights for TD-STDP) under uniform teleportation. is computed from spikes and averaged over frames . In it, counts spikes of within the engine’s window after a spike of , and is the spike count of neuron in the frame.
Info-theoretic entropy and downstream energy use
No experiment was run on the flow model as there are no spiking operations in the flow model. The full correlation table can be found in Table III. Overall, there is general agreement between energy usage and description length (Spearman correlations: +0.64 overall SOPs vs , +0.53 overall SOPs vs ), showing strong evidence for lower description length yielding lower energy usage. Moreover, there is even a weak negative correlation between performance and the two description length quantities (-0.19 Return vs and vs ), providing a small amount of evidence for Hinton’s compressibility argument. However, the agreement between info-theoretic entropy via description length and downstream energy usage, as well as the two measures of description length, depends on the model and the task.
The correlation between SOPs and becomes weak and even goes slightly negative in the full modularity regime in the TD-STDP backbone on Acrobot (-0.13 Spearman correlation). The correlation between SOPs and , and between and drops heavily on LunarLander, indicating that preemptive info-theoretic analysis holds weaker on this task.
In Figure 5, showing the correlation between SOPs and the two description lengths across values, we see a similar trend. There is a positive correlation between SOPs and across all three tasks (CartPole: +0.93, Acrobot: +0.43, LunarLander: +0.41). The correlation between all three quantities are strongest in CartPole (SOPs vs : +0.93, SOPs vs : +0.83, vs : +0.78). And LunarLander holds the weakest correlation, with theoretical description length analysis even serving counterintuitively (SOPs vs : +0.41, SOPs vs : -0.43, vs : -0.24). Across the values, in the plots on the rightmost column, correlations hold positive consistently in CartPole and Acrobot, and while SOPs and hold positive, the correlations with drop negative consistently. This opens the question of what attributes the LunarLander task has for preemptive info-theoretic analysis to hold so counterintuitively. This question is explored in the following subsection.
The Special Case of LunarLander
Synaptic operations (SOPs) here track the number of surviving synapses, not activity. Within each task, column spikes per decision barely change across regimes: about 5 in CartPole, 92 in Acrobot and 211–223 in LunarLander, since the rate-matching settings of these studies hold firing rates fixed. SOPs therefore vary only through the effective fan-out, the number of a neuron’s recurrent synapses still above threshold (Table IV).
| Task | Dynamic | Full | Partial |
| CartPole | 3.3 | 9.5 | 9.5 |
| Acrobot | 7.1 | 9.6 | 8.3 |
| LunarLander | 15.6 | 15.6 | 6.2 |
Table IV. Effective fan-out (recurrent synapses per neuron above ) of the final networks, mean over seeds.
, in contrast, measures how much of the weight crosses between columns. The two usually move together because one process drives both: the map term removes cross-column synapses, lowering the synapse count and at once. Without the map term, each task treats the cross-column synapses differently (Table V).
| Task | Mean cross-column weight | Within-column synapses | Within-column share |
| CartPole | 0.094 → 0.001 | 9.5 → 9.5 | 0.91 → 1.00 |
| Acrobot | 0.084 → 0.187 | 9.5 → 6.9 | 0.83 → 0.62 |
| LunarLander | 0.048 → 0.140 | 15.6 → 4.5 | 0.77 → 0.31 |
Table V. Recurrent weights of the partial ARM (MAP term off) from the start to the end of training, mean over seeds. Within-column synapses: per neuron, above . Within-column share: fraction of the total recurrent weight inside columns.
In CartPole the task rule removes the cross-column synapses by itself within about 200 episodes, so the partial arm ends up equal to the full one. In Acrobot the cross-column weights double, but the columns survive. In LunarLander the cross-column weights triple and most within-column synapses fall below threshold, mainly between episodes 2,500 and 5,500. The final within-column share of 0.31 is below the 0.77 of the random initial wiring, so the network ends up less modular than chance.
In LunarLander’s partial arm, then, the pruned synapses are the within-column ones, and fewer synapses go with more cross-column flow. This makes SOPs and negatively correlated ( over all regimes, within the partial arm). With the map term on, the cross-column synapses go and all 15.6 within-column synapses stay, which is why the dynamic and full arms end up with the same SOPs and .
Why the within-column synapses decay is not yet established. Candidates are the rate cost, homeostasis, or the task rule over LunarLander’s much longer training (10,000 episodes, against 400 for CartPole and 500 for Acrobot).
A fixed partition thus appears too rigid for this task, and unconstrained coupling too loose. The map term alone has limitations expressing a middle ground: its modulator charges every exit from column the same marginal cost, whatever the target column . A natural extension is an adaptive module-pair table learned for instance from the task’s credit, so that useful directed links between action columns are kept while the rest are pruned. We leave this to future work.
References
- [1]K. Malcolm and J. Casco-Rodriguez, “A comprehensive review of spiking neural networks: Interpretation, optimization, efficiency, and best practices,” arXiv:2303.10780 [cs.NE], 2023.arxiv.org/abs/2303.10780
- [2]P. A. Morella et al., “A memristor spiking-neuron integrated circuit with a scalable communication network and interface,” Science, vol. 345, no. 6197, pp. 668–673, 2014, doi: 10.1126/science.1254642.
- [3]M. Davies et al., “Loihi: A neuromorphic manycore processor with on-chip learning,” IEEE Micro, vol. 38, pp. 82–99, 2018.DOI
- [4]M. Prezioso et al., “Training and operation of an integrated neuromorphic network based on metal-oxide memristors,” Nature, vol. 521, pp. 61–64, 2015, doi: 10.1038/nature14441.
- [5]M. Davies et al., “Advancing neuromorphic computing with Loihi: A survey of results and outlook,” Proc. IEEE, vol. 109, no. 5, pp. 911–934, 2021.DOI
- [6]E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks,” arXiv:1901.09948 [cs.NE], 2019.arxiv.org/abs/1901.09948
- [7]T. P. Lillicrap, A. Santoro, L. Marris, C. J. Akerman, and G. E. Hinton, “Backpropagation and the brain,” Nat. Rev. Neurosci., vol. 21, pp. 335–346, 2020.
- [8]H. Markram, J. Lübke, M. Frotscher, and B. Sakmann, “Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs,” Science, vol. 275, no. 5297, pp. 213–215, 1997, doi: 10.1126/science.275.5297.213.DOI
- [9]G. Bi and M. Poo, “Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type,” J. Neurosci., vol. 18, no. 24, pp. 10464–10472, 1998.
- [10]P. U. Diehl and M. Cook, “Unsupervised learning of digit recognition using spike-timing-dependent plasticity,” Front. Comput. Neurosci., vol. 9, p. 99, 2015, doi: 10.3389/fncom.2015.00099.DOI
- [11]P. Date et al., “SuperNeuronMAT: An efficient matrix-based simulator for spiking neural networks,” arXiv:2608.008479 [cs.NE], 2026.
- [12]J. M. Beggs and D. Plenz, “Neuronal avalanches in neocortical circuits,” J. Neurosci., vol. 23, no. 35, pp. 11167–11177, 2003.
- [13]L. Lapicque, “Recherches quantitatives sur l’excitation électrique des nerfs traitée comme une polarisation,” J. Physiol. Pathol. Gen., vol. 9, pp. 620–635, 1907.
- [14]N. Brunel, “Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons,” J. Comput. Neurosci., vol. 8, no. 3, pp. 183–208, 2000.
- [15]G. B. Ermentrout and N. Kopell, “Parabolic bursting in an excitable system coupled with a slow oscillation,” SIAM J. Appl. Math., vol. 46, no. 2, pp. 233–253, 1986.
- [16]E. Montbrió, D. Pazó, and A. Roxin, “Macroscopic description for networks of spiking neurons,” Phys. Rev. X, vol. 5, no. 2, p. 021028, 2015.
- [17]S. Brin and L. Page, “The anatomy of a large-scale hypertextual web search engine,” Comput. Netw., vol. 30, pp. 107–117, 1998.
- [19]M. Rosvall, D. Axelsson, and C. T. Bergstrom, “The map equation,” Eur. Phys. J. Spec. Top., vol. 178, no. 1, pp. 13–23, 2009, doi: 10.1140/epjst/e2010-01179-1.DOI
- [20]M. Rosvall and C. T. Bergstrom, “Mapping change in large networks,” PLoS ONE, vol. 5, no. 1, p. e8694, 2010, doi: 10.1371/journal.pone.0008694.DOI
- [21]P. W. Holland, K. B. Laskey, and S. Leinhardt, “Stochastic blockmodels: First steps,” Soc. Netw., vol. 5, no. 2, pp. 109–137, 1983.
- [22]M. Kiselev, “CoLaNET – A spiking neural network with columnar layered architecture for classification,” arXiv:2409.01230 [cs.NE], 2025.arxiv.org/abs/2409.01230
- [23]M. Kiselev, “Classifying images with CoLaNET spiking neural network – the MNIST example,” arXiv:2409.07833 [cs.NE], 2024.arxiv.org/abs/2409.07833
- [24]M. Rosvall and C. T. Bergstrom, “Maps of random walks on complex networks reveal community structure,” Proc. Natl. Acad. Sci. USA, vol. 105, no. 4, pp. 1118–1123, 2008.
- [25]G. E. Hinton and D. van Camp, “Keeping the neural networks simple by minimizing the description length of the weights,” in Proc. 6th Annu. Conf. Comput. Learn. Theory (COLT), Santa Cruz, CA, USA, 1993, pp. 5–13, doi: 10.1145/168304.168306.DOI
- [26]M. Towers et al., “Gymnasium: A standard interface for reinforcement learning environments,” arXiv:2407.17032 [cs.LG], 2024.arxiv.org/abs/2407.17032
- [27]V. B. Mountcastle, “The columnar organization of the neocortex,” Brain, vol. 120, no. 4, pp. 701–722, 1997, doi: 10.1093/brain/120.4.701.DOI
- [28]D. Meunier, R. Lambiotte, and E. T. Bullmore, “Modular and hierarchically modular organization of brain networks,” Front. Neurosci., vol. 4, p. 200, 2010, doi: 10.3389/fnins.2010.00200.DOI
- [29]O. Sporns and R. F. Betzel, “Modular brain networks,” Annu. Rev. Psychol., vol. 67, pp. 613–640, 2016, doi: 10.1146/annurev-psych-122414-033634.DOI
- [30]W. Gerstner, M. Lehmann, V. Liakoni, D. Corneil, and J. Brea, “Eligibility traces and plasticity on behavioral time scales: Experimental support of neoHebbian three-factor learning rules,” Front. Neural Circuits, vol. 12, p. 53, 2018, doi: 10.3389/fncir.2018.00053.DOI
- [31]N. Frémaux and W. Gerstner, “Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules,” Front. Neural Circuits, vol. 9, p. 85, 2016, doi: 10.3389/fncir.2015.00085.DOI
- [32]W. Schultz, P. Dayan, and P. R. Montague, “A neural substrate of prediction and reward,” Science, vol. 275, no. 5306, pp. 1593–1599, 1997, doi: 10.1126/science.275.5306.1593.DOI
- [33]B. Engelhard et al., “Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons,” Nature, vol. 570, no. 7762, pp. 509–513, 2019, doi: 10.1038/s41586-019-1261-9.DOI
- [34]D. Attwell and S. B. Laughlin, “An energy budget for signaling in the grey matter of the brain,” J. Cereb. Blood Flow Metab., vol. 21, no. 10, pp. 1133–1145, 2001, doi: 10.1097/00004647-200110000-00001.DOI
- [35]H. B. Barlow, “Possible principles underlying the transformations of sensory messages,” in Sensory Communication, W. A. Rosenblith, Ed. Cambridge, MA, USA: MIT Press, 1961, pp. 217–234.
- [36]K. Friston, “The free-energy principle: A unified brain theory?” Nat. Rev. Neurosci., vol. 11, no. 2, pp. 127–138, 2010, doi: 10.1038/nrn2787.DOI
- [37]R. J. Douglas and K. A. C. Martin, “Neuronal circuits of the neocortex,” Annu. Rev. Neurosci., vol. 27, pp. 419–451, 2004, doi: 10.1146/annurev.neuro.27.070203.144152.DOI
- [38]P. Redgrave, T. J. Prescott, and K. Gurney, “The basal ganglia: A vertebrate solution to the selection problem?” Neuroscience, vol. 89, no. 4, pp. 1009–1023, 1999, doi: 10.1016/S0306-4522(98)00319-4.DOI
- [39]S. Chung and R. Kozma, “Reinforcement learning with feedback-modulated TD-STDP,” arXiv:2008.13044 [cs.LG], 2020.arxiv.org/abs/2008.13044
- [40]J. I. Yellott, “The relationship between Luce’s choice axiom, Thurstone’s theory of comparative judgment, and the double exponential distribution,” J. Math. Psychol., vol. 15, no. 2, pp. 109–144, 1977.
- [41]V. A. Traag, P. Van Dooren, and Y. Nesterov, “Narrow scope for resolution-limit-free community detection,” Phys. Rev. E, vol. 84, no. 1, p. 016114, Jul. 2011.
- [42]V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre, “Fast unfolding of communities in large networks,” J. Stat. Mech.: Theory Exp., vol. 2008, no. 10, p. P10008, Oct. 2008.
- [43]C. Curto, J. Geneson, and K. Morrison, “Fixed points of competitive threshold-linear networks,” arXiv:1804.00794, 2018.arxiv.org/abs/1804.00794
- [44]J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proc. Natl. Acad. Sci. USA, vol. 79, no. 8, pp. 2554–2558, 1982.
- [45]D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A learning algorithm for Boltzmann machines,” Cogn. Sci., vol. 9, no. 1, pp. 147–169, 1985.
- [46]E. P. Frady, S. J. Kent, B. A. Olshausen, and F. T. Sommer, “Resonator networks, 1: An efficient solution for factoring high-dimensional, distributed representations of data structures,” Neural Comput., vol. 32, no. 12, pp. 2311–2331, 2020.
- [47]T. Kohonen, “Self-organized formation of topologically correct feature maps,” Biol. Cybern., vol. 43, no. 1, pp. 59–69, 1982.
- [48]T. Kohonen, “The self-organizing map,” Proc. IEEE, vol. 78, no. 9, pp. 1464–1480, 1990.
- [49]C. Curto and K. Morrison. Relating network connectivity to dynamics: opportunities and challenges for theoretical neuroscience. Current Opinion in Neurobiology, 58:11–20, 2019. doi:10.1016/j.conb.2019.06.003.DOI
- [50]C. Curto, J. Geneson, and K. Morrison. Fixed points of competitive threshold-linear networks. Neural Computation, 31(9):94–155, 2019. doi:10.1162/neco_a_01151.DOI
- [51]C. Curto and K. Morrison. Graph rules for recurrent neural network dynamics. Notices of the American Mathematical Society, 70(4):536–551, 2023. doi:10.1090/noti2661.DOI
- [52]C. Curto, J. Geneson, and K. Morrison. Stable fixed points of combinatorial threshold-linear networks. Advances in Applied Mathematics, 154:102652, 2024. doi:10.1016/j.aam.2023.102652.DOI
- [53]C. Parmelee, S. Moore, K. Morrison, and C. Curto. Core motifs predict dynamic attractors in combinatorial threshold-linear networks. PLOS ONE, 17(3):e0264456, 2022. doi:10.1371/journal.pone.0264456.DOI
- [54]C. Parmelee, J. Londono Alvarez, C. Curto, and K. Morrison. Sequential attractors in combinatorial threshold-linear networks. SIAM Journal on Applied Dynamical Systems, 21(2):1597–1630, 2022. doi:10.1137/21M1445120.DOI
- [55]J. Londono Alvarez, K. Morrison, and C. Curto. Attractor-based models for sequences and pattern generation in neural circuits. Neural Computation, 38(3):257, 2026. Preprint: bioRxiv, doi:10.1101/2025.03.07.642121.DOI
- [56]D. Egas Santander, S. Ebli, A. Patania, N. Sanderson, F. Burtscher, K. Morrison, and C. Curto. Never theorems for fixed points of neural networks. In Research in Computational Topology 2, Association for Women in Mathematics Series, vol. 30. Springer, 2022. doi:10.1007/978-3-030-95519-9_6.
- [57]J. Londono Alvarez. Fixed point compositionality via low-rank gluing rules in inhibition-dominated threshold-linear networks. arXiv:2606.07336, 2026.arxiv.org/abs/2606.07336
- [59]C. Lienkaemper and G. K. Ocker. Diverse mean-field dynamics of clustered, inhibition-stabilized Hawkes networks via combinatorial threshold-linear networks. PRX Life, 2025. doi:10.1103/m8hf-q5pl. arXiv:2506.06234.DOI
- [60]R. Toro Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith. Reward machines: exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73:173–208, 2022. doi:10.1613/jair.1.12440.DOI
- [61]C. W. Omlin and C. L. Giles. Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6):937–972, 1996. doi:10.1145/235809.235811.DOI
- [62]M. Rigotti, D. D. Ben Dayan Rubin, X.-J. Wang, and S. Fusi. Internal representation of task rules by recurrent dynamics: the importance of the diversity of neural responses. Frontiers in Computational Neuroscience, 4:24, 2010. doi:10.3389/fncom.2010.00024.DOI




