betaFeedback

WorksComputer Science

Supervised Hebbian Class Networks: Learning Without Backpropagation

Abstract

This paper introduces Supervised Hebbian Class Networks as a method for training multi- layer perceptrons and convolutional networks without gradient calculations or backpropa- gation. Each neuron in a learned layer is assigned a class, while connections follow the usual fully connected or convolutional pattern. The method combines a class-based supervision rule shared across layers with a Lion optimizer adapted to Hebbian activity products. All layers train together throughout the run, and their activities provide class predictions di- rectly, without a separately trained classifier. The experiments test the method on MNIST with a multilayer perceptron and on CIFAR-10 with a convolutional networ

Introduction

Supervised Hebbian Class Networks assign each neuron in every learned layer to a class. That assignment determines how the label guides learning at the neuron’s incoming connections, while the connections themselves follow the usual fully connected or convolutional pattern.

For example, in a digit classifier, some neurons are assigned to digit 0, some to digit 1, and so on. An MLP keeps its usual connections: every neuron receives the preceding layer’s outputs, regardless of their assigned classes. A convolutional network keeps its usual spatial connections and shared filters; all neurons using the same filter are assigned the same class.

Prediction uses the network’s activities directly. In each layer, competition keeps a prescribed number of the largest activities in absolute value among neurons assigned to each class and sets the others to zero. The network adds their mean squared activities across layers and predicts the class with the largest score. For a correct prediction, the learning rule skips that image if its class score is sufficiently above the next-highest score. Otherwise, the rule reinforces connections entering neurons assigned to the correct class and suppresses those entering neurons assigned to the highest-scoring incorrect class. Every connection’s update uses the activities at its two ends.

All layers train jointly. Each minibatch passes through the complete network before any weights change. The learning rule is then applied in every layer using the recorded activities. All layers train together for the entire run, rather than being trained and frozen one at a time.

The paper contributes a supervised learning rule and an adaptation of the Lion optimizer for Hebbian learning. The supervision rule uses the relative gap between the correct-class score and the strongest incorrect-class score to decide whether to learn, then applies positive and negative learning signs to those classes throughout the network. The adapted optimizer applies Lion’s running-average and sign operations to local activity products and uses the receiving neuron’s activity to control weight growth. Together with competition and direct class scores, these components train all learned layers jointly without derivatives or backward error signals. Class assignments and joint Hebbian training have precedents, as discussed below; the contribution is the specific supervision and weight-update construction. The experiments demonstrate it with fully connected layers on MNIST and convolutional layers on CIFAR-10.

Related work

Classical Hebbian learning changes a weight in proportion to the product of its incoming and outgoing activities. Oja’s rule adds an activity-dependent term to control weight growth [17], and competition can determine which hidden neurons learn from an input [13]. Supervised Hebbian Class Networks use both positive and negative activity products, together with competition and Lion adapted to Hebbian learning.

Multilayer SoftHebb learns features through unsupervised Hebbian competition [10]. Its reported accuracies use successive layer training followed by a separately trained, gradient-based linear classifier; its appendix also reports similar results when all feature layers learn simultaneously. The method presented here adds class assignments and supervision throughout the learned layers, with classification directly from their activities.

Class assignments in hidden layers are not new to Hebbian learning. Amato et al. (2019) assign intermediate convolutional filters to classes, alongside filters shared by all classes. Their supervised rule substitutes label-defined teacher values for the receiving neuron’s activity. Supervised Hebbian Class Networks retain the measured activities and use the label and class scores to choose whether their products reinforce or suppress a connection. Supervised Hebbian Continual Representation Learning also assigns groups of neurons to classes and predicts by summing their activities [16]. It uses a single learned layer, freezing existing neurons and adding new ones as classes arrive.

In spiking networks, Le Mouel et al. (2014) use class-specific neuron pools and a margin that stops learning once the pools meet their response thresholds. Their Hebbian and anti-Hebbian updates are derived from approximate gradient descent on a margin-based loss; the thresholds constrain each pool’s response rather than the gap between the correct and strongest incorrect class scores used here. Selecting the strongest incorrect class also has precedents in multiclass margin methods, including passive-aggressive learning [4].

Paired Competing Neurons assign two competing neurons to each class, encouraging specialization for target and non-target examples [7]. Neuronal Competition Groups use several neurons per class with competition within each group [6]. Both use supervised spike-timing-dependent learning in a classification layer following an unsupervised feature extractor. CoLaNET organizes neurons into class-specific columns and combines anti-Hebbian and dopamine-modulated learning, but only its lowest layer has plastic input connections [11].

Deep Counterstream Associative Networks also perform supervised classification using local Hebbian-type learning [12]. Following an error, correcting target activities travel backward through learned connections to meet forward activities in a hidden layer. Supervised Hebbian Class Networks obtain all learning observations from one forward calculation and share class-dependent learning signs without propagating target activities backward.

Supervised Spike Agreement-Dependent Plasticity also learns without gradients, using agreement between a hidden neuron’s spikes and the correct-class output neuron’s spikes to guide hidden-layer learning [8]. It demonstrates one hidden layer whose neurons have no fixed class assignments. Its teaching signal depends on output-layer activity, whereas the rule here uses activities at each connection and shared class-dependent learning signs.

The Forward-Forward algorithm uses positive and negative examples and layer activity scores, with local gradient calculations in place of end-to-end backpropagation [9]. Hebbian Forward-Forward (HebbFF) replaces those gradients with activity products and also updates all layers within each epoch [14]. It appends labels at each layer, samples incorrect labels, and weights updates using each layer’s own score. Supervised Hebbian Class Networks need no candidate-label inputs: one forward calculation supplies all class scores, and their combined values choose the same correct and strongest incorrect groups for every layer.

Class groups and connections that mix their features also appear in channel-wise competitive Forward-Forward learning [18]. That method trains using layer-specific losses and interleaved schedule. A later extension, Channel-wise Spatial Gradient learning, retains class-assigned channels and interprets its updates using a Hebbian factor, but computes local loss gradients and trains layers in succession (Dupuis et al., [5]). Self-Contrastive Forward-Forward (SCFF) instead learns features from image pairs without labels, trains layers in succession, and uses a separately trained linear classifier (Chen et al., [3]).

The adapted update rule borrows the running-average and sign operations of Lion (Chen et al., [2]), applying them to Hebbian activity products rather than gradients.

Supervised Hebbian Class Networks

For a task with CC classes, every neuron in each of the LL learned layers has a fixed class assignment. A class group is the set of neurons assigned to the same class within a layer. In an MLP, layer ℓ\ell has nℓn_{\ell} neurons per class and CnℓCn_{\ell} neurons in total. In a convolutional network, it has nℓn_{\ell} filters per class and CnℓCn_{\ell} output channels in total; all spatial copies of a filter have the same class assignment. Training changes the connection weights while keeping these assignments fixed.

Figure 1 shows an MLP with classes assigned to the neurons in its learned layers. The input coordinates carry image values and have no class assignments; every neuron in the first learned layer receives all input coordinates. The learned layers are fully connected as usual. In the convolutional version, each filter spans a spatial patch and all input channels. A neuron’s assigned class is part of the model; the correct label of an image is used only during training and is never an input to prediction.

MLP with unassigned input coordinates and class-assigned neurons in its learned layers

Figure 1. An MLP with unassigned input coordinates and class-assigned neurons in its learned layers. Four input coordinates, two classes, and two learned layers are shown for illustration. Class assignments determine competition, scoring, and learning signs; connections follow the usual fully connected pattern. In the convolutional version, input pixels likewise have no class assignment, while each learned filter has an assigned class shared by all neurons using it.

Activities and competition

For neuron jj in group kk, write its weighted input as

zjk=∑iwijkui.(1)z_j^k = \sum_i w_{ij}^k u_i. \tag*{(1)}

Here uiu_i is an incoming activity and wijkw_{ij}^k is its connection weight. The index ii ranges over the preceding layer’s neurons in an MLP, or the input patch’s positions and channels in a convolution. A constant input represents the learned bias.

Output positions in a convolution. A convolution applies the same filter at many positions in an image. For example, sliding a 3×33 \times3 filter over a 32×3232 \times32 image, one pixel at a time and without padding, produces a 30×3030 \times30 output. Each of these 900 output positions corresponds to one placement of the input patch. The

index pp identifies that placement, not a pixel within the patch. A single index is sufficient to enumerate the output grid: p=1p=1 can denote the upper-left position, p=2p=2 the next position to its right, and so on across each row.

The three indices distinguish different parts of this calculation:

  • pp: which patch placement, or output position, is being processed.

  • ii: which input coordinate within that patch is used, including its channel. A 3×33 \times3 RGB patch contains 27 pixel values; the additional constant coordinate represents the bias.

  • jj: which filter within class group kk is applied.

For example, ii might identify the red value at the upper-left corner of a patch. As the patch moves to a new position pp, that input value changes, but its connection weight in filter jj remains the same. Each filter therefore produces one activity at each output position. In deeper layers, the patches contain activities from the preceding layer rather than image pixels.

Within each group, competition keeps the KℓK_{\ell} outputs with the largest absolute values and sets the others to zero. The resulting output is hjkh_j^k.

hjk={zjk,j is among the selected neurons or channels,0,otherwise.(2)h_j^k = \begin{cases} z_j^k, & j\text{ is among the selected neurons or channels},\\ 0, & \text{otherwise}. \end{cases} \tag*{(2)}

Competition preserves the signs of the outputs it keeps. For example, (−3,0.2,2,−0.1)(-3,0.2,2,-0.1) with Kℓ=2K_{\ell}=2 becomes (−3,0,2,0)(-3,0,2,0). This selection supplies the network’s nonlinearity; no additional activation function is applied. In a convolutional layer, competition selects KℓK_{\ell} channels within each class group separately at every output position pp. It selects which filters respond at a given position, not which positions in the image to retain. Prediction and training use the same selection rule.

Class scores

For image xx, the notation hℓ,p,jk(x)h_{\ell,p,j}^k(x) makes the layer ℓ\ell and output position pp explicit; jj and kk still identify the neuron or filter and its class group. Let PℓP_{\ell} be the number of output positions in that layer. In the example above, Pℓ=30×30=900P_{\ell}=30 \times30=900. An MLP has no sliding patches: jj identifies a neuron within its class group, and pp takes just one value, so Pℓ=1P_{\ell}=1.

Each class score adds the layer contributions. Within a layer, it averages h2h^2 across the KℓK_{\ell} selected neurons or channels and the PℓP_{\ell} output positions:

Sk(x)=∑ℓ=1L1PℓKℓ∑p=1Pℓ∑j=1nℓ(hℓ,p,jk(x))2,y^(x)=arg⁡max⁡k Sk(x).(3)S_k(x)=\sum_{\ell=1}^{L}\frac{1}{P_{\ell}K_{\ell}}\sum_{p=1}^{P_{\ell}}\sum_{j=1}^{n_{\ell}}\left(h_{\ell,p,j}^k(x)\right)^2,\qquad\hat{y}(x)=\underset{k}{\arg\max}\ S_k(x). \tag*{(3)}

An activity of −2-2 contributes four to the score, just as +2+2 does, although their signs remain different in the values passed forward and used for learning. The denominator PℓKℓP_{\ell}K_{\ell} is the number of output positions multiplied by the number of neurons or channels selected per class at each position. These class scores measure neural activity; they are not probabilities. The score has no learned coefficients and needs no separate classifier.

Learning from local activity

Consider one connection with weight wijw_{ij}, as shown in Figure 2. During the forward calculation, input uiu_i passes through this connection to neuron jj. The neuron combines all its weighted inputs, and competition determines its output hjh_j. Learning then multiplies two values from that calculation: the input uiu_i and the output hjh_j. This product, uihju_i h_j, is the Hebbian observation for that connection.

One layer with four inputs and two class groups, each containing three neurons

Figure 2. One layer with four inputs and two class groups, each containing three neurons. Every input connects to every neuron. Each ∑\sum denotes a neuron’s weighted sum; the example values beside the arrows are its activities before competition. With K=2K = 2, competition keeps the two largest absolute activities in each group, preserving their signs, and sets the third to zero. The resulting outputs hjkh_j^k pass to the next layer and are also used for learning. The highlighted connection uses input u2=2u_2 = 2 and output h11=3h_1^1 = 3 in its Hebbian product.

For the highlighted connection in Figure 2, u2=2u_2 = 2 and h11=3h_1^1 = 3, so the product is six. Basic Hebbian learning would add 6η6\eta to w211w_{21}^1, where η\eta is a positive step size. The supervised rule described below first assigns a positive or negative sign to the product, averages the observations, and controls the size of the resulting weight change.

Choosing the learning signs

For a correctly classified image, the rule skips learning if the correct class’s score is sufficiently higher than the next-highest score. Incorrect predictions, and correct predictions with too small a gap, contribute to weight updates. To specify this rule, let yy be the image’s correct label and rr its highest-scoring incorrect class. Define the relative score gap, or margin, as

γ=Sy(x)−Sr(x)Sy(x)+Sr(x)+ε,ε=10−30.(4)\gamma= \frac{S_y(x) - S_r(x)}{S_y(x) + S_r(x) + \varepsilon}, \qquad\varepsilon= 10^{-30}. \tag*{(4)}

If the prediction is correct and γ≥δ\gamma\ge\delta, the image makes no contribution to weight updates. Here δ>0\delta> 0 is the required margin. Otherwise, neurons assigned to the correct class use sign sy=+1s_y = +1 and neurons assigned to the highest-scoring incorrect class use sign sr=−1s_r = -1. All other signs are zero. These signs determine whether each receiving neuron’s local activity products are added or subtracted.

The signs apply to the receiving groups in every layer. A connection from a dog-associated neuron to a cat-associated neuron therefore uses the cat group’s sign. Its update still depends on the actual activities at both ends of that connection. Positive and negative observations have equal weight.

Observations from one image

The learning rule is first described for one image at a time. Its minibatch form is described separately in § 3.5; that section is not needed to follow the single-image rule below.

Fix one layer and a receiving class group kk. Their indices are omitted from the weights and activities to keep the notation simple. Two quantities are used to update each weight wijw_{ij}: the signed activity product AijA_{ij} and the receiving neuron’s squared activity QjQ_j.

Fully connected layer. For a neuron jj whose output hjh_j is nonzero and whose class has learning sign sk≠0s_k \ne0, these quantities are simply

Aij=skuihj,Qj=hj2.(5)A_{ij}=s_k u_i h_j,\qquad Q_j=h_j^2. \tag*{(5)}

For example, if ui=2u_i=2, hj=3h_j=3, and sk=+1s_k=+1, then Aij=6A_{ij}=6 and Qj=9Q_j=9. With negative supervision, Aij=−6A_{ij}=-6 while QjQ_j remains nine. The first quantity supplies the direction of learning; the second controls weight growth, as described in § 3.3. If hj=0h_j=0 or sk=0s_k=0, the neuron keeps its weights and stored history unchanged.

Convolutional layer: averaging within one image. A filter uses the same weights at many positions in an image. For each filter, the rule averages over positions where its output after competition is nonzero. Let Pj={p:hpj≠0}P_j=\{p:h_{pj}\ne0\} be this set, and let upiu_{pi} be coordinate ii of the patch at position pp. For a filter with sk≠0s_k\ne0 and nonempty PjP_j, the averages are

Aij=1∣Pj∣∑p∈Pjskupihpj,Qj=1∣Pj∣∑p∈Pjhpj2.(6)A_{ij}=\frac{1}{\lvert P_j\rvert}\sum_{p\in P_j}s_k u_{pi}h_{pj},\qquad Q_j=\frac{1}{\lvert P_j\rvert}\sum_{p\in P_j}h_{pj}^{2}. \tag*{(6)}

Zero outputs contribute nothing to either sum. The choice matters in the denominator: if a filter has nonzero outputs at 12 of 100 positions, both sums are divided by 12, not 100. An exactly zero output is excluded from this count even if the filter was selected by competition at that position. Thus the rule averages over nonzero responses, rather than over all positions where the filter was applied.

An alternative would be to average over positions where competition selected the filter, counting selected outputs even when they are exactly zero. The two choices give the same averages when all selected outputs are nonzero; otherwise, their denominators differ. The reported experiments use the nonzero-output count in Equation (6).

The input coordinate ii stays fixed while pp ranges over patch placements; different weights within the filter are not averaged together. This averaging is part of the convolutional learning rule even when training on one image at a time. If no position has a nonzero output, or if sk=0s_k=0, the filter’s weights and history stay unchanged.

In both architectures, AijA_{ij} can be positive or negative, while QjQ_j is always nonnegative. Negative supervision reverses the sign of the activity product, but leaves QjQ_j unchanged in both updates. For the reflected filters described in § 4, inputs in Equation (6) are placed in the coordinates of the selected filter orientation before their products are accumulated.

Lion adapted to Hebbian learning

Each connection stores a running average mijm_{ij} of its signed products. Combine this history with the current observation to obtain a direction:

Dij=sign⁡(0.125mij+0.875Aij)din,mij+=0.99mij+0.01Aij.(7)D_{ij}=\frac{\operatorname{sign}(0.125m_{ij}+0.875A_{ij})}{\sqrt{d_{\mathrm{in}}}},\qquad m_{ij}^{+}=0.99m_{ij}+0.01A_{ij}. \tag*{(7)}

The superscript ++ denotes the new value. The direction uses the old history, and dind_{\mathrm{in}} counts incoming coordinates, including the bias. The sign function returns −1-1, zero, or +1+1. This adapts Lion’s sign-and-history construction [2]: the current Hebbian observation and its running average take the places of the gradient and gradient history. Separate averaging rates determine the update direction and the stored history. No gradient is calculated.

The experiments use two forms of the weight update. In runs using semi-implicit Lion, the update is

wij+=wij+ηDij1+ηQj(8)w_{ij}^{+} = \frac{w_{ij} + \eta D_{ij}}{1 + \eta Q_j} \tag*{(8)}

(semi-implicit).

In runs using explicit Lion, the update is

wij+=wij+ηDij−ηQjwij(9)w_{ij}^{+} = w_{ij} + \eta D_{ij} - \eta Q_j w_{ij} \tag*{(9)}

(explicit).

Both forms use the same Lion direction DijD_{ij} and squared-activity observation QjQ_j. In Equation (8), the numerator moves the weight in the chosen direction and the denominator controls weight growth for both positive and negative observations. Unlike Lion’s usual weight decay, this control depends on the receiving neuron’s observed activity.

The difference is which weight appears in the activity-dependent term. The explicit form uses the old weight wijw_{ij}. The semi-implicit form uses the new weight: writing wij+=wij+ηDij−ηQjwij+w_{ij}^{+} = w_{ij} + \eta D_{ij} - \eta Q_j w_{ij}^{+} and collecting the terms in wij+w_{ij}^{+} gives Equation (8). The Lion direction and measured activities remain fixed in both forms. The semi-implicit denominator is always positive and at least one. Dividing by this denominator cannot reverse the sign of the numerator; it does not prove convergence of the full learning process. Both forms use the same step-size schedule.

Joint training and the scope of locality

For one training image, the complete learning procedure has four steps:

  1. Pass the image through every layer using the current weights. Save the inputs uu and outputs hh at each layer.

  1. Calculate the class scores. If the prediction is correct and the score gap meets the required margin, skip learning for this image. Otherwise, assign positive and negative learning signs to neurons of the correct and highest-scoring incorrect classes.

  1. In every layer, calculate each connection’s AijA_{ij} and QjQ_j using Equation (5) for a fully connected layer or Equation (6) for a convolutional layer.

  1. Update the weights and histories of neurons with nonzero outputs and learning signs, using Equations (7) and either 8 or 9. Then process the next image.

All observations come from the same forward calculation, completed before weights change. No updated layer is used to recalculate another layer’s observations for that image. Every layer trains throughout the run; no layer is frozen after an initial training stage. The skip rule still applies to each image, and neurons with no contributing observations keep their weights unchanged.

Each weight uses its own incoming activity, its receiving neuron’s output, and its own history. Competition compares activities within a group, and the class signs use the label and whole-network scores. Supervision is therefore shared across the network, but no layer receives an error vector propagated from another layer. There are no activation derivatives, backward passes, or separate classifier fits. The update is not claimed to follow the gradient of a global loss.

Minibatch training

The algorithm above can update after each image. To process several images efficiently on a computer, the experiments instead use minibatches. This subsection defines the minibatch form used for the reported results. Averaging observations before applying the sign operation and updating the history generally produces different weight changes from processing the same images one at a time. All images in a minibatch pass through the network using the same current weights. The learning signs and observations are calculated separately for each image. Let bb index the images, and write abija_{bij} and qbjq_{bj} for the values of AijA_{ij} and QjQ_j obtained from image bb using Equation (5) or 6.

For neuron or filter jj, let BjB_j contain the images with a nonzero learning sign for its class and at least one nonzero output from that neuron or filter. Average only over these contributing images:

Aij=1∣Bj∣∑b∈Bjabij,Qj=1∣Bj∣∑b∈Bjqbj.(10)A_{ij} = \frac{1}{\lvert B_j\rvert}\sum_{b\in B_j}a_{bij}, \qquad Q_j = \frac{1}{\lvert B_j\rvert}\sum_{b\in B_j}q_{bj}. \tag*{(10)}

These averages are used in Equations (7) and either 8 or 9, with one weight and history update per minibatch. If BjB_j is empty, the neuron keeps its weights and history unchanged. Each contributing image has equal weight, even when a convolutional filter has nonzero outputs at different numbers of positions in different images. Thus convolutional observations are averaged first over positions within each image, then over images. With a minibatch of one image, this procedure reduces to the single-image rule above.

Experiments

Both experiments use the same learning rule for MNIST and CIFAR-10, with the settings in Table 1. Both networks train jointly for 100 epochs in minibatches of 32, with each training image presented once per shuffled epoch. The initial step size is 0.00075, held constant for 20 epochs and then reduced with a cosine schedule over the remaining 80. Neither experiment uses image augmentation or weight averaging.

SettingMNISTCIFAR-10
Learned layers3 fully connected3 convolutional
Total neurons / channels per layer2560, 2560, 25601280, 1280, 1280
Neurons / channels per group256, 256, 256128, 128, 128
Outputs kept per group128, 128, 1288, 8, 8 per location
Kernel sizes—3×33 \times3, 3×33 \times3, 5×55 \times5
Learned weights, including biases15,121,92055,744,000
Required margin δ\delta0.100.05
Development training / validation50,000 / 10,00040,000 / 10,000
Final training / test60,000 / 10,00050,000 / 10,000

Table 1. Selected MLP and convolutional network. Every neuron has an assigned class; each layer divides its neurons or filters equally among the ten classes.

Inputs and architecture. MNIST pixels are scaled to [0, 1], centered using the training mean at each pixel, and divided by each image’s largest absolute centered value, leaving an all-zero image unchanged. CIFAR-10 pixels are scaled to [0, 1] and centered using the training RGB means.

The constant input and learned bias. A first-layer 3×33 \times3 RGB patch contains 27 pixel values. Write these centered values as p1,…,p27p_1,\ldots,p_{27}. Preprocess flattens the patch into a vector, appends one extra entry equal to 1, and divides the whole vector by its length:

u=[p1,…,p27,1]T∑i=127pi2+1.(11)u = \frac{[p_1,\ldots,p_{27},1]^{\mathsf{T}}}{\sqrt{\sum_{i=1}^{27}p_i^2+1}}. \tag*{(11)}

Each filter has 27 pixel weights and one additional learned weight, its bias bjkb_j^k. Its response before competition is

zjk=∑i=127wijkpi+bjk∑i=127pi2+1.(12)z_j^k = \frac{\sum_{i=1}^{27}w_{ij}^k p_i+b_j^k}{\sqrt{\sum_{i=1}^{27}p_i^2+1}}. \tag*{(12)}

The appended 1 is not a pixel or padding. Its weight lets the filter add an offset to its weighted pixel sum; that offset can change which filters win the competition. The same filter weights and bias are used at every patch location. The bias follows the same local learning rule as the other weights, using the final entry of uu as its incoming activity. In this first layer that entry is 1/∑ipi2+11/\sqrt{\sum_i p_i^2+1} because the entire vector is normalized.

The MNIST first layer likewise appends 1 to its 784 image values and normalizes the resulting 785-entry vector. Deeper layers in both networks append 1 to their incoming activities without this length normalization, so their responses are simply the weighted activity sum plus the learned bias. Weights on nonconstant inputs start as independent Gaussian values normalized to unit length for each receiving neuron; bias weights and running averages start at zero.

The CIFAR-10 network uses convolutions without padding, moving each filter one pixel at a time. Each filter evaluates its ordinary and horizontally reflected orientations, retaining the response with larger absolute value and preserving its sign. The two orientations share weights. Learning maps the selected orientation’s observations to those same weights. After each of the first two layers, pooling selects the signed largest-magnitude value in each 2×22 \times2 window at stride one, then smooths and downsamples using the fixed filter

116(121242121)(13)\frac{1}{16}\begin{pmatrix}1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1\end{pmatrix} \tag*{(13)}

with stride two and reflected boundaries, following the smoothing-before-downsampling approach of Zhang (2019). Reflection does not repeat the boundary value: for example, [a,b,c][a,b,c] is extended to [b,a,b,c,b][b,a,b,c,b]. The three convolutional outputs have spatial sizes 30×3030 \times30, 13×1313 \times13, and 2×22 \times2.

Development and final testing. Architecture, competition, margin and step size follow the earlier validation scans on the splits in Table 1. The comparison omits dropout and varies only the weight update. Training on each dataset comprised four runs with semi-implicit Lion and four with explicit Lion. The two models in each pair share the same initialization and example order. The four seeds are the original seed selected during development and three additional seeds. Each model trains from scratch on the full training set, including former validation images, with preprocessing statistics computed from training images only. The reported results include every run at epoch 100, evaluated on the same 10,000 official test images for each dataset. These test sets were also used in earlier experiments; the comparison is not a fresh test of the decision to omit dropout.

Computational performance

The training kernels were implemented in Mojo 1.2.01 to allow direct optimization of GPU operations without requiring backpropagation or automatic differentiation. On an NVIDIA GeForce RTX 4090, the original reference runs on the complete training sets averaged approximately 2 seconds per epoch for MNIST and 28 seconds per epoch for CIFAR-10, excluding evaluation and checkpoint saving.

Results and discussion

Switching from the semi-implicit to the explicit Lion update changed mean test accuracy by +0.05 percentage points on MNIST and −0.02 on CIFAR-10 (Table 2). The 95% paired tt intervals across the four seed differences are [−0.07, 0.17] and [−0.54, 0.50] points, respectively. They describe variation across seeds on these fixed tests. Neither comparison establishes an advantage for either update, or equivalence between them.

DatasetLion updateMean ± SDMinimumMaximum
MNISTSemi-implicit98.14±0.0498.14 \pm0.0498.1098.19
MNISTExplicit98.19±0.0998.19 \pm0.0998.0698.27
CIFAR-10Semi-implicit80.69±0.2880.69 \pm0.2880.4181.08
CIFAR-10Explicit80.67±0.1680.67 \pm0.1680.4580.82

Table 2. Official test accuracy (%) after 100 epochs. Each row summarizes four matched seeds. SD is the sample standard deviation across runs; the endpoints are the lowest and highest observed accuracies.

Table 3 compares the ordinary sum of three layer scores with each layer scored alone for the semi-implicit models. The sum improves on the final layer alone by an average of 0.06 percentage points on MNIST and 8.68 on CIFAR-10. Earlier layers still compute the features used by deeper layers; only the activities counted in the score change.

Activities included in scoreMNISTCIFAR-10
Layer 1 alone56.9311.19
Layer 2 alone91.6132.56
Layer 3 alone98.0872.01
Layers 1+2+398.1480.69

Table 3. Test accuracy (%) using different layers for the class score. Entries are means over the four semi-implicit runs per dataset, without retraining.

For published context, SCFF reports test accuracies of 98.70%±0.01%98.70\% \pm0.01\% on MNIST and 80.75%±0.12%80.75\% \pm0.12\% on CIFAR-10 [3]. Its features are learned without labels and classified by a separate linear model trained with backpropagation. Channel-wise competitive Forward-Forward reports test errors of 0.58%±0.08%0.58\% \pm0.08\% on MNIST and 21.89%±0.44%21.89\% \pm0.44\% on CIFAR-10 for its configuration with a learned softmax classifier [18]. The Supervised Hebbian Class Network’s CIFAR-10 result is close to SCFF’s reported accuracy; both published methods report stronger MNIST results than the Supervised Hebbian Class Network. Architectures, preprocessing, supervision, classifiers, training schedules, and computational budgets differ, so these are published reference points rather than controlled comparisons.

The main result is that MLPs and convolutional networks can learn jointly from local activity products when each neuron is assigned a class. That assignment determines which receiving neurons are reinforced or suppressed, while the activities at each connection determine its update. The usual connections let neurons use features computed by neurons assigned to other classes. No separate classifier is needed to translate these representations into a prediction.

Conclusion

Supervised Hebbian Class Networks provide a method for jointly training MLPs and convolutional networks with a class assigned to every neuron in each learned layer. The relative score gap controls whether learning occurs, and the correct and strongest incorrect classes determine positive and negative Hebbian observations throughout the network. Lion adapted to Hebbian learning combines sign-based updates and local running averages of these observations with an activity-dependent term that controls weight growth. Both Lion updates achieve about 98.2% mean test accuracy on MNIST and 80.7% on CIFAR-10. The contribution is a practical form of joint multilayer learning that requires neither gradient calculations nor backpropagation.

References

  1. [2]Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms. arXiv:2302.06675, 2023. URL https://arxiv.org/abs/2302.06675.
  2. [3]Xing Chen, Dongshu Liu, Jérémie Laydevant, and Julie Grollier. Self-Contrastive Forward-Forward algorithm. Nature Communications, 16:5978, 2025. doi: 10.1038/s41467-025-61037-0. URL https://www.nature.com/articles/s41467-025-61037-0.DOI
  3. [4]Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. Online passive-aggressive algorithms. Journal of Machine Learning Research, 7:551–585, 2006. URL https://jmlr.org/papers/v7/crammer06a.html.
  4. [5]Théo Dupuis, Adrien Renaudineau, Guillaume Bourgeois, Joel Minguet Lopez, Gabriele Navarro, Jean-Michel Portal, Elisa Vianello, Jacques-Olivier Klein, and Damien Querlioz. Local learning with imperfect analog synapses. arXiv:2610.27910, 2026. URL https://arxiv.org/abs/2610.27910.
  5. [6]Gaspard Goupy, Pierre Tirilly, and Ioan Marius Bilasco. Neuronal competition groups with supervised STDP for spike-based classification. In Advances in Neural Information Processing Systems, volume 37, 2024a. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/bfe593e151614c691323c08c0079c1f6-Abstract-Conference.html.arxiv.org/abs/2410.17066
  6. [7]Gaspard Goupy, Pierre Tirilly, and Ioan Marius Bilasco. Paired competing neurons improving STDP supervised local learning in spiking neural networks. Frontiers in Neuroscience, 18:1401690, 2024b. doi: 10.3389/fnins.2024.1401690. URL https://doi.org/10.3389/fnins.2024.1401690.
  7. [8]Gouri Lakshmi S, Athira Chandrasekharan, Harshit Kumar, Muhammed Sahad E. Bikas C. Das, and Saptarshi Bej. Building supervision into hebbian plasticity through spike agreement. arXiv:2601.08526, 2026. URL https://arxiv.org/abs/2601.08526v2.
  8. [9]Geoffrey Hinton. The Forward-Forward algorithm: Some preliminary investigations. arXiv:2212.13345, 2022. URL https://arxiv.org/abs/2212.13345.
  9. [10]Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learning without feedback. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=8gd4M-_Rj1.arxiv.org/abs/2209.11883
  10. [11]Mikhail Kiselev. CoLaNET – a spiking neural network with columnar layered architecture for classification. arXiv:2409.01230, 2024. URL https://arxiv.org/abs/2409.01230v6. Revised March 2025.
  11. [12]Andreas Knoblauch. Supervised hebbian learning in deep counterstream associative networks. arXiv:2606.29528, 2026. URL https://arxiv.org/abs/2606.29528.
  12. [13]Dmitry Krotov and John J. Hopfield. Unsupervised learning by competing hidden units. Proceedings of the National Academy of Sciences, 116(16):7723–7731, 2019. doi: 10.1073/pnas.1820458116. URL https://pmc.ncbi.nlm.nih.gov/articles/PMC6475390/.DOI
  13. [14]Andrii Krutsylo. The Hebbian Forward-Forward algorithm. In 17th Annual Workshop on Optimization for Machine Learning, 2025. URL https://opt-ml.org/papers/2025/paper147.pdf.
  14. [16]Pawel Morawiecki, Andrii Krutsylo, Maciej Wołczyk, and Marek Śmieja. Hebbian continual representation learning. In Proceedings of the 56th Hawaii International Conference on System Sciences, pp. 1259–1268, 2023. URL https://hdl.handle.net/10125/102785.arxiv.org/abs/2207.04874
  15. [17]Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15:267–273, 1982. doi: 10.1007/BF00275687. URL https://doi.org/10.1007/BF00275687.
  16. [18]Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou, and Theocharis Theocharides. Convolutional channel-wise competitive learning for the Forward-Forward algorithm. Proceedings of the AAAI Conference on Artificial Intelligence, 38(13):14536–14544, 2024. doi: 10.1609/aaai.v38i13.29369. URL https://arxiv.org/abs/2312.12668.

Paper details

vlnc:W_e8qcvr53
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents