betaFeedback

WorksComputer Science

Halting in Recurrent Sequence Models

Abstract

Recurrent sequence models, such as recurrent neural networks (RNNs) and state space models (SSMs), are often trained to emit an end-of-sequence (EOS) token when their output sequence is complete. During training, EOS is treated as another token without a special loss term or incentive. Although this training paradigm is widely used, it is not understood how models decide to output EOS. We study a minimal version of the EOS halting problem in which networks are trained to emit a given input sequence an instructed number of times, followed by EOS. We train tanh RNNs and linear SSMs on this task, finding that the RNNs encode the number of tokens left to copy in a linear subspace, of which two principal components capture most of the variation. Hidden states with the same remaining count converge to a shared representation regardless of their instructed repeat count. By modulating the hidden state using an affine transformation at the boundary between copied sequences, we shift the RNN's state to the representation shared by states with one fewer copy remaining. The unchanged network emits EOS one copy earlier while reproducing the shortened target sequence in 94.4% of new sequences, revealing that this representation causally determines the stopping time. In the SSMs, the difference between trajectories with different repeat counts is separated across two-dimensional rotation planes whose contributions to the EOS comparison cancel during copying and align near the stopping time. Rotating the planes independently removes most of the intended stopping while preserving the copied content, which shows that the stopping time depends on the planes' relative phases. These results identify two candidate primitives of EOS halting in recurrent models: a shared encoding of the number of remaining tokens in tanh RNNs and phase alignment across fixed rotations in linear SSMs.

Introduction

How does a recurrent neural network (RNN) know when to stop outputting new tokens when trained on a task with a variable sequence length? In such a task setting, RNNs commonly end generation by emitting an end-of-sequence (EOS) token after their content. Given an input xtx_t and hidden state ht−1h_{t-1}, the model updates its state and assigns a score to every output token according to

ht=Fθ(ht−1,xt),zt=Wyht.h_t = F_\theta(h_{t-1}, x_t), \qquad z_t = W_y h_t.

Here FθF_\theta is the recurrent update with parameters θ\theta, WyW_y is the output matrix (decoder), and ztz_t is a vector containing the scores for each token vv in the task’s vocabulary V\mathcal{V}. The EOS margin

mt=zt,EOS−max⁡v≠EOSzt,v(1)m_t = z_{t,\mathrm{EOS}} - \max_{v \ne\mathrm{EOS}} z_{t,v} \tag*{(1)}

provides us with a mathematical object that defines when EOS is output: the model emits content while mt≤0m_t \le0 and emits EOS once mt>0m_t > 0. When subsequent inputs are fixed, the current state determines the entire future trajectory and can therefore determine the stopping time even while its current margin remains negative. The margin only records when EOS is emitted. Why its sign changes at a particular step has to be explained by the hidden dynamics, which carry the count instruction until then. Existing attempts at characterizing EOS prediction do not provide a transition-level mechanism of how a count instruction is transported into the EOS comparisons (§ 2).

The repeat-copy task augments the copy task by adding a count instruction signaling that the sequence to be copied should be copied K∈NK \in\mathbb{N} times, followed by EOS [11]. Under different count instructions JJ and KK, J≠KJ \ne K, the same model produces hidden trajectories htJh_t^J and htKh_t^K. At each output step shared by such trajectories, the trajectory difference

ΔtJK=htJ−htK\Delta_t^{JK} = h_t^J - h_t^K

isolates the effect of the repeat count, since both trajectories store the same sequence. Its evolution describes how the count reaches the EOS comparisons.

We consider a minimal RNN architecture and show that its states carry the number of output steps remaining, which at the fixed sequence length is equivalent to the position within a copy together with the number of copies remaining. Trajectories with the same number of remaining steps approach the same states regardless of the count instruction. Within each copy, WhW_h mixes ΔtJK\Delta_t^{JK}, and the tanh activation then attenuates each of its components according to the current pair of hidden states. An affine state transformation learned from separate sequences shifts a state at a copy boundary to the states with one fewer copy remaining, which causes EOS to be emitted one copy earlier without changing the tokens output until that point. For comparison, experiments using a linear SSM reveal that ΔtJK\Delta_t^{JK} evolves within two-dimensional rotation planes whose contributions to the EOS comparisons cancel during copying and increase near the correct stopping time. Independently rotating these contributions removes most of the intended stopping behavior while preserving the copied content (Figure 1).

Figure 1. The count instruction changes the EOS time while the stored sequence remains fixed. A, RNN trajectory differences on held-out sequences projected onto two principal components obtained from separate sequences. Open circles mark the state after the count instruction, stars mark the earlier target EOS, and arrows follow the autonomous recurrence. B, mean contributions of the 64 SSM rotation blocks to the limiting EOS comparison for count change 4 → 2. Between copy boundaries the contributions have mixed signs. At the target EOS, most contributions have the same sign and the EOS margin crosses zero. C, repeat-copy examples ending after two or four copies of SS. RNN seed 5, the model with the median energy in the two plotted components, and SSM seed 1 are shown.

Related work

Prior work on halting using an EOS token has examined sequence probability, decoding criteria, learned stopping, and length generalization [24, 26, 31, 33]. Newman et al. [27] find that EOS training organizes recurrent states into length manifolds and terminal attractors. Grant et al. [9] show that in models trained on tasks using a latent count variable, hidden state interventions on the network’s representation of the count variable leads to altered output behavior.

Adaptive computation methods add an explicit halting variable that controls the number of internal updates a model makes along its trajectory [1, 3, 5, 7, 10]. Structured state space models (SSMs) implement recurrent sequence processing through constrained linear transitions [14, 15, 28, 30], including the scaled rotation blocks in the linear SSM studied here.

Controlled sequence tasks reveal memory, position, and algorithmic structure in recurrent states [2, 6, 12, 17, 20, 21, 22]. Fixed points and local linearizations describe their evolution under the recurrent transition [16, 25, 29, 32]. Causal interventions test the behavioral role of variables identified by linear decoders [4, 8, 18, 19].

Stopping as a hitting-time problem

Task. We use the repeat-copy task [11], in which each input sequence consists of L=8L = 8 symbols S=(s0,…,sL−1)S = (s_0, \ldots, s_{L-1}) with si∈{−1,+1}4s_i \in\{-1, +1\}^{4}, yielding sixteen possible content tokens. After the symbols and a delimiter, an instruction cK∈{−1,+1}3c_K \in\{-1, +1\}^{3} with a single +1+1 entry encodes K∈{2,3,4}K \in\{2, 3, 4\} and requests the output

S S ⋯ S⏟K copies EOS.\underbrace{S\ S\ \cdots\ S}_{K\ \mathrm{copies}}\ \mathrm{EOS}.

Unlike Graves et al. [11], who supply the repeat count as a scalar on a separate input channel, we present it as a one-hot token after the delimiter.

Models. We use two models: an RNN with one hidden layer and tanh activation, and a linear SSM. Both models have width d=128d = 128. A matrix WxW_x first maps each input into the hidden state before the bias-free output matrix WyW_y returns the token scores. The recurrent operator is WhW_h for the RNN and AA for the SSM:

ht=tanh⁡(Whht−1+Wxxt),zt=Wyht,(2)h_t = \tanh(W_h h_{t-1} + W_x x_t), \qquad z_t = W_y h_t, \tag*{(2)}
ht=Aht−1+Wxxt,zt=Wyht,(3)h_t = A h_{t-1} + W_x x_t, \qquad z_t = W_y h_t, \tag*{(3)}

The SSM transition contains 64 learned and scaled 2×22 \times2 rotation blocks. We train five seeds for each architecture, using 100,000 updates for the RNNs and 5,000 updates for the SSMs. Every checkpoint emits EOS at the intended step on the held-out sequences used in the analyses below, and only RNN seed 4 miscopies a single content token, in one of 512 sequences.

Categorical stopping. Two paradigms are generally used to signal completion. The first is categorical stopping, which defines EOS as one token of the model’s vocabulary V\mathcal{V} and compares its score directly with the scores assigned to all other ν∈V\nu\in\mathcal{V}. The second encodes the EOS signal in a separate binary output “head” that outputs 1 when the model signals completion. We consider categorical stopping throughout, where the stopping time depends on how the recurrent state reaches the part of the output space that favors EOS.

Under categorical stopping, the comparison between EOS and each content token is linear in the hidden state. Let wv⊤w_v^{\top} denote row vv of WyW_y. For each content token jj, the difference Cj=wEOS−wjC_j = w_{\mathrm{EOS}} - w_j defines the EOS comparison

gj(h)=CjTh,g_j(h) = C_j^{\mathsf{T}} h,

and stacking the rows CjTC_j^{\mathsf{T}} gives g(h)=Chg(h) = Ch with m(h)=min⁡jgj(h)m(h) = \min_j g_j(h), so that mt=m(ht)m_t = m(h_t). We call the comparison that attains this minimum the limiting EOS comparison. Note that the EOS token is assigned the largest value in the network’s output layer exactly when m(h)>0m(h) > 0. Generally, many distinct hidden states lead to the model outputting EOS [13, 27]. Thus, we can define the set HEOS={h∣m(h)>0}\mathcal{H}_{\mathrm{EOS}} = \{h \mid m(h) > 0\} containing all valid hidden states associated with emission of EOS.

Stopping can then be viewed as a hitting-time problem. Once the model has finished reading all input symbols, its input is set to zero and the hidden-state recurrence becomes autonomous, ht+1=F(ht)h_{t+1}=F(h_t), and stopping must therefore be encoded by hth_t alone. Hence, chained application of FF determines when the trajectory first enters HEOS\mathcal{H}_{\mathrm{EOS}}. We call this first entry time the EOS hitting time,

τ(h)=min⁡{u≥0∣m(Fu(h))>0},\tau(h)=\min\{u\ge0\mid m(F^u(h))>0\},

so that τ(h)\tau(h) counts the steps that remain before EOS. Correct stopping requires τ\tau to equal the number of output steps the count instruction specifies, and ΔtJK\Delta_t^{JK} carries the difference between the two required hitting times along the trajectory.

Metrics. All accuracies below are measured on held-out sequences. Stopping accuracy is the fraction of sequences that emit EOS at the intended step, content accuracy is the fraction of content tokens that match the target, and complete-sequence accuracy is the fraction of sequences whose entire output, content and EOS position together, matches the target. Where an intervention is applied part way through a rollout, we additionally report continuation accuracy, the fraction of reconstructed rollouts whose decoded output matches the one the unmodified network produces.

How the hidden state encodes stopping

Fix a model M\mathcal{M}, let tct_c denote the step at which it reads the count instruction, and consider two hidden-state rollouts of the same input sequence under count instructions JJ and KK. The hidden states are identical until the count instruction creates the preactivation difference

ΔatcJK=Wx(cJ−cK).\Delta a_{t_c}^{JK}=W_x(c_J-c_K).

Because subsequent inputs are zero, the autonomous recurrence needs to carry this difference until EOS is output.

RNN

During autonomous output, both RNN trajectories satisfy

at+1κ=Whhtκ,ht+1κ=tanh⁡(at+1κ),κ∈{J,K}.a_{t+1}^{\kappa}=W_hh_t^{\kappa},\qquad h_{t+1}^{\kappa}=\tanh(a_{t+1}^{\kappa}),\qquad\kappa\in\{J,K\}.

For scalar preactivations aa and bb, we define the tanh secant

s(a,b)={tanh⁡(a)−tanh⁡(b)a−b,a≠b,1−tanh⁡2(a),a=b.s(a,b)= \begin{cases} \dfrac{\tanh(a)-\tanh(b)}{a-b}, & a\ne b,\\ 1-\tanh^2(a), & a=b. \end{cases}

Now let Dt+1JKD_{t+1}^{JK} be the diagonal matrix containing these secants for the corresponding preactivations in the two rollouts, and define the resulting change in the EOS comparisons as ΔgtJK=g(htJ)−g(htK)=CΔtJK\Delta g_t^{JK}=g(h_t^J)-g(h_t^K)=C\Delta_t^{JK}. This difference then follows the exact recurrence

Δt+1JK=Dt+1JKWhΔtJK,Δgt+1JK=CDt+1JKWhΔtJK.(4)\begin{aligned} \Delta_{t+1}^{JK}&=D_{t+1}^{JK}W_h\Delta_t^{JK},\\ \Delta g_{t+1}^{JK}&=CD_{t+1}^{JK}W_h\Delta_t^{JK}. \tag*{(4)} \end{aligned}

This identity holds for any pair of trajectories. From its injection through Wx(cJ−cK)W_x(c_J-c_K), this difference moves along the autonomous hidden trajectory before affecting the EOS comparisons (Figure 6). Across the five RNNs, the first two components of an uncentered PCA basis obtained from separate sequences contain on average 74.5%74.5\% of the energy of ΔtJK\Delta_t^{JK}. Projected onto these components, the differences on new sequences trace coherent curves, with the two adjacent count changes following nearly the same path (Figure 1A).

Figure 6. Propagation of the RNN trajectory difference from the instruction to the EOS comparisons for model seed 5 and count change 4→24 \to2. A, the preactivation response Wx(cJ−cK)W_x(c_J-c_K) and resulting hidden response at the instruction. Hidden units are ordered by the step of their largest response on separate sequences. B, the hidden response during autonomous output, with horizontal lines at copy boundaries. C, the norm of ΔtJK\Delta_t^{JK}, its effect on the EOS comparisons, and the difference between the two EOS margins, each scaled by its largest absolute value.

The transport of ΔtJK\Delta_t^{JK} consists of a fixed mixing step followed by a state-dependent scaling step. At each step, WhW_h first mixes the current difference. Then, Dt+1JKD_{t+1}^{JK} scales each hidden unit by a value in (0,1](0,1]. Since s(a,b)s(a,b) depends on the two current states, this scaling changes with the stored sequence, the repeat count, and the output position. To test which of these dependencies matter, we replace the secants in Equation (4) and propagate ΔtJK\Delta_t^{JK} with the replaced secants. Note that these replacements run open loop: the propagated difference is decoded through WyW_y, but the network itself never receives the resulting states. Removing the scaling altogether, or moving each secant to a different hidden unit, leaves almost no correct continuation (Table 1 and Figure 2). Shifting the same secants in time retains a higher continuation accuracy. Secants taken from other sequences retain more, and their fitting-set mean at the same step yields the highest continuation accuracy of all. Hidden-unit identity and temporal order thus have the largest effects, while the values themselves are largely shared across sequences. The fixed mixing by WhW_h admits a further separation into normal and nonnormal parts, neither of which reproduces the stopping times on its own (Section A).

Secants used in the transportContinuation accuracy
Exact1.000
Identity, DtJK=ID_t^{JK}=I0.000
Permuted across hidden units0.010
Mean across hidden units, same sequence and step0.091
Circularly shifted in time0.316
Taken from other sequences0.669
Fitting-set mean, same step0.859

Table 1. Replacements for the tanh secants DtJKD_t^{JK} in Equation (4), evaluated across the five RNNs and the three count changes. Each replacement propagates ΔtJK\Delta_t^{JK} over the final copy of the shorter target. We add the result to the trajectory of the longer count and decode it through WyW_y, without rerunning the network. Continuation accuracy is the fraction of these reconstructions whose output matches the output of the unmodified network. The exact row is a numerical check.

Figure 2. Transport of the RNN trajectory difference. A, mean tanh secants for model seed 5 and count change 4→24 \to2, with hidden units ordered by the step of strongest attenuation on separate sequences. B, multiplication by WhW_h expands the trajectory difference before tanh attenuates it. Faint lines denote model means and solid lines denote the cohort mean. C, EOS margins along the two trajectories. D, continuation accuracy under the exact secants and their controlled replacements.

The transported difference separates trajectories by the number of output steps that remain, which we describe by the current position within the copied sequence and the number of copies remaining.

Indexing the output states from t=0t=0, for a count instruction KK let

Hn,pK(S)=h(K−n)L+pK(S),H^{K}_{n,p}(S)=h^{K}_{(K-n)L+p}(S),

with p∈{0,…,L−1}p \in\{0,\ldots,L-1\} the index within the current copy of SS and nn copies of SS remaining. Applying the network’s autonomous transition function to Hn,pK(S)H^{K}_{n,p}(S),

F(Hn,pK(S))={Hn,p+1K(S),p<L−1,Hn−1,0K(S),p=L−1.(5)F\left(H^{K}_{n,p}(S)\right)= \begin{cases} H^{K}_{n,p+1}(S), & p<L-1,\\ H^{K}_{n-1,0}(S), & p=L-1. \end{cases} \tag*{(5)}

Equation (5) describes the change in nn and pp within a single rollout. The position advances until the copy ends, and the number of copies remaining then decreases by one. Because LL is fixed, (n,p)(n,p) is equivalent to the number of output steps remaining, which equals the hitting time τ(Hn,pK(S))=nL−p\tau\left(H^{K}_{n,p}(S)\right)=nL-p of § 3.

Rollouts that began with different count instructions can contain the same value for nn. A model that is asked for four copies of the input, for example, reaches n=2n=2 after completing two copies. The same model asked for three copies reaches n=2n=2 after completing one copy of SS. States that agree on (n,p)(n,p) and SS should then coincide regardless of the count instruction:

Hn,pK(S)≈Hn,pK′(S),K,K′≥n.(6)H^{K}_{n,p}(S)\approx H^{K'}_{n,p}(S),\qquad K,K'\ge n. \tag*{(6)}

We measure this proximity across 5 RNN seeds using 512 held-out sequences. For each model, the distance between the states in Equation (6) is averaged and divided by the mean distance between states with adjacent values of nn (Section A). In all five models, states that agree on (n,p)(n,p) and SS lie close to each other regardless of the count instruction that produced them. Their mean distance is 0.165 of this reference, with full-state cosine between 0.996 and 0.998 (Panel A of Figure 3). The largest ratio of this distance to the distance between adjacent values of nn at any count and position is 0.862.

If we average the hidden states over the LL positions of each copy, we obtain one cycle center per copy. Cycle centers with the same nn lie even closer together, at 0.109 of the distance between cycle centers with adjacent nn. Thus, the states agree more closely once the variation within a copy is removed. Panel B of Figure 3 shows the cycle centers ordered by nn in a PCA projection obtained from separate sequences. The individual states carry the number of output steps remaining in a similarly low-dimensional form once the mean state of each emitted symbol, which dominates their leading principal components, is subtracted. A linear readout from the first two principal components of these centered states then recovers τ\tau on new sequences with R2R^2 between 0.73 and 0.98 across the five models, which exceeds every one of 32 random two-dimensional projections per model (Panel C of Figure 3).

Figure 3. RNN states indexed by the number of output steps remaining. A, distances between states that share (n,p)(n,p) and SS under different count instructions, between states one copy apart, and between states storing other content, each relative to the distance between states one copy apart. Gray lines denote models and the blue line denotes their mean. B, cycle centers projected onto two principal components obtained from separate sequences. Arrows follow one complete copy. C, evaluation states of model seed 3, the model with the median R2R^2, after subtracting the fitting-set mean state of each emitted symbol, projected onto the first two principal components of the fitting states and colored by τ\tau. D, stopping and content accuracy after the affine state transformation of Equation (7) as a function of its rank. E, EOS margins after the native recurrence advances a state by two copies.

Subtracting the cycle center from each state leaves the motion through pp. We call the resulting sequence of centered hidden states across the LL positions of SS a centered path, since it describes how the state moves within one copy. The position-wise average of the centered paths from the other copies of the same held-out sequence recovers 0.872 of the variation of each path. Centered paths of successive copies of the same sequence have a mean cosine of 0.939. Shifting the path by one position drops this cosine to −0.138-0.138, and a path from another sequence gives −0.003-0.003. Thus, the centered path follows pp and depends on SS. This is consistent with a position that advances through SS and resets after each complete copy, while the number of copies remaining decreases. Note that it is equally consistent with a single countdown of τ\tau.

Moving a state from nn to n−1n-1 should remove one copy from the future output. We test this with an affine state transformation learned from separate fitting sequences. For each sequence SiS_i in the fitting set, we take the source state hih_i and the target state hi′h_i' at the same output step under adjacent count instructions:

hi=Hn,0K(Si),hi′=Hn−1,0K−1(Si),δi=hi′−hi.h_i=H^{K}_{n,0}(S_i),\qquad h_i'=H^{K-1}_{n-1,0}(S_i),\qquad\delta_i=h_i'-h_i.

Let hˉ\bar{h} and δˉ\bar{\delta} denote the corresponding means. A rank kk linear map MkM_k then estimates the centered difference from the centered source state, which yields the state transformation

Tk(h)=h+δ^k(h),δ^k(h)=δˉ+Mk(h−hˉ).(7)T_k(h)=h+\hat{\delta}_k(h),\qquad\hat{\delta}_k(h)=\bar{\delta}+M_k(h-\bar{h}). \tag*{(7)}

We evaluate TkT_k on new sequences by giving it only the hidden state hh that lies at a copy boundary, i.e., after one copy has completed and another is started at the next time step. The unchanged RNN outputs EOS one copy earlier when continuing from T128(h)T_{128}(h). The tokens that the RNN copies before EOS are not affected by the transformation (Table 2 and Figure 3). T128T_{128} does not isolate the number of copies remaining as a separate variable, since it acts on the complete hidden state. The complete target is reproduced less often when we only add the mean change of the fitting sequences with the same count and cycle (Table 2). Thus, most of the change that removes one copy is shared by all sequences. We measure how much T128T_{128} depends on the stored sequence by using two further controls. The first control shuffles the target states across the fitting sequences, and the second control replaces hh by the mean target state. Neither control produces a single complete output. The accuracy of the copied content then drops to roughly the level expected by chance over the sixteen content tokens. Both controls keep the −hi-h_i term in δi=hi−−hi\delta_i = h_i^{-} - h_i. Hence, they only remove the relation between the stored sequence and its target state.

InterventionStoppingContentComplete
RNN, affine state transformation T128T_{128} from nn to n−1n-1 copies remaining
Fitted transformation, new sequences0.9950.9940.944
Mean change, same count and cycle0.8290.9760.728
Targets shuffled across sequences0.4560.0490.000
hh replaced by mean target state0.5200.0460.000
SSM, amplitude-preserving phase rotations
Independent rotation per plane0.015–0.0470.999–1.0000.015–0.047
Rotation shared by all planes0.268–0.3161.0000.268–0.316
SSM, fixed direction (r−3)ξ(r-3)\xi at the first output state, r∈{1,2,4,5}r \in\{1,2,4,5\}
Fitted direction ξ\xi0.911–0.9760.998–1.0000.911–0.974
Mean count-instruction difference0.548–0.7240.950–0.9910.547–0.724
Norm-matched random direction0.000–0.0010.953–0.9750.000–0.001

Table 2. Interventions on the stopping mechanism, evaluated on held-out sequences. All accuracies are computed over every rollout. RNN entries are means over the five models, and SSM entries are ranges across the five models. The fixed-direction rows pool the targets r=1,2,4,5r = 1, 2, 4, 5, starting from a rollout of three copies. Since the SSM rotations preserve almost all of the copied content, their complete-sequence accuracy rounds to the same range as their stopping accuracy.

Linear SSM

The linear SSM defined in Equation (3) carries the trajectory difference through a fixed linear recurrence. After uu autonomous output steps,

Δtc+uJK=AuWx(cJ−cK),Δgtc+uJK=CAuWx(cJ−cK).\Delta^{JK}_{t_c+u}=A^uW_x(c_J-c_K), \qquad\Delta g^{JK}_{t_c+u}=CA^uW_x(c_J-c_K).

Each transition block has the form Aq=ρqRot⁡(ωq)A_q=\rho_q\operatorname{Rot}(\omega_q). Let PqP_q select block qq, and let rqJKr_q^{JK} and ϕqJK\phi_q^{JK} be the amplitude and phase of PqWx(cJ−cK)P_qW_x(c_J-c_K). The trajectory difference in this plane is

PqΔtc+uJK=ρqurqJK[cos⁡(ϕqJK+uωq)sin⁡(ϕqJK+uωq)].P_q\Delta^{JK}_{t_c+u}=\rho_q^u r_q^{JK}\begin{bmatrix}\cos(\phi_q^{JK}+u\omega_q)\\ \sin(\phi_q^{JK}+u\omega_q)\end{bmatrix}.

The output matrix sums the contribution vq,uJKv_{q,u}^{JK} from every plane:

Δgtc+uJK=∑q=164ρqurqJKCPq⊤[cos⁡(ϕqJK+uωq)sin⁡(ϕqJK+uωq)].\Delta g^{JK}_{t_c+u}=\sum_{q=1}^{64}\rho_q^u r_q^{JK}CP_q^\top\begin{bmatrix}\cos(\phi_q^{JK}+u\omega_q)\\ \sin(\phi_q^{JK}+u\omega_q)\end{bmatrix}.

The fixed relative phases make the plane contributions cancel during copying and align near the stopping time, increasing the effect of the count instruction on the EOS comparisons. Their coherence ∥∑qvq,uJK∥2/(∑q∥vq,uJK∥22)1/2\lVert\sum_q v_{q,u}^{JK}\rVert_2/(\sum_q\lVert v_{q,u}^{JK}\rVert_2^2)^{1/2} lies between 2.61 and 3.60 at EOS across the five models (Figure 4).

Figure 4. Coherence of the rotation-plane contributions at EOS in each SSM, compared with independent amplitude-preserving rotations of the planes. Stopping and content accuracy under these rotations are given in Table 2.

Amplitude and phase are separable, meaning we can intervene on them independently. Rotating the trajectory difference within each plane on its own (Section A) preserves every plane amplitude and changes only the phases relative to the other planes, which removes almost all of the intended stopping behavior while leaving the copied content intact (Table 2). In contrast, a rotation shared by all planes keeps the phases of the planes relative to each other, i.e., every plane is rotated by the same angle. The model then still stops at the intended step in between 0.268 and 0.316 of the rollouts (Table 2). Thus, the relative phase across planes is necessary for correct EOS timing. Note that it is not sufficient, since the shared rotation keeps every relative phase and still removes most of the intended stopping. Hence, the orientation of the trajectory difference relative to WyW_y matters as well. Both interventions keep the amplitude of every plane fixed. Thus, they cannot be used to test whether the amplitudes contribute to EOS timing.

Because the SSM is linear, the stopping time can also be set directly by adding a fixed direction to the hidden state. For each model, we fit one direction ξ\xi such that adding (r−3)ξ(r-3)\xi to the first output state of a rollout with three copies produces rr copies of SS, followed by EOS. We fit ξ\xi on separate sequences for r∈{1,2,5}r \in\{1,2,5\} by requiring the target token to win every future comparison by a margin of 0.1, and hold out r=4r = 4 (Section A). On new sequences, the same direction sets the number of copies for every rr from 1 to 5, including r=5r = 5, one copy more than the model was trained on (Table 2). The held-out r=4r = 4 is reached in every sequence of every model. The direction meets the pre-specified success criteria, a complete-sequence accuracy of at least 0.75 at every target and 0.9 on average, in four of the five models, and it fails at r=6r = 6. Moving the state along the mean difference that the count instruction itself produces per copy reaches the target less often, and norm-matched random directions almost never do.

What the two mechanisms share

Because both architectures use a linear decoder WyW_y to generate the EOS token, each must associate the count with EOS emission. We find that the models achieve this association via opposite changes in the trajectory difference. We measure its magnitude as ∥ΔtJK∥2\lVert\Delta_t^{JK} \rVert_2 and its visibility through the EOS comparisons as ∥CΔtJK∥2/∥ΔtJK∥2\lVert C\Delta_t^{JK} \rVert_2/\lVert\Delta_t^{JK} \rVert_2. During the last copy of SS, the trajectory difference grows in the RNN by factors between 1.52 and 2.16 and contracts in the SSM by factors between 0.61 and 0.65. Nonetheless, visibility rises in four of the five RNNs, by factors between 1.08 and 5.52, and in every SSM, by factors between 1.12 and 1.29. Every model also reaches EOS with positive alignment to the limiting comparison. The cosines lie between 0.214 and 0.622 in the RNN and between 0.527 and 0.580 in the SSM (Figure 5). The trajectory difference thus has to become visible to CC at the intended step, while its magnitude at that step remains a property of the recurrent transition. A separate comparison shows that choosing among several stopping times requires more principal components than copying with a fixed stopping time (Section B and Figure 7).

Figure 5. Magnitude of ΔtJK\Delta_t^{JK} and its visibility through the EOS comparisons. A, magnitude of ΔtJK\Delta_t^{JK} relative to the start of the last copy. B, visibility through all EOS comparisons. C, cosine with the weakest EOS comparison at the target state. D, coherence of the SSM plane contributions. Faint lines denote models and dark lines denote their means.

Figure 7. Fixed short-copy dynamics under reduced recurrences. A, agreement of the rank three system across five models. B, EOS margins from the full and reduced systems on the mixed sequences. C, contributions of output position and symbols to the exhaustive state table. D, output agreement for the reduced system, random subspaces, and the linearized RNN. Rank three preserves every output in four of the five RNNs and the stopping time in four of the five SSMs.

Discussion

Stopping with an EOS token requires the nonlinear RNNs and linear SSMs studied here to encode a timing signal in their hidden states. Using the repeat-copy task, this signal is visible as a difference between trajectories generated from the same sequence under different repeat counts. The difference persists throughout the model’s autonomously generated output, including steps at which the EOS margin remains negative. Its presence at these steps shows that the hidden dynamics contain information about the future stopping time before this information can affect the current output. The EOS margin mtm_t thus acts as an observable of the effect of the model’s hidden dynamics, which implement the timing computation.

The form of this timing computation depends on the recurrent transition. The RNNs we consider here transport the trajectory difference through repeated mixing by their recurrent matrix WhW_h and state-dependent scaling by the activation function’s secants. Their hidden states carry the remaining step count τ=nL−p\tau= nL - p, which at the fixed sequence length is equivalent to the pair of nn, the number of copies of SS remaining, and pp, the current position within SS, so that our experiments do not determine whether the network represents the two indices separately. An affine state transformation estimated on separate sequences shifts a state at a copy boundary to the states with one fewer copy remaining, and the unmodified network then ends its autonomous output one copy earlier while copying every preceding token correctly.

We find that the linear SSM transports ΔtJK\Delta_t^{JK} through fixed rotation planes. Contributions from these planes cancel in the EOS comparisons during copying before reinforcing near the stopping time. Independent phase rotations preserve the response amplitudes and copied symbols; most of the intended stopping behavior disappears. A shared rotation retains more stopping behavior because it preserves the relative phases. Relative phase across planes and the orientation of the response toward WyW_y therefore both govern when the trajectory difference affects the EOS comparisons. A single fixed direction in the SSM state also sets the stopping time, to any number of copies from one to five.

Limitations

Because LL is fixed, there exists a bijection from (n,p)(n,p) to the remaining step count τ=nL−p\tau= nL - p, making our nested description of the RNN’s dynamics indistinguishable from a single counter of the remaining number of output steps. The affine state transformation, applied at copy boundaries, does not distinguish these either. Varying sequence length independently of repeat count would distinguish the aforementioned axes. Besides, the explicit count tokens and deterministic sequences also make the timing response identifiable. The analysis in this paper concerns instructed halting in small tanh RNNs and linear SSMs. Experiments using gated RNNs and nonlinear SSMs will be needed to determine whether the same forms of transport occur in other recurrent architectures.

Experimental details

The nonlinear RNNs we use to train on repeat-copy have width 128 and use the default dense PyTorch initialization for WhW_h. The RNNs are optimized for 100,000 updates and the SSMs for 5,000, chosen based on how long they took to converge. We train both architectures with Adam [23] using batch size 128, unit gradient clipping, and a cosine learning rate schedule from 10−310^{-3} to 10−510^{-5} over 100,000 updates. Training uses the cross-entropy loss. We do not provide the models with a ground truth signal or output feedback. All training, fitting, evaluation, and control streams are derived from the single data seed 123, and the five models of each cohort use the model seeds 1 to 5.

RNN transport of the trajectory difference. We evaluate five RNN seeds, 1 through 5, for the count changes 3→23 \to2, 4→24 \to2, and 4→34 \to3 from the shorter target’s final copy through EOS. Separate streams supply 256 fitting sequences, 512 evaluation sequences, and the hidden unit permutations, content exchanges, and temporal shifts. The cycle comparison uses the same fitting and evaluation sequences. It reuses eight position-specific maps from the first copy at later copies and includes an absolute time control fitted separately at every output step.

SSM phase interventions. The intervention parameters are fixed in advance, and we evaluate seeds 1 through 5 on new sequences and phase draws. The phase controls rotate ΔtJK\Delta_t^{JK} at 8, 4, or 1 steps before the target EOS, using either an independent rotation for each plane or one common rotation for all planes, with angles drawn uniformly from (−π,π)(-\pi,\pi). Neither control uses output weights or token labels. Stopping accuracy is computed over all rollouts, without filtering. The fixed direction ξ\xi is fitted on 64 sequences and evaluated on 256 new sequences, against 32 norm-matched random directions per model.

Remaining copy geometry. RNN seeds 1 through 5 use the same 256 fitting and 512 evaluation sequences. A wider cohort, again with model seeds 1 through 5, is trained on repeat counts from one to six and evaluated on the same sequences. We estimate the affine state transformation from the fitting sequences and apply it at copy boundaries in the evaluation sequences. The cycle centers and centered paths use the same 512 evaluation sequences, with the principal component axes obtained from the 256 fitting sequences. For each sequence, the history-collapse statistic averages distances between states that share (n,p)(n,p) under different initial counts and normalizes them by the distance between adjacent values of nn. For the remaining-step readout, the symbol means, the principal components, and the linear readout of τ\tau are all estimated on the fitting sequences and evaluated on the evaluation sequences, and the random projections are 32 uniformly drawn orthonormal two-dimensional bases per model. We predict each centered path from the other copies of the same sequence. The controls shift its position by one or substitute a different sequence. Of the wider cohort, four models solve every count from one to six. The fifth, seed 2, does not and is excluded. Two of the four, seeds 1 and 3, show the same state organization and the same causal edit at every count, while the other two keep substantial history dependence at high counts. The main cohort does not exclude any model. The stronger state-transformation controls use the same sequences and a separate stream for the target permutations.

Task compression. The cross-width comparison uses model seeds 1 through 5. The short-copy cohort uses model seeds 6 through 10, since seeds 1 through 5 would reproduce the width-four models of the cross-width comparison. For short copy, the principal component basis contains only (−1,−1)(-1,-1) and (+1,+1)(+1,+1). The reduced rollout is evaluated on the held-out inputs (−1,+1)(-1,+1) and (+1,−1)(+1,-1), against sixty-four seeded Haar control bases per model. All four inputs enter the native competence check and the state factorization.

Transport verification. Using tct_c for the count input step, at step tt, ΔtJK\Delta_t^{JK} changes the EOS comparisons by

ΔgtJK=CDtJKWhDt−1JKWh⋯DtcJKWx(cJ−cK).\Delta g_t^{JK}=CD_t^{JK}W_hD_{t-1}^{JK}W_h\cdots D_{t_c}^{JK}W_x(c_J-c_K).

Across the float32 trajectories in Figure 2, the maximum state and EOS comparison discrepancies are 9.40×10−69.40 \times10^{-6} and 8.43×10−58.43 \times10^{-5}. The discrepancies measure the numerical agreement between the direct network rollout and the exact pairwise factorization.

Normal and nonnormal mixing. The transport in (4) combines fixed mixing by WhW_h with state-dependent scaling by Dt+1JKD_{t+1}^{JK}. To examine the fixed mixing, we obtain a spectrum-preserving normal surrogate WNW_N from the real Schur decomposition by replacing each two-dimensional block with a scaled rotation that has the same conjugate eigenvalues. Its residual Wres=Wh−WNW_{\mathrm{res}}=W_h-W_N separates the response as

Δt+1JK=Dt+1JKWNΔtJK+Dt+1JKWresΔtJK,WNTWN=WNWNT.\Delta_{t+1}^{JK}=D_{t+1}^{JK}W_N\Delta_t^{JK}+D_{t+1}^{JK}W_{\mathrm{res}}\Delta_t^{JK}, \qquad W_N^{\mathsf{T}}W_N=W_NW_N^{\mathsf{T}}.

Across the five RNNs and four fixed Schur orderings, WNW_N and WresW_{\mathrm{res}} have mean Frobenius norms of 0.357 and 0.931 relative to WhW_h. After tanh scaling, their contributions have root mean square norms of 0.863 and 0.391 relative to the complete next response. The normal surrogate alone preserves 0.019 of the required EOS hitting times. The residual alone preserves 0.0004. Their sum recovers every hitting time. Removing either component from the autonomous recurrence reduces complete-sequence accuracy to zero in every model. The normal surrogate carries most of the one-step count response. The residual supplies the correction required for the hidden difference to reach the EOS region.

Advancing a trajectory. For hidden state h=Hn,pK(S)h=H_{n,p}^{K}(S) on an unmodified network trajectory and a desired remaining count 1≤n⋆≤n1\le n^\star\le n, let ℓ=(n−n⋆)L\ell=(n-n^\star)L. Advancing the autonomous recurrence by ℓ\ell steps reaches Hn⋆,pK(S)H^{K}_{n^\star,p}(S), as shown in the following:

Φn→n⋆(Hn,pK(S)):=Fℓ(Hn,pK(S))=h(K−n)L+p+(n−n⋆)LK(S)=Hn⋆,pK(S).(8)\Phi_{n\to n^\star}\left(H^{K}_{n,p}(S)\right) := F^\ell\left(H^{K}_{n,p}(S)\right) = h^{K}_{(K-n)L+p+(n-n^\star)L}(S) = H^{K}_{n^\star,p}(S). \tag*{(8)}

Under Equation (8), the same trajectory advances by complete copies without changing the stored sequence or current position. The remaining stopping time then changes from nL−pnL-p to n⋆L−pn^\star L-p:

τ(Hn,pK(S))=nL−p,τ(Φn→n⋆(Hn,pK(S)))=n⋆L−p.\tau\left(H^{K}_{n,p}(S)\right)=nL-p,\qquad\tau\left(\Phi_{n\to n^\star}\left(H^{K}_{n,p}(S)\right)\right)=n^\star L-p.

At any boundary between copies of SS, the endpoint map Φn→0=FnL\Phi_{n\to0}=F^{nL} reaches the EOS state through the unmodified recurrence. The affine state transformation in Equation (7) approximates the one-copy transition Φn→n−1\Phi_{n\to n-1} from separate sequences and applies the resulting state relation to new content.

The SSM implements this advancement with the fixed matrix AℓA^\ell. For the RNN, let h(0)=hh^{(0)}=h, a(j+1)=Whh(j)a^{(j+1)}=W_hh^{(j)}, and h(j+1)=tanh⁡(a(j+1))h^{(j+1)}=\tanh(a^{(j+1)}). Define α(a)=tanh⁡(a)/a\alpha(a)=\tanh(a)/a for a≠0a\ne0, α(0)=1\alpha(0)=1, and G(a)=diag⁡(α(a1),…,α(ad))G(a)=\operatorname{diag}(\alpha(a_1),\ldots,\alpha(a_d)). Repeated substitution expresses Fℓ(h)F^\ell(h) as a matrix that depends on the current state:

Fℓ(h)=[G(a(ℓ))Wh]⋯[G(a(1))Wh]⏟Aℓ(h)h.F^\ell(h)=\underbrace{\left[G(a^{(\ell)})W_h\right]\cdots\left[G(a^{(1)})W_h\right]}_{A_\ell(h)}h.

The entries of Aℓ(h)A_\ell(h) depend on the intermediate states along the chosen trajectory. For ℓ=16\ell=16, this matrix maps a state with four copies remaining to the state reached two copies later. Across all five models and 512 evaluation sequences per model, the unchanged recurrence produces fifteen negative EOS margins followed by a positive margin on step sixteen. Every intervening content token remains correct except one, which RNN seed 4 also miscopies without the transformation.

The exact cycle transformation defines the target state h⋆=Φn→n⋆(h)h^\star=\Phi_{n\to n^\star}(h). For h≠0h\ne0, a rank-one change to the recurrent matrix sends hh to the successor of this target in one update:

ΔWh⋆=Wh(h⋆−h)h⊤h⊤h.\Delta W_h^\star=\frac{W_h(h^\star-h)h^\top}{h^\top h}.

For this choice, tanh⁡((Wh+ΔWh⋆)h)=tanh⁡(Whh⋆)=F(h⋆)\tanh((W_h+\Delta W_h^\star)h)=\tanh(W_hh^\star)=F(h^\star). After a single edited update reaches F(h⋆)F(h^\star), restoring WhW_h continues along the original dynamics from this successor. The rank-one construction depends on the current state and target. The affine state transformation in Equation (7) is fitted on separate sequences and applies across content.

Reduced dynamics for simpler tasks

We compare fixed short-copy and cued-delay tasks by projecting the first output state onto a principal component basis obtained from separate trajectories, then advancing it through the original recurrence. The smallest rank that reproduces every target token defines the autonomous projection rank. At widths 4, 8, 16, and 32, the median short-copy ranks are 2, 3, 3, and 3 in the RNN and 3, 4, 5, and 4 in the SSM. The corresponding cued-delay ranks are 4, 6, 14, and 17 in the RNN and 4, 7, 11, and 32 in the SSM. The cued-delay rank exceeds the short-copy rank at every width in both architectures under the exact output criterion (Figure 7).

At width four, both architectures solve the short-copy and cued-delay tasks. For the fixed three-output copy program, let μ\mu be the mean of the six RNN states produced by the homogeneous sequences (−1,−1)(-1,-1) and (+1,+1)(+1,+1). The first three singular directions of these states form B∈R4×3B\in\mathbb{R}^{4\times3}, with ξt=B⊤(ht−μ)\xi_t=B^\top(h_t-\mu). The reduced recurrence is

ξt+1=B⊤[tanh⁡(Wh(μ+Bξt))−μ],y^t=arg⁡max⁡vwv⊤(μ+Bξt).\xi_{t+1}=B^\top\left[\tanh\left(W_h(\mu+B\xi_t)\right)-\mu\right],\qquad\hat{y}_t=\arg\max_v w_v^\top(\mu+B\xi_t).

Starting from the first output state, this rank three recurrence reproduces every output for the separate mixed sequences (−1,+1)(-1,+1) and (+1,−1)(+1,-1) in four of the five RNNs. Mean output agreement is 0.768 for random rank three subspaces and 0.933 for a first order linearization of tanh⁡\tanh at μ\mu.

The corresponding SSM reduction replaces tanh⁡(Whh)\tanh(W_hh) with AhAh. At rank three, it preserves the EOS position in four of the five models and reproduces the complete mixed sequences in one. The full four-dimensional state reproduces every output in every model. For output position pp and input symbols a,b∈{−1,+1}a, b \in\{-1, +1\}, the exhaustive state table has the decomposition

hp(a,b)=μp+aup+bvp+abηp.h_p(a,b) = \mu_p + a u_p + b v_p + ab\eta_p.

Here μp\mu_p is the mean state at position pp, upu_p and vpv_p are the two symbol effects, and ηp\eta_p is their interaction. Output position accounts for median fractions 0.759 and 0.743 of centered variation in the RNN and SSM. The corresponding interaction fractions are 0.018 and numerical zero.

References

  1. [1]Andrea Banino, Jan Balaguer, and Charles Blundell. PonderNet: Learning to ponder. arXiv preprint arXiv:2107.05407, 2021. URL https://arxiv.org/abs/2107.05407.
  2. [2]Róbert Csordás, Christopher Potts, Christopher D. Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 248–262. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.blackboxnlp-1.17.DOI
  3. [3]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HyzdRiR9Y7.arxiv.org/abs/1807.03819
  4. [4]Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021. doi: 10.1162/tacl_a_00359.DOI
  5. [5]Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJg7KhVKPH.
  6. [6]Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179–211, 1990. doi: 10.1207/s15516709cog1402_1.DOI
  7. [7]Michael Figurnov, Artem Sobolev, and Dmitry Vetrov. Probabilistic adaptive computation time. Bulletin of the Polish Academy of Sciences: Technical Sciences, 66(6):811–820, 2018. doi: 10.24425/bpas.2018.125928.DOI
  8. [8]Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 9574–9586, 2021.arxiv.org/abs/2106.02997
  9. [9]Satchel Grant, Noah D. Goodman, and James L. McClelland. Emergent symbol-like number variables in artificial neural networks. Transactions on Machine Learning Research, 2025. URL https://arxiv.org/abs/2501.06141.
  10. [10]Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016. URL https://arxiv.org/abs/1603.08983.
  11. [11]Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. arXiv preprint arXiv:1410.5401, 2014. URL https://arxiv.org/abs/1410.5401.
  12. [12]Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems, volume 28, 2015.arxiv.org/abs/1506.02516
  13. [13]Andreas Grivas, Nikolay Bogoychev, and Adam Lopez. Low-rank softmax can have ungrammable classes in theory but rarely in practice. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6738–6758, 2022. doi: 10.18653/v1/2022.acl-long.465.DOI
  14. [14]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. URL https://arxiv.org/abs/2312.00752.
  15. [15]Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC.arxiv.org/abs/2111.00396
  16. [16]Doron Haviv, Alexander Rivkind, and Omri Barak. Understanding and controlling memory in recurrent neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2663–2671. PMLR, 2019. URL https://proceedings.mlr.press/v97/haviv19a.html.arxiv.org/abs/1902.07275
  17. [17]Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent orthogonal networks and long-memory tasks. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2034–2042. PMLR, 2016. URL https://proceedings.mlr.press/v48/henaff16.html.arxiv.org/abs/1602.06662
  18. [18]John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1275.DOI
  19. [19]John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138. Association for Computational Linguistics, 2019. doi: 10.18653/v1/N19-1419.DOI
  20. [20]Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems, volume 28, 2015. URL https://proceedings.neurips.cc/paper/2015/hash/26657d5ff9020d2abefe558796b99584-Abstract.html.arxiv.org/abs/1503.01007
  21. [21]Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015. URL https://arxiv.org/abs/1506.02078.
  22. [22]Arjun Karuvally, Peter DelMastro, and Hava T. Siegelmann. Episodic memory theory of recurrent neural networks: Insights into long-term information storage and manipulation. In Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML), volume 221 of Proceedings of Machine Learning Research, pages 371–383. PMLR, 2023. URL https://proceedings.mlr.press/v221/karuvally23a.html.
  23. [23]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https://arxiv.org/abs/1412.6980.
  24. [24]Mingbo Ma, Renjie Zheng, and Liang Huang. Learning to stop in structured prediction for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1884–1889. Association for Computational Linguistics, 2019. doi: 10.18653/v1/N19-1187.DOI
  25. [25]Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/d921c3c762b1522c475ac8fc0811bb0f-Abstract.html.arxiv.org/abs/1906.10720
  26. [26]Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212–223. Association for Computational Linguistics, 2018. doi: 10.18653/v1/W18-6322.DOI
  27. [27]Benjamin Newman, John Hewitt, Percy Liang, and Christopher D. Manning. The EOS decision and length extrapolation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 276–291. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.blackboxnlp-1.26.DOI
  28. [28]Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26670–26698. PMLR, 2023. URL https://proceedings.mlr.press/v202/orvieto23a.html.arxiv.org/abs/2303.06349
  29. [29]Jimmy Smith, Scott Linderman, and David Sussillo. Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems. In Advances in Neural Information Processing Systems, volume 34, pages 16700–16713, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/8b77b4b5156dc11dec152c6c71481565-Abstract.html.arxiv.org/abs/2111.01256
  30. [30]Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Ai8Hw3AXqks.arxiv.org/abs/2208.04933
  31. [31]Pavel Sountsov and Sunita Sarawagi. Length bias in encoder decoder models and a case for global conditioning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1516–1525. Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1158.DOI
  32. [32]David Sussillo and Omri Barak. Opening the black box: Low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Computation, 25(3):626–649, 2013. doi: 10.1162/NECO_a_00409.DOI
  33. [33]Yilin Yang, Liang Huang, and Mingbo Ma. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054–3059. Association for Computational Linguistics, 2018. doi: 10.18653/v1/D18-1342.DOI

Paper details

vlnc:W_6ex9h5vw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents