betaFeedback

WorksComputer Science

Internalizing Chain-of-thought into Looped Transformers

Abstract

Chain-of-thought (CoT) enables constant-bit-size transformers to perform general-purpose computation through generated tokens (Li and Wang, 2025a), whereas looped transformers carry out the same computation by recurrence in hidden states (latent thought). We construct a family of constant-bit-size looped transformers that interpolates between CoT and latent thought. For a prescribed loop count L=L(n)L=L(n) and a fixed integer k≥1k\ge1 satisfying s(n)1/k=Ω(L)s(n)^{1/k}=Ω(L), our transformer simulates a time-t(n)t(n), space-s(n)s(n) Turing machine using a context window of O(s(n)/L)O(s(n)/L) and O(t(n)s(n)1/k/L)O(t(n)s(n)^{1/k}/L) generated tokens, reducing both resources by a factor of LL relative to the corresponding unlooped queue-machine simulation. The construction also holds with softmax attention at constant bit-size, under the finite-precision kernel specified in this paper.

Paper details

vlnc:W_wb4jenw2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools, Paper Assistant Tool (PAT)

View statement
Looking for related papers…