betaFeedback

WorksComputer Science

HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

Abstract

We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single "omni" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it.

\begin{axis}[
  width=0.9\linewidth, height=6.4cm,
  xmode=log, xmin=3e9, xmax=1.2e14, ymin=36, ymax=59,
  xlabel={Pretraining tokens (log scale)}, ylabel={7-task 0-shot average},
  label style={font=\small, color=inkmuted}, tick label style={font=\footnotesize, color=inkmuted},
  axis line style={black!35}, tick style={black!35},
  grid=major, grid style={draw=black!7},
  legend style={draw=none, fill=white, font=\footnotesize, at={(0.02,0.98)}, anchor=north west},
  legend cell align=left,
  every node near coord/.append style={font=\scriptsize},
]
\addplot[only marks, mark=*, mark size=2.2pt, color=refgray, draw=white, line width=0.4pt] coordinates {
  (4e12,56.29) (3.6e13,54.78) (2e12,49.34) (6e12,48.09) (3e11,45.34) (1.8e11,43.61) (5e10,42.23) (3e11,38.60)};
\addlegendentry{Reference models (same harness)}
\addplot[only marks, mark=*, mark size=2.8pt, color=hobbyblue, draw=white, line width=0.4pt] coordinates {(1e10,42.97) (4e10,44.05)};
\addlegendentry{HobbyLM-130M / 500M (same harness)}
\addplot[only marks, mark=o, mark size=2.8pt, line width=1.1pt, color=hobbyblue] coordinates {(1e11,47.54)};
\addlegendentry{HobbyLM-1B (evaluated separately)}
\begin{scope}[font=\scriptsize, text=black!80]
\node[anchor=north] at (axis cs:1e10,42.4) {HobbyLM-130M};
\node[anchor=south] at (axis cs:3.2e10,44.6) {HobbyLM-500M};
\node[anchor=north west] at (axis cs:5.2e10,41.9) {MicroLlama-300M};
\node[anchor=south] at (axis cs:1e11,48.2) {HobbyLM-1B};
\node[anchor=west] at (axis cs:2.1e11,43.4) {OPT-350M};
\node[anchor=west] at (axis cs:3.5e11,45.5) {Pythia-410M};
\node[anchor=west] at (axis cs:3.5e11,38.6) {Pythia-160M};
\node[anchor=east] at (axis cs:1.75e12,49.6) {SmolLM2-135M};
\node[anchor=east] at (axis cs:3.5e12,56.4) {SmolLM2-360M};
\node[anchor=west] at (axis cs:7e12,47.9) {Gemma-3-270M};
\node[anchor=north] at (axis cs:3.6e13,54.1) {Qwen3-0.6B};
\end{scope}
\end{axis}

Figure 1. Zero-shot average on seven tasks (HellaSwag, OpenBookQA, WinoGrande, ARC-c, ARC-e, BoolQ, PIQA) against pretraining tokens, for base checkpoints of up to about 1B parameters (HobbyLM-1B: 1.04B total, 305M active). Filled points were run through the same evaluation wrapper (Table 4). HobbyLM-1B (hollow) was evaluated by its builders on the same tasks with the same metric choices, but with lm-eval 0.4.13 rather than 0.4.9.1 (Appendix § 13). HobbyLM-130M and -500M outscore MicroLlama-300M, OPT-350M and Pythia-160M, which saw 1.25–30×\times more tokens; Pythia-410M (300B tokens) remains ahead of HobbyLM-500M.

Introduction

Small language models are usually built by distilling or fine-tuning larger released checkpoints. HobbyLM takes the other route. Every model in the family is trained from random initialization, on public data, with rented serverless or on-demand GPUs and no cluster. We wanted to answer two questions. How far can a sub-billion-parameter sparse MoE get when the architecture and the systems work are chosen carefully? And can one person own every layer of the stack, from the optimizer to the GGUF reader to the desktop application?

Our contributions follow, in the order the work was done. Figure 1 summarizes the base-model results.

  • Architecture (§ 3). A fine-grained sparse MoE decoder whose components were chosen by ablation. QK-normalization turned out to be by far the most important single choice.

  • HobbyLM-500M and inference (§ 4). Pretraining at 130M and 500M; context extension to 8k; chat, masked-diffusion and interpretability variants; and inference through a llama.cpp-compatible export and a from-scratch Rust CPU engine.

  • Multimodality and function calling (§ 5). A TinyLLaVA-style extension to image, video, audio and speech, co-trained into one checkpoint, plus function calling and a text-only computer-use agent driven by accessibility trees.

  • HobbyLM-1B (§ 6). An independent 1B build that adopts the architecture unchanged and adds a curated 100B-token data mixture, stratified shard ordering and an anneal phase. Its builders also release an instruction-tuned model and an on-device macOS computer-use adapter.

  • HobbyLM-Image (§ 7). A separate text-to-image diffusion transformer trained in a deep-compression latent space.

  • Open artifacts. Appendix § 10 lists all released weights and code for every model, including HobbyLM-1B. Appendices § 11–§ 13 and § 15 give training configurations, evaluation settings and prompt formats for the 500M family and the image model; HobbyLM-1B’s are in § 6 and Appendices § 13, § 14 and § 15.

§ 2 places this work among related open models.

Who did what.

This paper has two distinct bodies of work, and we keep them apart throughout. Harish Santhanalakshmi Ganesan designed the HobbyLM architecture, ran the ablations, and trained the 130M and 500M models, and also did all 500M post-training, including the multimodal, function-calling and computer-use models, and wrote the inference engines, the desktop application and the image model (§ 3–§ 5 and § 7). Hariharan M. S. and Prabhurajhan R. took the architecture as specified in § 3 and built HobbyLM-1B in their own code base. They curated the 100B-token training data, built the data pipeline and training infrastructure, ran pretraining and the anneal, and carried out the evaluation and analysis in § 6. Table tab:provenance gives the breakdown by component, and Appendix § 16 gives detailed author contributions.

ComponentContentDesigned / built by
ArchitectureMoE decoder, routing, balancing, normalization, optimizer split, presets, ablationsHarish Santhanalakshmi Ganesan
130M / 500M pretrainingFineWeb training, context extension, lm-eval wrapperHarish Santhanalakshmi Ganesan
500M post-trainingChat (SFT+DPO), diffusion LM, SAE, multimodal omni model, function calling, computer useHarish Santhanalakshmi Ganesan
InferenceGGUF export, Rust engine hobby-rs, desktop app hobby-chat, demo SpaceHarish Santhanalakshmi Ganesan
HobbyLM-ImageDiT design, data curriculum, training, Rust portHarish Santhanalakshmi Ganesan
HobbyLM-1B architectureThe 1B preset of SectionAdopted (Harish Santhanalakshmi Ganesan)
HobbyLM-1B build100B-token mixture, stratified shard order, H200 training, anneal, evaluation, analysisHariharan M. S., Prabhurajhan R.
HobbyLM-1B-Instructinstruction tuning of HobbyLM-1B; GGUF build; patched CPU runtimesHariharan M. S., Prabhurajhan R.
HobbyLM-1B MLXMLX implementation for Apple SiliconPrabhurajhan R.
HobbyLM-1B Computer Use v0.1Mac computer-use LoRAPrabhurajhan R.

Background and related work

Small open language models.

Open model suites below one billion parameters include Pythia [49], OPT [50] and, more recently, SmolLM2 [31], Gemma 3 [51] and Qwen3 [10]. The recent models train on 2–36T tokens and rely heavily on curated data such as FineWeb-Edu [26], DCLM [30] and FineMath [31]. MobileLLM [20] argues that deep, thin shapes suit sub-billion models. HobbyLM works in a far smaller token budget (10–100B tokens) and asks how much careful architecture and systems work can recover. Our training stack borrows from the modded-nanogpt speedrun [25], including the Muon optimizer [21, 22].

Sparse Mixture-of-Experts language models.

Sparsely gated experts [3] were scaled up by Switch Transformers [4] and stabilized by ST-MoE’s router z-loss [5]. DeepSeekMoE [6] introduced fine-grained and shared experts, and fine-grained scaling laws [19] support high granularity. DeepSeek-V3 [7] combined sigmoid routing with auxiliary-loss-free balancing [8]. OLMoE [9] is the closest open blueprint: a fully open MoE with extensive design-choice ablations at 1B active parameters. Ling 2.0 [33] uses a routing and normalization layout close to ours, which we exploit for llama.cpp compatibility. Our contribution is not a new MoE mechanism. It is an ablation-validated combination of these components at 62M–305M active parameters, together with evidence of what transfers to multimodal, agentic and diffusion variants.

Small multimodal and agentic models.

LLaVA [52, 53] established the frozen-encoder-plus-projector recipe, and TinyLLaVA [54] showed that it works with small language models. We follow that recipe with SigLIP [55, 56], Whisper [57] and CLAP [58] encoders, and co-train all modalities in one checkpoint. For function calling we evaluate on BFCL [71] and use the open needle metric code [77]. For GUI agents, much recent work grounds actions in screenshots, for example Aria-UI [68]. Our computer-use model instead reads the operating system’s accessibility tree as text.

Diffusion language models.

Masked diffusion language models [82, 81] generate by iterative unmasking. DiffuLLaMA [84] adapts autoregressive weights to this objective, and Fast-dLLM [83] accelerates decoding with caching and parallel commitment. We apply the adaptation recipe at 500M and measure where parallel decoding pays off.

Efficient text-to-image generation.

Diffusion transformers [86] trained with rectified-flow objectives [91, 92, 93] are now the standard text-to-image backbone. SANA [88] and its deep-compression autoencoder DC-AE [87] cut the token count by compressing 32×32\times per spatial dimension. HobbyLM-Image builds on this latent space to train a 1024px generator on a hobby budget.

Architecture

Designed by Harish Santhanalakshmi Ganesan

HobbyLM is a decoder-only transformer [1] with pre-normalization RMSNorm [13], rotary position embeddings [12], tied input/output embeddings [17] and the GPT-2 byte-level BPE tokenizer [2]. The vocabulary is 50,257 tokens padded to 50,304. The unused padded slots later serve as modality sentinels (§ 5). The feed-forward network of every layer except the first is a fine-grained sparse MoE [3, 4, 6]. Figure 2 gives an overview. We drew the design from the recent small-MoE literature (DeepSeekMoE/V3 [6, 7], OLMoE [9], Qwen3 [10]) and from fine-grained MoE scaling laws [19]. We then kept or dropped each disputed choice according to the ablations in § 3.5.

[
  font=\small,
  box/.style={draw, rounded corners=2pt, minimum height=0.62cm, inner sep=3pt, align=center},
  arr/.style={-{Latex[length=2mm]}, thick},
  node distance=0.32cm]
% left: block stack
\node[box, minimum width=4.4cm] (emb) at (0,0) {Token embedding (tied)};
\node[box, minimum width=4.4cm] (l0) at (0,1.1) {Layer 0: attention + dense SwiGLU};
\node[box, minimum width=4.4cm] (l1) at (0,2.2) {Layers $1..L{-}1$: attention + MoE};
\node[box, minimum width=4.4cm] (fn) at (0,3.3) {Final RMSNorm $\rightarrow$ tied LM head};
\draw[arr] (emb) -- (l0); \draw[arr] (l0) -- (l1); \draw[arr] (l1) -- (fn);
% right top: attention
\node[box, minimum width=5.6cm] (qkv) at (8.2,4.6) {fused $W_{qkv}$: $Q$ ($n_q$ heads), $K,V$ ($n_{kv}$ heads)};
\node[box, minimum width=5.6cm] (qkn) at (8.2,3.7) {per-head RMSNorm on $Q$ and $K$ (QK-norm)};
\node[box, minimum width=5.6cm] (rope) at (8.2,2.8) {RoPE $\rightarrow$ causal SDPA (GQA)};
\draw[arr] (qkv) -- (qkn); \draw[arr] (qkn) -- (rope);
\node[draw, dashed, rounded corners, fit=(qkv)(qkn)(rope), inner sep=5pt, label={[font=\small\bfseries]above:Attention}] (att) {};
% right bottom: MoE
\node[box, minimum width=5.6cm] (rt) at (8.2,1.2) {router (fp32): $s=\sigma(xW_g)$};
\node[box, minimum width=5.6cm] (sel) at (8.2,0.3) {select top-$k$ of $s+b$; gate with raw $s$};
\node[box] (ex) at (6.75,-0.75) {$k$ of $E$ routed experts};
\node[box] (sh) at (9.85,-0.75) {shared expert};
\draw[arr] (rt) -- (sel); \draw[arr] (sel.south -| ex.north) -- (ex.north);
\node[draw, dashed, rounded corners, fit=(rt)(sel)(ex)(sh), inner sep=5pt, label={[font=\small\bfseries]below:Sparse MoE FFN}] (moe) {};
\draw[dashed] (l1.east) -- (att.west); \draw[dashed] (l1.east) -- (moe.west);

Figure 2. The HobbyLM decoder. Every layer is pre-norm. Layer 0 uses a dense SwiGLU FFN; all later layers use a sparse MoE with EE routed SwiGLU experts, top-kk sigmoid routing, an auxiliary-loss-free selection bias bb, and (in the 500M and 1B presets) one shared expert.

Attention

Attention uses grouped-query attention (GQA) [11] with a single fused QKVQKV projection and no biases. The head dimension is decoupled from the model width, following Qwen3 [10]: the 500M model uses 12 query heads of size 128 (12×128=153612\times128=1536) on a 768-wide residual stream. We apply RMSNorm per head to QQ and KK before RoPE (QK-norm [14, 15]). This bounds the attention logits. In our ablations removing QK-norm was by far the most damaging single change (Table 2). The same intervention later fixed the training divergence of the image model (§ 7). Keys and values are expanded to the query heads and passed to scaled-dot-product attention with a causal mask. The masked-diffusion variant (§ 4.4) is the only exception and uses no mask.

Sparse MoE feed-forward

Each expert is a SwiGLU network [16] with a narrow hidden width ff, so granularity G=fdense/fG=f_{\text{dense}}/f is high [19]. Following DeepSeekMoE [6], the first layer stays dense, because early-layer routing is unstable. For a token representation x∈Rdx\in\mathbb{R}^d the router computes sigmoid scores in fp32,

si=σ ⁣(x⊤wi),i=1,…,E,\begin{equation} s_i = \sigma\!\left(x^\top w_i\right), \qquad i=1,\dots,E, \tag*{(1)} \end{equation}

and selects experts using biased scores, as in DeepSeek-V3’s auxiliary-loss-free balancing [8, 7]:

T(x)=TopK⁡i(si+bi,  k),y=∑i∈T(x)si FFNi(x)  +  FFNshared(x).\begin{equation} \mathcal{T}(x) = \operatorname{TopK}_i\big(s_i + b_i,\;k\big), \qquad y = \sum_{i\in\mathcal{T}(x)} s_i\,\mathrm{FFN}_i(x) \;+\; \mathrm{FFN}_{\text{shared}}(x). \tag*{(2)} \end{equation}

The bias bb only affects which experts are selected. The gate weights are the raw scores sis_i, so the balancing signal does not compete with the language-modeling gradient. We do not renormalize the selected gates to sum to one. Renormalization was the default in our initial design, but the ablation (Table 2) favored leaving it off. OLMoE likewise weights the selected experts by their unrenormalized routing probabilities [9]. After each forward pass in training, the bias moves by a fixed step toward balanced load,

bi←bi+u⋅sign⁡ ⁣(cˉ−ci),\begin{equation} b_i \leftarrow b_i + u\cdot\operatorname{sign}\!\big(\bar{c} - c_i\big), \tag*{(3)} \end{equation}

where cic_i is the number of tokens routed to expert ii, cˉ\bar c is the balanced count and u=10−3u=10^{-3}. Under data parallelism the bias is averaged across ranks after every optimizer step. Bias updates are frozen for the final 5% of training. Because the update runs on every forward pass, the effective step per optimizer step is uu times the number of gradient-accumulation micro-batches. We keep two small regularizers. The first is a Switch-style balance term αE∑ifiPi\alpha E\sum_i f_i P_i with α=10−3\alpha=10^{-3} [4]. The second is the ST-MoE router z-loss [5], also with coefficient 10−310^{-3}. We add a z-loss of 10−410^{-4} on the final logits [18].

Dropless expert computation.

Routing has no capacity limit and drops no tokens. The (token, expert) pairs are sorted by expert, and the gathered rows go through a single grouped GEMM per projection (torch._grouped_mm) against the experts’ weights, stored as 3-D tensors W13∈RE×d×2fW_{13}\in\mathbb{R}^{E\times d\times 2f} and W2∈RE×f×dW_2\in\mathbb{R}^{E\times f\times d}. The outputs are scaled by their gates and scattered back with an indexed add. The shared expert runs on every token. At this scale no expert parallelism is needed: the 1B model fits on a single GPU.

Initialization.

All weights are initialized from N(0,0.022)\mathcal{N}(0, 0.02^2). Residual output projections (attention output and every expert’s down-projection) are additionally scaled by 1/2L1/\sqrt{2L}. The router is scaled by a further 0.10.1, following the reduced initialization scale that Switch Transformers recommends for stability [4] and ST-MoE adopts [5].

Optimizer

We use Muon [21, 22] for every matrix-shaped hidden parameter, including the 3-D expert stacks. Each expert’s matrix is orthogonalized independently with a batched Newton–Schulz iteration (5 steps, bf16). The update is rescaled by 0.2max⁡(m,n)0.2\sqrt{\max(m,n)} to match AdamW’s update RMS, as in Moonlight [22], and uses Nesterov momentum 0.950.95 and decoupled weight decay 0.10.1. The router, the tied embedding/output matrix and all vector parameters use AdamW [24] with β=(0.9,0.95)\beta=(0.9,0.95). We keep the router on AdamW because orthogonalization flattens the singular values of the gate matrix and so would discard routing signal. Learning rates are 0.020.02 (Muon) and 3×10−43\times10^{-4} (AdamW) at a 262,144-token batch, scaled by B/262,144\sqrt{B/262{,}144} for larger batches. The schedule is warmup–stable–decay [27]: 100 warmup steps, a constant phase, then linear decay over the last 40% of training to 0.1×0.1\times the peak. Training uses fp32 master weights, bf16 autocast, TF32 matmuls and an fp32 router. The training stack borrows much of its engineering from modded-nanogpt [25], including the data shard format and loader, on-device loss accumulation and asynchronous host-to-device prefetch.

Model presets

Table 1 lists the three presets. Parameter counts come from instantiating each configuration. The models are deep and thin, as MobileLLM recommends for sub-billion models [20], and their total-to-active parameter ratios are 2.3–3.4×\times.

PresetddLLnq/nkv×dhn_q/n_{kv}\times d_hdense FFNE×fE\times ftop-kksharedTotalActive
130M512128/2×648/2\times64153632×19232\times19280140.0M62.1M
500M7681612/3×12812/3\times128230436×32036\times32061500.8M169.0M
1B10242016/8×12816/8\times128281664×22464\times224811037.1M305.0M

Table 1. HobbyLM presets. GG is the dense-to-expert FFN width ratio. All presets use 1 dense leading layer, RMSNorm with ϵ=10−6\epsilon=10^{-6}, RoPE θ=104\theta=10^4 (raised to 10610^6 for the 8k-context 500M), tied embeddings and vocabulary 50,304.

Ablations

We tested each contested choice with a single-knob ablation at 130M (32 experts, top-4 baseline at the time) for 3,000 steps on FineWeb. All runs were launched in parallel, one per H100. Table 2 reports the validation losses. QK-norm dominates: removing it raises the loss by 0.80 nats. The router z-loss and auxiliary-loss-free balancing each matter at the 0.04 level, and so does keeping at least 32 experts. Sigmoid and softmax gating were indistinguishable, and so were 0 and 1 shared experts at 32 experts. We adopted the two winning changes into the defaults: top-8 routing for the 130M preset and no top-kk renormalization. We kept a shared expert in the larger presets, which have more experts, following the literature [6].

VariantChange from baselineVal lossΔ\Delta
topk8top-8 instead of top-43.6441−0.025-0.025
no_renormno top-kk gate renormalization3.6542−0.015-0.015
softmaxsoftmax instead of sigmoid gating3.6677−0.001-0.001
shared1add one shared expert3.6678−0.001-0.001
baseline–3.66880
scale_embmultiply embeddings by d\sqrt{d}3.6775+0.009+0.009
aux_lossclassic auxiliary loss instead of aux-free bias3.7048+0.036+0.036
no_zlossremove router z-loss3.7097+0.041+0.041
experts1616 experts instead of 323.7166+0.048+0.048
no_qknormremove QK-norm4.4715+0.803+0.803

Table 2. Single-knob ablations at 130M, 3,000 steps, FineWeb validation loss (baseline 3.6688).

Throughput options

We ported several throughput techniques from modded-nanogpt behind flags and evaluated them for speed and quality (Table 3). The most useful was a chunked cross-entropy (fused_ce). It computes the vocabulary projection and the loss in row chunks under activation checkpointing, so the full (T×V)(T\times V) fp32 logit tensor is never materialized. The loss and gradients are bit-identical to the baseline on CPU, and peak memory drops by 21%. This allows roughly twice the micro-batch size: at micro-batch 32 on a 1B probe, the baseline ran out of memory, while fused_ce reached about 74k tokens/s on one H100. The Polar Express orthogonalization schedule [23] was neutral at our horizon. An FP8 output head gave no speedup and, in our implementation, passed no gradient through its backward pass. We dropped it.

Optionms/steptokens/sPeak GB130M val loss
baseline222.536,82133.34.0739
fused_ce209.239,16626.44.0681
polar224.836,44533.34.0790
fused_ce+polar210.138,98826.44.0855
FP8 head223.836,60533.810.90 (no gradient)

Table 3. Throughput options on a 1B speed probe (one H100, micro-batch 8, sequence 1024) and quality on a 130M, 1,000-step run.

HobbyLM-500M: pretraining, variants and inference

Work by Harish Santhanalakshmi Ganesan

Pretraining

Both models were trained on FineWeb [26], pre-tokenized with GPT-2 into modded-nanogpt binary shards, on 8×\timesH100 with PyTorch DDP [28] and torch.compile [29]. Each optimizer step covers 2202^{20} tokens (1,048,576).

  • 130M: 9,537 steps (≈10{\approx}10B tokens), about 742 ms/step, about 2 hours. Final validation loss 3.3016.

  • 500M: 38,147 steps (≈40{\approx}40B unique tokens from FineWeb-100B), about 1.21 s/step, about 13 hours. Final validation loss 3.0281.

Peak learning rates were Muon 0.040.04 and AdamW 6×10−46\times10^{-4} (the 4=2\sqrt{4}=2 batch scaling).

Context extension.

For multimodal inputs the 500M model needed a longer context. We continued pretraining at sequence length 2048 for 2,000 steps (≈2.1{\approx}2.1B tokens, learning rate ×0.5\times0.5, θ=104\theta=10^4). Validation loss improved to 2.999 at 2048. A second extension to 8,192 tokens raised RoPE θ\theta to 10610^6 and reached validation loss 2.94 at 8k.

Zero-shot evaluation

We evaluate with the EleutherAI LM Evaluation Harness [35, 40] (v0.4.9.1) through a wrapper that scores log-likelihoods with the MoE and the GPT-2 tokenizer. The wrapper encodes context and continuation jointly, as the harness does for Hugging Face models. We report acc_norm where defined and acc otherwise, all 0-shot, on HellaSwag [41], OpenBookQA [45], WinoGrande [43], ARC [44], BoolQ [46] and PIQA [42]. We ran all reference models through the same wrapper and task set. We checked the wrapper by reproducing published averages for two models: MicroLlama [37] 42.23 against 42.36 on its model card, and TinyLlama-1.1B-3T [36] 52.75 against 52.99. Table 4 shows that the ranking follows pretraining tokens more closely than parameters. The models ahead of HobbyLM-500M were trained on 300B to 36T tokens, 7.5–900×\times our budget. Among models trained on at most 300B tokens, the 130M model (10B tokens) outscores MicroLlama-300M (50B), OPT-350M (180B) and Pythia-160M (300B). On two further tasks, the 500M model scores 40.0 on LAMBADA [47] and 70.3 on SciQ [48].

ModelParamsTokensHellaOBQAWinoARC-cARC-eBoolQPIQAAvg
SmolLM2-360M360M4T56.4138.0059.3538.4868.2262.0871.4956.29
Qwen3-0.6B600M36T53.8134.6058.5638.3158.0869.8270.2954.78
SmolLM2-135M135M2T43.1333.2052.2529.4458.5960.3768.3949.34
Gemma-3-270M270M6T41.2830.6053.4328.3357.0757.7768.1248.09
Pythia-410M410M300B40.1428.8052.9623.9845.4558.5067.5245.34
HobbyLM-500M501M/169M40B41.5629.8051.3022.4442.7650.9869.5344.05
OPT-350M350M180B36.5528.4052.8024.7440.4957.6164.6943.61
HobbyLM-130M140M/62M10B32.5428.2052.1723.4637.7161.3165.4042.97
MicroLlama-300M300M50B34.2230.4051.3023.2139.2352.5764.6942.23
GPT-2 (124M)124M–31.1527.4051.4622.5339.6049.9462.2440.62
Pythia-160M160M300B30.2126.6049.6424.8336.9542.4259.5238.60

Table 4. 0-shot results, all base (not instruction-tuned) checkpoints, all run through the same harness wrapper. Reference models: SmolLM2 [31], Qwen3 [10], Gemma 3 [51], Pythia [49], OPT [50], GPT-2 [2] and MicroLlama [37].

Chat model

We fine-tuned the 500M model on about 1.5M SmolTalk conversations [31] (smol-smoltalk and the conversational, non-reasoning subsets of SmolTalk2), with the loss on assistant turns only. The run lasted 28k steps, ending at loss 1.50. We then applied DPO [69] on UltraFeedback preference pairs [70] with β=0.1\beta=0.1 and learning rate 5×10−75\times10^{-7}, which raised preference accuracy from 0.50 to 0.64. This follows the SmolLM2 recipe. The chat model averages 42.5 on the seven tasks, against 44.0 for the base model. ARC-challenge and WinoGrande held or improved; BoolQ dropped the most (51.0 to 44.4). At decoding time a repetition penalty of 1.3 removes most of the looping we observed with greedy decoding.

Masked-diffusion variant

We converted the autoregressive 500M model into a masked-diffusion language model in the style of LLaDA and MDLM [81, 82]. As in DiffuLLaMA [84], we started from the autoregressive weights and replaced the causal mask with full bidirectional attention. Training draws t∼U(ϵ,1)t\sim\mathcal U(\epsilon,1) per sequence, masks each token independently with probability tt, and minimizes the masked-token cross-entropy weighted by 1/t1/t. A free vocabulary slot serves as the mask token. After about 20B tokens (8×\timesB200) the validation loss was 3.52. A chat variant was then fine-tuned on SmolTalk by masking only the assistant response. We found decoding settings matter more than we expected: greedy commitment with more steps, a repetition penalty of 1.4–1.5 and low-confidence remasking turned incoherent samples into fluent multi-sentence prose. We implemented the semi-autoregressive block sampler with a cached prefix in our Rust engine (§ 4.6). On an H100 the diffusion model generates 117.7 tokens/s (128 tokens, 32 steps), about 2.7×\times the autoregressive model’s ∼44{\sim}44 tokens/s. For comparison, Fast-dLLM reports 2–3.6×\times from KV caching alone, though against a vanilla diffusion baseline rather than an autoregressive one [83]. On a compute-bound laptop CPU the order reverses (∼6.5{\sim}6.5 against ∼52{\sim}52 tokens/s). Whether diffusion decoding wins depends on whether the hardware is memory-bound or compute-bound. The diffusion chat model adopts the conversational register but hallucinates more than the autoregressive chat model.

Interpretability with sparse autoencoders

We trained a top-kk sparse autoencoder [85] with 12,288 latents and k=32k=32 on the residual stream after layer 8 of the 8k-context base model. Activations were streamed online from FineWeb (50M tokens), with no activation cache. The SAE explains 97.5–98.4% of the variance at L0=32L_0=32, with about 20 dead latents. Inspection of top-activating tokens shows concept features (e.g. music, data, devices), synonym clusters (ignored / overlooked / neglected) and syntactic patterns (“% off”, “Table NN”). We labeled the features automatically from their top tokens. They are shown together with live expert-routing maps in our public demo.

Inference

llama.cpp compatibility.

The architecture has the same structure as llama.cpp’s [34] bailingmoe2 model (Ling 2.0 [33]): GQA with per-head QK-norm, sigmoid-gated experts with a selection bias, a shared expert and leading dense layers. We export to GGUF [107] with that tensor layout. The 3-D expert stacks are transposed to llama.cpp’s {d,f,E}\{d,f,E\} order and the selection bias is stored as exp_probs_b. A one-line patch adds support for a head dimension decoupled from the model width. Generations matched the PyTorch reference token for token. Quantizing to Q4_K_M reduced the file from 2.16 GB to 362 MB and raised CPU decoding from about 13 to 43 tokens/s. The vision and speech encoders export to llama.cpp’s multimodal projector formats (phi4 for SigLIP2 and voxtral for Whisper). Release GGUFs declare a custom hobbylm architecture name; the bailingmoe2 export remains available.

hobby-rs, a from-scratch CPU engine.

hobby-rs is a Rust inference engine with no llama.cpp, ONNX or Python dependency at run time. Its parts:

  • a GGUF v3 reader that takes every hyperparameter from the file’s metadata;

  • a GPT-2 BPE tokenizer built from the embedded vocabulary;

  • a KV cache;

  • sparse MoE execution that computes only the selected experts, batching prefill tokens per expert;

  • a SIMD dot product and an int8×\timesint8 Q8_0 kernel in the style of ggml’s maddubs.

Its output is token-for-token identical to llama.cpp. On a 15 W Intel i7-1250U laptop, enabling native AVX2 code generation and the int8 kernel raised Q8 decoding from about 28 to about 52 tokens/s. The engine also implements the diffusion samplers (fixed-schedule and confidence-threshold parallel decoding) and accepts precomputed image or speech embeddings. These let it run the multimodal model on a CPU.

hobby-chat and the demo.

hobby-chat is a desktop application built with Tauri [105] that embeds hobby-rs. It runs the SigLIP2 and Whisper encoders on the CPU through candle [106], and serves an OpenAI-compatible HTTP endpoint that our agent loops use. It also drives a Windows UI-Automation [79] computer-use loop (§ 5.4) and connects to tools through an MCP client. A public Gradio Space provides chat with every variant, image QA, image generation, per-token MoE routing visualizations and SAE feature browsing.

Multimodality and function calling

Work by Harish Santhanalakshmi Ganesan

Design

We extend the 500M model in the style of LLaVA and TinyLLaVA [52, 54]. Frozen encoders produce feature sequences, a small trainable two-layer GELU MLP [104] projects each sequence to the model width, and the projected features replace a sentinel token in the input embeddings. The sentinels live in the unused padded vocabulary slots (IDs 50257–50262), so we never changed the tokenizer. Table tab:encoders lists the encoders.

  • Speech features are stacked two frames at a time, halving Whisper’s 1,500 frames to 750 tokens.

  • Video reuses the image path on sampled frames, keeping the raw per-frame tokens. Pooled frame tokens fell outside the projector’s training distribution and produced unusable output.

The encoder features are pre-computed and cached wherever possible, so training never runs the vision transformer. During splicing, every feature position has its training target masked except the last. The last feature position predicts the token that follows the sentinel.

ModalityFrozen encoderTokens per inputProjector
Image (early)SigLIP2 so400m/14 @384729×\times11521.48M
Image (from joint11)SigLIP2 so400m/16 @5121024×\times11521.48M
Videosame as image, per frame729 or 1024 per frameshared
SoundCLAP HTSATflattened HTSAT feature map,×\times7681.18M
SpeechWhisper-small encoder750×\times1536 (2-frame stacking)1.77M

Staged training

All stages ran on Modal, almost all on 8×\timesH100.

  1. Vision alignment. Projector only, on the LLaVA-Pretrain LCS-558K captions [52, 80], 1,500 steps. The loss fell from 6.66 to 3.02.

  2. Vision instruction tuning. We unfroze the model and fine-tuned on LLaVA-Instruct-150K [52] with COCO images [63], using AdamW at 2×10−52\times10^{-5} (model) and 10−410^{-4} (projector) with a cosine schedule, for one epoch (2,500 steps, final loss 1.58).

  3. Sound. Audio-projector-only alignment on Clotho [60] (3,839 clips, 1,200 steps; loss 4.46 to 2.77). The model distinguishes, for example, birds, crickets, engines and pouring water.

  4. Speech. Speech-projector-only ASR alignment on LibriSpeech train-clean-100 [61] (about 12K utterances, 1,500 steps). This was followed by spoken question answering on VoiceAssistant-400K [62], with the model unfrozen.

  5. Joint co-training. A single interleaved cycle over all modality “paths”, so no modality drifts while others are trained. We grew the cycle across eight runs (Table tab:joint). Each run resumes from the previous one.

RunAdded
joint5image, video, sound, speech (spoken QA), text
joint6single-shot tool calls
joint7text agent loop (query→\tocall→\toresult→\toanswer) and spoken agent loop (TTS'd queries)
joint8text must-call up-weighted
joint9Nemotron agentic SFT (v2) and Nemotron VLM captioning , streamed
joint10OCR (rendered text→\tomarkdown)
joint118k context (θ=106\theta=10^6) and SigLIP2 @512 (1,024 tokens per image)
joint12SmolTalk chat, mobile actions , Aria-UI desktop grounding ; 18 paths, 13 modalities/tasks
Results.

Table 5 lists the multimodal results. POPE [64], GQA [65] and VQAv2 [66] were run with lmms-eval [67] through a custom model adapter. Our chat-trained model answers in sentences, so strict single-word exact match under-scores it. We report a normalized exact match (leading yes/no for POPE; the salient content word not already in the question for GQA and VQAv2) and, for the final model, also containment scoring. POPE exposes the clearest weakness: the stage-2 model answers “yes” to every probe, so accuracy is 50% and recall is 1.0. Speech recognition with only the 1.77M-parameter projector trained, and the language model frozen, reaches 28.54% WER on LibriSpeech test-clean. Most errors are homophones (francs→\tofranks). Two zero-shot behaviors appeared without any training on the combination:

  • The model grounds a spoken question in an image (“how many people?” →\to “three girls”).

  • Video works through the image path with no video training data.

CheckpointBenchmark (scoring)Score
Vision stage 2 (384px, 2k ctx)POPE accuracy / F150.0 / 66.7
GQA (normalized exact match)27.85
VQAv2 val (normalized exact match)37.29
joint12 (512px, 8k ctx)GQA (strict / containment)20.6 / 39.2
VQAv2 val (strict / containment)28.2 / 47.0
Speech stage 1 (projector only)LibriSpeech test-clean WER ↓\downarrow28.54

Table 5. Multimodal results of the 500M model. For the stage-2 model, GQA and VQAv2 use 2K-question subsets and POPE the full 9K set.

Function calling

We train single-shot function calling in the prompt format TOOLS: [schemas] \n USER: … ASSISTANT: followed by a JSON list of calls. We score it with the metrics of the open needle benchmark code [77]: JSON parse rate, name F1, argument and value accuracy, parameter hallucination and exact match. The first model was fine-tuned on 268K single-call examples from Nemotron-Agentic-v1 [72] for 4,000 steps. These were the rows whose first tool call fits in 2,048 tokens. We then tried two levers (Table 6).

  • Grammar-constrained decoding. A character-level JSON state machine restricts tokens to valid tool names and argument keys and leaves values free. It removed parameter hallucination (3.8% to 0.0%) but left exact match unchanged. The remaining errors were semantic: a valid but wrong tool, or a value formatted differently from the reference (us vs. US).

  • Token-weighted loss. Weighting the loss 3×\times on names, 2×\times on values and 1.5×\times on keys gave about one point everywhere.

Adding cleaner data mattered more. Training on Nemotron, BitAgent [75] and a small multi-call set [76] (about 816K examples, 6,000 steps) gave 59.0% exact match and 100% name F1 on BitAgent held-out data. The same model scores 37.3% on Nemotron and 5.3% on the multi-call set. Most of the spread between datasets reflects how hard each evaluation set is.

ModelEval setJSONName F1ArgsParam. halluc. ↓\downarrowExact
Nemotron SFT, greedyNemotron (400)93.282.251.83.838.5
+ grammarNemotron (400)–––0.038.5
+ weighted loss + grammarNemotron (400)94.383.452.90.639.5
Combined data + grammarBitAgent–100.0–0.059.0
Combined data + grammarNemotron––––37.3

Table 6. Function calling on held-out data (needle-style metrics, %). “Grammar” is grammar-constrained decoding.

BFCL and the call/abstain tension.

On BFCL v3 [71] (AST scoring, 150 examples per category, grammar-constrained), the combined model scores:

  • 22.7 on simple and 29.3 on multiple;

  • about 0 on parallel, because the training data contains only the first call of each conversation;

  • 77.8 on relevance, but only 15.3 on irrelevance.

Adding multi-turn trajectory data taught abstention: irrelevance rose to 90.7 with a 21% trajectory mix and to 95.3 with trajectories only. However, it also collapsed must-call accuracy (simple 22.7 to 0.7 to 0.0). At 500M parameters, rigid calling and abstention trade off against each other. Our explanation is that the prose summaries in the trajectories teach the model that an assistant turn is usually text. They do not teach it to tell matching tools from mismatched ones. For deployment we therefore use a forced-call policy whenever tools are supplied. The model’s reply is seeded with [, the predicted name is snapped to the closest valid name, and the JSON is repaired if needed.

Computer use from accessibility trees

Our GUI agent reads text only: it never sees pixels. The screen is serialized from the Windows UI-Automation tree [79] as lines of the form [ControlType] "Name" (state). The model emits one of 12 actions as JSON: click, double-click, right-click, hover, type, key press, scroll, drag, select, set value, open app and wait. The action names its target element by name and control type rather than by coordinates. Because a 2,048-token context could not hold raw trees (58% of 100-element trees overflowed), we cap each tree at 45 elements, interactive controls first, identically at training and inference. We built the training data in three generations, with Gemini [78] as the instruction writer and our own validators as filters:

  1. synthetic screens of 30 application archetypes (7,200 training rows);

  2. instructions written over 28 harvested real Windows trees (1,810 rows);

  3. multi-step goals expanded into GOAL / DONE / NEXT rows, with a 13th finish action (1,472 rows).

Every generated action is checked against the tree: elements that are not present are rejected, and wrong control types are corrected. Table 7 gives held-out results for the released model. Parameter hallucination is zero by construction of the data, and most exact-match misses are ambiguous values (“give it a minute” →\to wait(60) vs. a reference of 7). On long goals, per-step errors compound, and the agent has no mechanism to recover from a wrong step. In the omni model (joint7 onward) the same tool skills also work from speech. For example, a spoken weather question is answered by calling the tool, observing the result and summarizing it in one checkpoint.

SplitJSONName F1Value acc.ExactParam. halluc.
Planning (multi-step goals)96.594.7–82.60.0
Grounding (real application trees)∼96{\sim}9695.59178.40.0
Grounding (synthetic screens)10090.788.672.50.0

Table 7. Held-out computer-use results of HobbyLM-Computer-Use (%).

HobbyLM-1B: adopting the architecture

Work by Hariharan M. S. and Prabhurajhan R. The architecture is the 1B preset of § 3, designed by Harish Santhanalakshmi Ganesan

HobbyLM-1B was built in a separate, private code base (revision 86dbb90). The public repository https://github.com/fuellabs-ai/hobbylm-1b documents the released models; it does not contain the training code. Table tab:adopt separates what was adopted from what was added. Several analyses in this section only make sense in light of specific architectural choices: router precision follows from sigmoid routing with a selection bias at 64 experts, and the micro-batch trade-off follows from per-forward bias updates. Where that is the case, the text links back to § 3.

Adopted unchanged1B preset (Table ); sigmoid top-8-of-64 routing with aux-loss-free bias (u=10−3u=10^{-3}, frozen at 95%); shared expert; QK-norm; GQA; router z-loss10−310^{-3}, balance loss10−310^{-3}, final-logit z-loss10−410^{-4}; chunked cross-entropy; fp32 router; Muon/AdamW split andB\sqrt{B}learning-rate scaling; warmup--stable--decay schedule; shard format
Added by the 1B build100B-token curated mixture; tokenizer study; stratified shard ordering; anneal phase on high-quality data; 4×\timesH200 DDP training configuration; checkpoint-backup automation; evaluation; data-order and router-precision analyses; instruction tuning; macOS computer-use adapter

Data

Mixture.

HobbyLM-1B was trained on a schedule of 100B tokens drawn from five sources (Table 8):

  • FineWeb-Edu [26], as pre-tokenized GPT-2 shards [108];

  • DCLM-baseline [30];

  • codeparrot-clean [32], used as a fallback because bigcode/starcoderdata was gated for the team’s access token at the time; it is mostly Python;

  • FineMath-4plus [31];

  • an anneal set of Cosmopedia-v2 from the SmolLM-Corpus [110] and FineMath-4plus.

Validation uses one held-out FineWeb shard [109].

SourceDatasetShardsTokensSharePhase
Educational webFineWeb-Edu (shards 000001–000600)60060B60%Main
Filtered webDCLM-baseline-1.015015B15%Main
Codecodeparrot-clean10010B10%Main
MathFineMath-4plus10010B10%Main
AnnealCosmopedia-v2 (3.5B) + FineMath-4plus (1.5B)505B5%Anneal

Table 8. HobbyLM-1B training data.

Tokenizer.

The team kept the GPT-2 tokenizer, which preserved direct comparability with the 130M experiments.

Shards and ordering.

All data was pre-tokenized into 100M-token shards in the format of § 3: a 1,024-byte header (magic 20240520, version, token count) followed by uint16 token IDs, with documents separated by the end-of-text token. Documents of 80 characters or fewer were dropped. Shards are consumed whole, so their order determines what the model sees over time. The team uses a seeded, stratified order that interleaves the four main-phase families in proportion to their shard counts. They verified the order on all 950 main-phase shards before launch (Table 9). § 6.4.0.1 shows why this matters.

ThirdShardsEduDCLMCodeMath
131763.09%15.77%10.73%10.41%
231763.09%15.77%10.41%10.73%
331663.29%15.82%10.44%10.44%
Target63.2%15.8%10.5%10.5%

Table 9. Family mix in each third of the 1B main-phase shard order.

Tokens consumed vs. unique.

The schedule consumes 95,36795{,}367 steps × 1,048,576\times\,1{,}048{,}576 tokens ≈100.0\approx 100.0B tokens.

  • The main phase (81,062 steps, ≈85.0{\approx}85.0B tokens) draws from a 95B-token pool.

  • The anneal (14,305 steps, ≈15.0{\approx}15.0B tokens) makes about three passes over the 5B anneal set.

The anneal’s FineMath portion streams from the start of the same split used for the main math shards, so the two likely overlap by about 1.5B tokens. The team therefore estimates 88.5–90B unique tokens.

Training

Table tab:config1b gives the configuration. The learning-rate schedule spans all 95,367 steps. The main phase stops at step 81,062. The anneal resumes from that checkpoint with the same optimizer state and schedule position, and changes only the data. At the resume, the Muon learning rate was 0.0175, confirming that the cooldown continued rather than restarted.

Hardware4×\timesNVIDIA H200 SXM, one node
ParallelismPyTorch DDP via torchrun; gradient all-reduce only on the last micro-step
Sequence / micro-batch / accumulation1,024 / 32 per GPU / 8
Tokens per optimizer step1,048,576
Precisionbf16 autocast, fp32 master weights, TF32 matmuls, fp32 router logits
Losschunked cross-entropy; final-logit z-loss10−410^{-4}; router z-loss10−310^{-3}; balance loss10−310^{-3}
Compilationwhole-model torch.compile
Peak learning rateMuon 0.04, AdamW6×10−46\times10^{-4}(base 0.02 /3×10−43\times10^{-4}, scaled by 2)
Muon / AdamWNesterov 0.95, weight decay 0.1, 5 Newton--Schulz steps /β=(0.9,0.95)\beta=(0.9,0.95), weight decay 0.1
Schedule100-step warmup; constant; linear decay over the last 40% to0.1×0.1\timespeak
Gradient clipping1.0
Expert balancingaux-loss-free bias,u=10−3u=10^{-3}, frozen at step 90,598 (95%)
Seed1337
Compute.

Main pretraining took about 68 h and the anneal about 8 h of wall-clock time on four GPUs: about 76 h in total, or about 304 GPU-hours. That total excludes the 130M experiments, failed launches, sanity checks, evaluation and post-training. Billing for the instance, including data staging and checks, was about $1,167.

Engineering issues.
  • Pre-step hang. On the first provider, every launch hung before step 0. The model code contained two nested torch.compile decorators inside the custom operators of the experimental FP8 output head (§ 3.6). The head was not enabled in the 1B configuration. After both decorators were removed, compiled runs worked on a second provider. Because the environment also changed, the exact cause was not established.

  • Checkpoint storage. Checkpoints (weights plus optimizer state, about 8.5 GB each) filled local storage after an upload quota was hit. The backup process now deletes local checkpoints only after a confirmed upload.

Results

We report the builders’ release evaluation of the final base model (Table 10): lm-eval 0.4.13, fp32, 0-shot, all examples of each split, with acc_norm for HellaSwag, ARC, PIQA and OpenBookQA and acc for WinoGrande and BoolQ. These are the same metric choices as Table 4, which used lm-eval 0.4.9.1. The seven-task average is 47.54. Like Table 4, it mixes acc and acc_norm; the builders’ model card reports no average for that reason.

TaskMetricFinal (step 95,367)
HellaSwagacc_norm43.66 ±\pm 0.49
PIQAacc_norm67.85 ±\pm 1.09
WinoGrandeacc52.41 ±\pm 1.40
ARC-Easyacc_norm54.76 ±\pm 1.02
OpenBookQAacc_norm35.00 ±\pm 2.14
ARC-Challengeacc_norm29.27 ±\pm 1.33
BoolQacc49.82 ±\pm 0.87
Average47.54

Table 10. HobbyLM-1B base, 0-shot release evaluation (lm-eval 0.4.13, fp32). ±\pm is the lm-eval standard error.

One BoolQ document (id 3154) exceeds the base model’s 1,024-token context and was left-truncated; without it, BoolQ is 49.83. Standard errors and run details are in the base model card.

Analysis

Data order at 130M.

Before the 1B run, the team tested the mixture on the 130M preset (140.0M total, 62.1M active) with 5B tokens: 4,050 main-phase and 720 anneal steps (Table 11). Their first loader expanded each source’s file pattern in order. Even with shuffling enabled, it chose among sources rather than across all shards. In practice it consumed educational web, then DCLM, then code, and never reached math. Validation loss rose from 3.74 to 5.13 at the end of that run, and the curated mixture scored below the FineWeb baseline even after a rescue anneal. With stratified ordering, the same data and model matched the baseline (43.34 vs 42.97, single runs) with half the tokens. Shard order is effectively a training hyperparameter whenever sources are stored separately. A cheap check catches the failure before launch: print the source of the first shards and the mix in each third (as in Table 9). The baseline’s average (42.97) equals that of the 130M FineWeb-10B model in Table 4.

RunDataTokensAverage
BaselineFineWeb10B42.97
Sequential loadercurated mix, consumed source by source5B40.70
Stratified ordercurated mix, interleaved5B43.34

Table 11. 130M data-order experiments, 7-task average; the baseline is the 130M FineWeb-10B model of Table 4.

Router precision.

The released implementation uses fp32 routing. A saved precision diagnostic showed changes in routing statistics under bf16, but did not establish a benchmark degradation. These observations do not demonstrate that fp32 routing is universally necessary. Quantized GGUF builds keep the router in fp32. On seven fixed prompts, Q8_0 reproduced the recorded Hugging Face fp32 outputs on all seven, and Q4_K_M on six; F32 GGUF also matched six. Relative to F32 GGUF, perplexity increased by approximately 0.02% for Q8_0 and 1.1% for Q4_K_M on a corpus composed of evaluation prompts and model-generated answers. These are limited compatibility and quantization checks, not a broad capability evaluation.

HobbyLM-1B-Instruct

Hariharan M. S. and Prabhurajhan R. also fine-tuned the base model into HobbyLM-1B-Instruct, which follows simple instructions, rewrites and extracts text, and makes single function calls in a Python-list format. The released checkpoint is not fine-tuned directly from the 1,024-token base (Appendix § 14, Table 15). Its lineage first extends the base to 2,048 tokens over 550 steps, then applies a 100-step function-calling pilot, followed by a 150-step extension to 4,096 tokens with plain RoPE (θ=104\theta=10^4, no RoPE scaling) that starts from the pilot’s weights. The resulting checkpoint (R2) failed its recorded 4K retrieval evaluation, so training at sequence length 4,096 should not be read as demonstrated retrieval across that context.

R2 initializes a four-stage SFT path: 500 recovery steps over 15 data buckets; 250 continuation steps over 18 buckets; 2,000 further steps over 19 buckets to step 2,750; and a final 700-step Broad-Instruct continuation over 19 buckets, four of them rebuilt. The SFT step counter starts at the recovery stage, so step 3,450 counts 3,450 SFT optimizer steps along the selected branch, excluding the 800 extension and pilot steps. The preceding continuation run also produced checkpoints up to step 4,750; these are not ancestors of the release. Checkpoint metadata and the inspected trainer are consistent with optimizer, token-counter and sampler state being restored between SFT stages, with selected buckets reinitialized during data migrations. The final stage’s launch command and the exact trainer file executed remotely were not preserved. The released weights are identical to the raw step-3,450 checkpoint (280 of 280 tensors match).

Public SFT sources include SmolTalk [31] subsets for dialogue, rewriting, constraints and summarization; Nemotron-SFT-Agentic-v2 [39]; and callable examples derived from Glaive function-calling v2 [38], ToolACE [112] and xLAM-60k [111]. Other buckets contain internally generated math, Python, grounded-answering, instruction-following, structured-output and tool-use examples; the upstream sources of the semantic-abstention bucket are not fully traced. Table tab:sftmix1b lists the prepared final-stage datasets and their configured sampling shares; these describe the mixture, not the number of distinct examples consumed. Training uses 4,096-token sequences and 256 sequences per optimizer step. The step-3,450 checkpoint records about 3.62B processed token positions and 78.1M supervised tokens (SFT plus replay) over the selected path; neither is a count of unique tokens. The optimizer, objective and loss-masking details are in Appendix § 14. The model has no preference or safety tuning. It is released with GGUF (Q4_K_M, Q8_0, F32) and MLX builds (Appendix § 10).

Table 12 compares the two models under the same evaluation (lm-eval 0.4.13, fp32, 0-shot, all examples). Changes are observed differences only; we ran no significance test. CommonsenseQA and MMLU are at chance level for both models (20% and 25%). On IFEval (all 541 prompts, official lm-eval 0.4.13 scoring, the model’s chat template, greedy decoding, a 2,048-token response budget), HobbyLM-1B-Instruct scores 22.92 prompt-level strict, 37.77 instruction-level strict, 24.95 prompt-level loose and 40.53 instruction-level loose. 135 of 541 responses hit the 2,048-token limit. Two prompts were flagged by an overlap check against the final-stage SFT mixture; earlier stages and pretraining were not checked.

BenchmarkMetricBaseInstruct
HellaSwagacc_norm43.6646.64
ARC-Easyacc_norm54.7654.12
ARC-Challengeacc_norm29.2728.41
PIQAacc_norm67.8568.88
WinoGrandeacc52.4150.75
OpenBookQAacc_norm35.0036.20
BoolQacc49.8253.36
7-task average47.5448.34
SIQAacc40.8940.33
CommonsenseQAacc19.9818.18
SciQacc / acc_norm82.90 / 75.8082.70 / 77.10
MMLU (0-shot)acc25.3124.78

Table 12. HobbyLM-1B base vs. Instruct, 0-shot (lm-eval 0.4.13, fp32, %). The average is over the seven tasks of Table 4.

CPU inference.

The GGUF builds keep the router and expert-bias tensors in F32. Like the 500M export (§ 4.6), they use llama.cpp’s bailingmoe2 layout with the one-line decoupled-head-dimension patch. Stock Ollama v0.35.1 fails to load them, so the release ships patched llama.cpp and Ollama builds and a double-click Windows chat bundle. On one AMD Ryzen 9 5900HX, the Q4_K_M Instruct model generates about 14 tokens/s with about 1.3 GB peak memory.

Computer use.

Prabhurajhan R. also trained HobbyLM-1B Computer Use v0.1, a LoRA adapter for HobbyLM-1B-Instruct that turns plain-English requests into Mac actions offline. It is released with its app, harness and evaluation suite (Appendix § 10); we do not evaluate it in this paper.

HobbyLM-Image: text-to-image generation

Work by Harish Santhanalakshmi Ganesan

HobbyLM-Image is a separate 333M-parameter diffusion transformer (DiT) [86] trained with a flow-matching objective [92, 91]. It generates 1024×10241024\times1024 images. Training the main lineage cost about $300 of GPU time, about $445 including all experiments. That budget is possible because the model works in a deep-compression latent space rather than a conventional VAE latent.

Architecture

Latent space.

We use the frozen DC-AE f32c32 autoencoder released with SANA 1.1 [87, 88, 89]. It compresses 32×32\times per spatial dimension, so a 102421024^2 image becomes a 32×32×3232\times32\times32 latent. Because the latent is already so compressed, the DiT uses patch size 1: every latent pixel is one token. After applying DC-AE’s published scaling factor, we further normalize latents to unit variance using their empirical standard deviation on the training cache. We found that this stabilized training. In our check, DC-AE round-trip reconstruction was about 26 dB PSNR at 512px.

Two-panel canvas.

The model was designed to both generate and edit images. Its input is a two-panel canvas [ noisy target∣source ][\,\text{noisy target}\mid\text{source}\,] concatenated along width, plus two mask channels (34 input channels). For generation the source panel is blank. For editing it holds the source latent. A learned left/right panel embedding and a task embedding tell the model which case applies. The loss is taken on the left (target) panel only. At 512px the canvas is 512 tokens. At 1024px it is 2,048 tokens.

Blocks.

The DiT has 16 layers of width 1024 with 16 heads and MLP ratio 3 (GELU-tanh). Each block applies:

  1. AdaLN-Zero [86] self-attention, conditioned on the timestep and task embeddings;

  2. ungated cross-attention to the text features;

  3. an AdaLN-Zero MLP.

Positions are learned absolute embeddings. Text conditioning uses the frozen CLIP ViT-L/14 text encoder [90] (prompts padded or truncated to 64 tokens, width 768). The original plan conditioned on our own 500M VLM. We replaced it after the VLM failed explicit grounding probes (§ 5; joint12 scored near the language-prior baseline on strict GQA and VQAv2). As in the language model, we apply RMSNorm to QQ and KK per head before the dot product, in both self- and cross-attention. Without QK-norm the DiT diverged repeatedly. Lowering the learning rate only delayed the divergence, and adding QK-norm removed it.

Training

For a clean latent z1z_1, noise z0∼N(0,I)z_0\sim\mathcal N(0,I) and tt drawn from a logit-normal distribution [93], the model vθv_\theta regresses the rectified-flow velocity:

zt=(1−t) z0+t z1,L=E[ w ∥vθ(zt,t,c)−(z1−z0)∥2].\begin{equation} z_t = (1-t)\,z_0 + t\,z_1,\qquad \mathcal L = \mathbb E\big[\,w\,\|v_\theta(z_t,t,c) - (z_1 - z_0)\|^2\big]. \tag*{(4)} \end{equation}

For edits, the weight ww is 1+1.5 m1+1.5\,m, which up-weights the masked foreground mm. For generation, w=1w=1. To prevent occasional metastable loss jumps, we clamp the per-sample loss at 4.0, skip non-finite batches, clip gradients at 1.0 and keep an EMA of the weights (decay 0.999). For classifier-free guidance [94], the text condition is dropped 10% of the time, and masks are dropped 50% of the time so that mask-free inference stays in distribution. We use AdamW with β=(0.9,0.95)\beta=(0.9,0.95) and weight decay 0.02, linear warmup, and a cosine decay to 10% of the peak learning rate. The peak is 5×10−55\times10^{-5} from scratch and 2×10−52\times10^{-5} for continued training. On a B200 the 512px DiT trains at about 887 images/s, which made a 100M-image-view run cost about $217.

Data curriculum

Each stage was triggered by a specific failure we saw in samples (Table tab:imgdata). The 512px base model learned from CC3M and CC12M [95, 96] plus editing pairs from OmniEdit [98] and MagicBrush [97]. These web-alt-text images taught it stock-photo watermarks. Continued training on ImageNet [100] objects and watermark-free Midjourney-v6 recaptions [102] removed the watermarks. UltraEdit [99] and OmniEdit edits were replayed alongside to retain editing. The Midjourney-heavy mix in turn made objects look painterly and melted. At 1024px we fine-tuned on FLUX-Reason-6M [101], images generated by FLUX.1-dev [103] that are literal and accurate. This corrected object geometry. A caption-filtered subset of images with people and a small amount of Midjourney data were added for human subjects and aesthetics. When we moved from 512px to 1024px, we interpolated the learned position embeddings bicubically. The final checkpoint, 1024flux4, reached a flow-matching loss of 0.76, the lowest in the lineage. The loss was still decreasing when training stopped.

StageRes.DataSteps
Base512CC3M (140K), CC12M (320K), OmniEdit (40K), MagicBrush (2K)--
Continued512ImageNet (50K), Midjourney-v6 recap (1M), UltraEdit (100K), OmniEdit (100K); 20--25% edit replay45K + 120K
Sharpness1024continued mix at 1024px20K
Accuracy1024FLUX-Reason-6M (500K + 150K people), Midjourney (50K)up to 100K per run

Sampling and results

Sampling integrates the learned velocity field with an Euler solver from noise to data, using about 100 steps and a guidance scale around 5:

z←z+1n(vθ(z,t,∅)+γ [vθ(z,t,c)−vθ(z,t,∅)]).\begin{equation} z \leftarrow z + \tfrac{1}{n}\big(v_\theta(z,t,\varnothing) + \gamma\,[v_\theta(z,t,c) - v_\theta(z,t,\varnothing)]\big). \tag*{(5)} \end{equation}

Figure 3 shows samples from the final checkpoint. Single objects and scenes are recognizable and well lit. Fine structure is often wrong (for example the bus and bicycle in the right panel of Figure 3), and people, hands and multi-person scenes are soft. We did not compute FID or GenEval. A 512px sibling checkpoint performs instruction-based editing through the source panel.

Figure 3, panel 1 Figure 3, panel 2

Figure 3. 1024×10241024\times1024 samples from HobbyLM-Image (final checkpoint, CFG ≈5\approx5, about 100 Euler steps), downscaled. Left: scene prompts. Right: object prompts. The object panel shows typical failures in fine structure (bus, bicycle).

CPU implementation

We also ported the full pipeline to hobby-rs with no ML framework dependency. The port consists of:

  • a safetensors reader;

  • the CLIP text encoder and its BPE tokenizer;

  • the DiT and the CFG flow-matching sampler;

  • the DC-AE decoder (residual blocks, linear attention and GLU-MBConv blocks, following the diffusers reference);

  • a PNG writer.

A staged verifier compares every stage with PyTorch. The CLIP features match to a maximum absolute difference of 1.7×10−51.7\times10^{-5}, and the DiT velocities to about 10−510^{-5}. Generating a 102421024^2 image takes minutes on a laptop CPU. We accept this cost in exchange for having no framework dependency.

Limitations

  • Token budget. The 500M model saw 40B tokens and the 1B model 100B, one to three orders of magnitude fewer than the strongest models of similar size. Every downstream variant inherits this ceiling: the models are fluent but hallucinate.

  • Single runs. Every model was trained once, with one seed. The ablations are single runs at 130M and 3,000 steps, so differences of about 0.01 are within noise.

  • Multimodal grounding. POPE shows the stage-2 VLM answering “yes” to every probe. Strict GQA/VQAv2 scores of the final omni model are near language-prior baselines. The stage-2 GQA and VQAv2 numbers use 2K-question subsets.

  • Agents. Function calling is strong on single-call distributions similar to training data but weak on parallel calls. Computer-use errors compound over long goals. The agents act on a real desktop; our harnesses limit execution to a safe action set or a dry-run mode by default.

  • 1B comparability. Same-harness baselines for the 1B model are pending. HobbyLM-1B base has a 1,024-token context. HobbyLM-1B-Instruct is configured for 4,096 tokens, but long-context retrieval is unverified, and its parent checkpoint failed a 4K retrieval test. The HobbyLM-1B training and data-preparation code is not public. No model in the family has been evaluated for safety or bias, and none has been safety-tuned.

  • Image model. We report no FID or GenEval scores. The training images include synthetic images from other generators, which may carry over their biases.

Conclusion

A single carefully ablated sparse-MoE architecture supported an unusually wide range of models on a small budget: base, chat, diffusion, omni-modal, function-calling and computer-use models at 500M, an independently built 1B model, and the same stability recipe (QK-norm) in a separate image generator. Three lessons carried across the whole project:

  • Get the normalization and routing math right first.

  • Treat data order and decoding as hyperparameters.

  • Measure on the hardware you care about, because memory- versus compute-bound regimes can reverse conclusions.

Acknowledgments

We thank Jarvis Labs for GPU credits toward the HobbyLM-1B training run.

AI assistance.

For the HobbyLM-1B work, the authors used Anthropic’s Claude through Claude Code and OpenAI’s Codex to assist with writing and debugging data-preparation, training, evaluation, conversion and release scripts; inspecting and summarizing saved experimental artifacts; and drafting and revising documentation and portions of the manuscript. The authors take responsibility for the manuscript and its claims.

Artifacts

Table tab:artifacts lists every released artifact.

ArtifactLocationDescription
HobbyLM-Baserootxhacker/HobbyLM-Base500M base model, 8k context (Section )
HobbyLM-Chatrootxhacker/HobbyLM-ChatSmolTalk SFT and UltraFeedback DPO
HobbyLM-Computer-Userootxhacker/HobbyLM-Computer-Usefunction calling and accessibility-tree agent (Section )
HobbyLM-Omnirootxhacker/HobbyLM-Omnijoint12 omni checkpoint with vision and speech projectors (Section )
HobbyLM-Diffusionrootxhacker/HobbyLM-Diffusionmasked-diffusion chat model (Section )
HobbyLM-SAErootxhacker/HobbyLM-SAEtop-kkSAE on layer 8 of HobbyLM-Base
HobbyLM-ggufrootxhacker/HobbyLM-ggufGGUF builds of the language models (hobbylm architecture)
HobbyLM-Imagerootxhacker/HobbyLM-Image333M text-to-image DiT (Section )
HobbyLM-Playgroundspaces/rootxhacker/HobbyLM-Playgroundinteractive demo
HobbyLM codegithub.com/harishsg993010/HobbyLMtraining, evaluation, export, hobby-rs, hobby-chat
HobbyLM-1Bharims95/hobbylm-1B1B base model (Section )
HobbyLM-1B-Instructharims95/hobbylm-1B-instructinstruction-tuned 1B (Section )
HobbyLM-1B MLXharims95/hobbylm-1B-mlxInstruct for Apple silicon (MLX)
HobbyLM-1B GGUFharims95/hobbylm-1B-ggufInstruct in Q4_K_M, Q8_0 and F32; base in F32. Needs the patched llama.cpp from the v1.0.0 release; stock Ollama does not load these files
HobbyLM-1B Computer Use v0.1Hobbylm/hobbylm-1B-computer-useLoRA adapter for HobbyLM-1B-Instruct, Mac actions
HobbyLM-1B chatspaces/harims95/hobbylm-1b-chatinteractive demo
HobbyLM-1B codegithub.com/fuellabs-ai/hobbylm-1bmodel code, GGUF converter, runtime patches, computer-use app

Training configuration

The pretraining runs (Table 13) use the optimizer and schedule of § 3. All post-training runs (Table tab:hp-post) use AdamW with β=(0.9,0.95)\beta=(0.9,0.95) and no weight decay, a linear warmup, and a cosine decay to zero. The configuration of HobbyLM-1B is in Table tab:config1b, and that of HobbyLM-Image in Table tab:hp-img. Where the repository does not record a value used by a specific run, we give the training script’s default and mark it with †^\dagger. Values that are neither recorded nor defaulted are marked “–”.

130M500M500M ctx-2k500M ctx-8k
Initializationrandomrandom500M500M ctx-2k
Steps9,53738,1472,000–
Tokens per step1,048,5761,048,5761,048,576–
Tokens≈10{\approx}10B≈40{\approx}40B≈2.1{\approx}2.1B–
Sequence length1,0241,0242,0488,192
RoPE θ\theta10410^410410^410410^410610^6
Peak LR (Muon / AdamW)0.04 / 6×10−46\times10^{-4}0.04 / 6×10−46\times10^{-4}×0.5\times0.5–
Warmup / decay100 steps / linear over the last 40% to 0.1×0.1\times peak
MuonNesterov 0.95, weight decay 0.1, 5 Newton–Schulz steps (bf16)
AdamWβ=(0.9,0.95)\beta=(0.9,0.95), weight decay 0.1
Gradient clipping1.0
Expert biasu=10−3u=10^{-3} per forward pass, frozen after 95% of steps
Precisionfp32 master weights, bf16 autocast, TF32, fp32 router
Step time742 ms1,208 ms––
Final validation loss3.30163.02812.9992.94

Table 13. Pretraining and context-extension configuration (8×\timesH100, DDP, torch.compile).

StageTrainable / dataLRWarmupMicroSteps
Chat SFTfull model; SmolTalk (≈1.5{\approx}1.5M)2×10−5†2\times10^{-5\dagger}150†^\dagger8†^\dagger28,000
Chat DPOfull model; UltraFeedback,β=0.1\beta=0.15×10−75\times10^{-7}--2†^\dagger2,000†^\dagger
Diffusion chat SFTfull model; SmolTalk, sequence 1,0242×10−52\times10^{-5}150†^\dagger168,000
Vision stage 1projector; LCS-558K10−3†10^{-3\dagger}100†^\dagger32†^\dagger1,500
Vision stage 2model + projector; LLaVA-Instruct-150K2×10−52{\times}10^{-5}/10−410^{-4}60†^\dagger82,500
Sound stage 1 (1×\timesH100)audio projector; Clotho------1,200
Speech stage 1 (1×\timesH100)speech projector; LibriSpeech-100------1,500
Spoken QA (1×\timesH100)model + speech projector; VoiceAssistant-400K------1,200
joint5model + all projectors; 5 paths2×10−52{\times}10^{-5}/10−410^{-4}80†^\dagger42,000
joint6 / joint7+ tools / + agent loops2×10−52{\times}10^{-5}/10−410^{-4}80†^\dagger4†^\dagger2,400 / 3,200
Tool SFTfull model; Nemotron (268K)2×10−52\times10^{-5}150†^\dagger84,000
Tool SFT, combinedfull model; Nemotron + BitAgent + multi-call (≈816{\approx}816K)2×10−5†2\times10^{-5\dagger}150†^\dagger8†^\dagger6,000
BackboneDiT, 16 layers, width 1024, 16 heads, MLP ratio 3, patch 1, 333M parameters
Inputtwo-panel DC-AE f32c32 latent plus 2 mask channels (34 channels)
ConditioningCLIP ViT-L/14 text features (64 tokens×\times768), timestep, task
Objectiverectified flow, logit-normaltt, foreground weight1+1.5m1+1.5mfor edits
OptimizerAdamW,β=(0.9,0.95)\beta=(0.9,0.95), weight decay 0.02, gradient clipping 1.0
Learning rate5×10−55\times10^{-5}from scratch,2×10−52\times10^{-5}continued; warmup 1,000 / 500 steps; cosine to 10%
Micro-batch32 (512px pilot), 64 (512px continued), 16 (1024px)
RegularizationEMA 0.999; per-sample loss clamp 4.0; text dropout 10%; mask dropout 50%
SamplingEuler, about 100 steps, CFG about 5

Compute

Table 14 lists the compute recorded for the main runs. All 500M-family and image work ran on Modal serverless GPUs. Short runs (projector alignment, ablations, evaluations) are not listed individually.

RunHardwareWall-clockGPU-hours / cost
130M pretraining (10B tokens)8×\timesH100≈2{\approx}2 h≈16{\approx}16
500M pretraining (40B tokens)8×\timesH100≈13{\approx}13 h≈104{\approx}104
500M context extension to 2k8×\timesH100≈1{\approx}1 h≈8{\approx}8
Tool SFT (4,000 steps)8×\timesH100≈23{\approx}23 min≈3{\approx}3
joint co-training (1,600 steps)8×\timesH100≈40{\approx}40 min≈5{\approx}5
Diffusion LM adaptation (≈20{\approx}20B tokens)8×\timesB200–≈{\approx}$155
HobbyLM-Image (all experiments)H100 / B200–≈{\approx}$445
HobbyLM-1B (100B tokens)4×\timesH200≈76{\approx}76 h≈304{\approx}304

Table 14. Recorded compute for the main runs. GPU-hours are wall-clock time multiplied by the number of GPUs; where only the cost was recorded, we give the cost.

Evaluation details

Language modeling.

For the 130M and 500M models and the reference models (Table 4), we use lm-evaluation-harness v0.4.9.1 [35], 0-shot, through a custom LM wrapper. The wrapper scores log-likelihoods with the MoE and the GPT-2 tokenizer. It encodes context and continuation jointly, matching the harness’s handling of Hugging Face models; this fix changed averages by at most 0.08 points. We report acc_norm where the task defines it and acc otherwise. Reference models were loaded from Hugging Face and run through the same task configuration. HobbyLM-1B and HobbyLM-1B-Instruct were evaluated by their builders with a separately recorded setup: lm-eval 0.4.13, fp32, 0-shot, all examples of each split, with the same acc_norm/acc choices. The wrapper description above does not apply to them. No complete training-data exposure audit was done for these lm-eval tasks.

Vision–language.

We run lmms-eval [67] with a custom model adapter. The image sentinel is placed before the prompt USER: <question>\nASSISTANT:. Decoding is greedy with a repetition penalty of 1.3 and no repeated trigrams, with at most 8 new tokens for POPE and 32 otherwise.

  • Normalized exact match takes the leading yes/no for POPE. For GQA and VQAv2 it takes the salient content word that does not occur in the question.

  • Containment generates the full answer and counts it correct when the normalized gold answer appears in it.

Function calling.

The needle-style metrics [77] compare a predicted JSON call list with the reference:

  • JSON parse rate;

  • name F1 over tool names;

  • argument accuracy;

  • value accuracy;

  • parameter hallucination: the rate of argument keys absent from the schema;

  • exact match of the full call list.

BFCL v3 [71] uses AST matching. Every predicted parameter must exist in the reference, and each value must be in the reference’s list of acceptable values. Parallel calls are matched without regard to order. Relevance requires a call, and irrelevance requires abstention. Computer-use splits are scored with the same metrics on held-out rows, split by screen (single step) or by trajectory (planning).

HobbyLM-1B-Instruct training details

This appendix was prepared from the 1B authors’ investigation of saved checkpoints and configuration files. Counter values are as recorded in the checkpoints, not an independent recount of the remote training stream.

StageIdentitySteps
Base (step 95,367)947e80d9pretraining
2K extension0add7bf0550
Function-calling pilot3f60c420100
4K plain-RoPE extension (R2)4a0c8215150
SFT D: recovery, step 5007f1e7213500
SFT E: continuation, step 750df65ea5a250
SFT F: continuation, step 2,75062bba97b2,000
SFT G: Broad-Instruct release, step 3,450ef62430d700

Table 15. Selected checkpoint path of HobbyLM-1B-Instruct. Identity is the SHA-256 prefix of each stage’s output. The 800 extension and pilot steps precede the SFT step counter. The step-2,751–4,750 continuation and a separate step-3,400 summarization branch are not ancestors of step 3,450.

SFT stageSupervised tokens
D, step 0 →\rightarrow 50011,869,621
E, step 500 →\rightarrow 7505,789,137
F, step 750 →\rightarrow 2,75043,678,658
G, step 2,750 →\rightarrow 3,45016,776,587
Cumulative78,114,003

Table 16. Supervised-token increments (SFT plus replay) recorded in the checkpoints. The final raw-position counter is 3,617,587,200=3,450×1,048,5763{,}617{,}587{,}200 = 3{,}450\times1{,}048{,}576, which includes padded and masked positions.

SFT bucketSourceRowsShare
Nemotron Policy-Bnvidia/Nemotron-SFT-Agentic-v2; consumed file's revision unknown130,41122.6%
Verified math v2internal generation159,85211.6%
Math answer-key v1internal generation120,0008.7%
smol-magpie-ultraSmolTalk, revision 5feaf2fd261,2048.5%
Tool trajectory v1internal generation30,0007.5%
Python coding v2internal generation89,9986.0%
Multispan synthesis v2internal generation89,9865.4%
Semantic abstentionupstream sources not fully traced5,4555.2%
Document knowledge v1internal generation77,7274.7%
Python coding v1internal generation60,0004.0%
Precise instruction v2internal generation48,2363.6%
Callable positivesGlaive function-calling v2 / ToolACE / xLAM-60k, revisions e7f4b645 / 6bda777c / 26d14ebf17,9943.1%
smol-rewriteSmolTalk, 5feaf2fd4,5992.7%
Structured output v2internal generation30,7022.3%
Grounded summarizationSmolTalk smol-summarize, 5feaf2fd2,1431.5%
Short-context tool resultinternal generation1,9781.3%
Tool-result natural answerinternal generation6540.5%
Grounded JSON extractionsmol-summarize emails, constructed targets5280.45%
smol-constraintsSmolTalk, 5feaf2fd5,4000.4%

The final stage rebuilt the magpie, rewriting, summarization and extraction buckets; earlier magpie, rewriting and extraction counts were 261,254, 4,624 and 548. Internal generators have recorded seed tags, but their code versions are not linked. Replay uses pretraining-style shards whose file list and revisions were not audited. A configured share is neither a measured supervised-token share nor a measure of a bucket’s contribution to learning.

Batch and replay.

Each optimizer step uses 256 sequences of 4,096 tokens (micro-batch 8, gradient accumulation 32). Pretraining-style replay is configured at about 0.46% of sampled rows, with an expected contribution of about 20% of supervised tokens; its actual supervised-token share was not measured separately.

Objective.

In the inspected training implementation, SFT supervision covers the response, including its end-of-text token, and replay rows are supervised on all tokens. With replay weight λ=1\lambda=1, the chunked cross-entropy is normalized over the combined supervised tokens, and the auxiliary and z-loss terms are added separately. The displayed logging formula differs from this backward objective, so we describe the inspected implementation.

Optimizer.

Muon updates the matrix parameters and AdamW the rest, with nominal peak learning rates of 8×10−48\times10^{-4} and 1.2×10−51.2\times10^{-5}, AdamW β=(0.9,0.95)\beta=(0.9,0.95) and weight decay 0.1. Stages D and E end at about 0.1×0.1\times the nominal rates, and F and G at 0.5×0.5\times; the final rates are 4×10−44\times10^{-4} (Muon) and 6×10−66\times10^{-6} (AdamW). Equal rates at G’s endpoints do not show that the rate was constant during G; G’s full schedule and launch flags are unknown.

Execution caveats.

The remote trainer’s runtime hash was not recorded. Code history supports the reconstruction, but does not establish byte-for-byte identity with what ran. Part of the sampler coverage comes from a reconstructed step-500 data state, and some buckets have unknown wrap counts, so per-bucket coverage is not exact measured consumption.

Prompt formats

The formats below are those of the 130M/500M family (§ 4–§ 5); HobbyLM-1B-Instruct uses different formats, given at the end of this appendix. All use plain-text role prefixes rather than special tokens. In the 500M training data, each assistant turn ends with the end-of-text token.

Chat.
SYSTEM: <optional system prompt>
USER: <message>
ASSISTANT: <response><|endoftext|>
Function calling.
TOOLS: [{"name": ..., "description": ..., "parameters": {...}}, ...]
USER: <request>
ASSISTANT: [{"name": "<tool>", "arguments": {...}}]<|endoftext|>
TOOL: <result>
ASSISTANT: <answer><|endoftext|>
Computer use (planning).
SCREEN:
[Window] "Calculator"
[Button] "Seven"
...
GOAL: <goal>
DONE: <instruction 1>; <instruction 2>   (or: nothing yet)
NEXT: [{"name": "click", "arguments": {"name": "Seven", "control_type": "Button"}}]
Multimodal.

The projected features replace a sentinel token (image 50257, sound 50258, video 50261, speech 50262) placed before the text turn:

<IMAGE> USER: <question>
ASSISTANT: <answer><|endoftext|>
HobbyLM-1B-Instruct.

Training formats vary across the 1B SFT data buckets. The bundled chat template follows the role-marker convention of the inspected conversational and tool-use examples. Rewriting, summarization and JSON-extraction examples use instruction-and-input formats without these markers, and format verification does not cover every bucket. The role markers are ordinary tokenizer text, not dedicated special tokens. An inspected tool-result example:

<|user|>
Convert 32.0 miles to kilometers.
<|assistant|>
[convert_length(value=32.0, from_unit='miles', to_unit='km')]
<|tool|>
{"value": 32.0, "from_unit": "miles", "to_unit": "km", "converted": 51.5}
<|assistant|>
That's roughly 51.5 km.<|endoftext|>

In this example only the final natural-language answer and the end-of-text token are supervised; the earlier assistant and tool text is context. This does not mean every assistant turn is unsupervised in every bucket.

Author contributions

Harish Santhanalakshmi Ganesan designed the HobbyLM architecture and ran the ablation study. Harish Santhanalakshmi Ganesan also:

  • trained the 130M and 500M models, including context extension;

  • built all 500M post-training: the chat, diffusion-LM, SAE, multimodal omni, function-calling and computer-use models;

  • wrote the GGUF export, the hobby-rs engine, the hobby-chat application and the demo Space;

  • designed and trained HobbyLM-Image;

  • wrote § 1–§ 5 and § 7.

Hariharan M. S. and Prabhurajhan R. adopted the architecture to build HobbyLM-1B. They:

  • curated the 100B-token training data;

  • built the data pipeline and training infrastructure;

  • ran the pretraining and anneal phases;

  • built HobbyLM-1B-Instruct and its GGUF release;

  • carried out the evaluation and analysis reported in § 6, which is based on their draft.

Prabhurajhan R. developed the HobbyLM-1B MLX implementation for Apple Silicon. Prabhurajhan R. built HobbyLM-1B Computer Use v0.1.

References

  1. [1]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762
  2. [2]Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. In OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdfcdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
  3. [3]Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=B1ckMDqlgopenreview.net/forum?id=B1ckMDqlg
  4. [4]Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120), 1–39. https://arxiv.org/abs/2101.03961
  5. [5]Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., & Fedus, W. (2022). ST-MoE: Designing Stable and Transferable Sparse Expert Models. https://arxiv.org/abs/2202.08906
  6. [6]Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., et al. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1280–1297. https://doi.org/10.18653/v1/2024.acl-long.70
  7. [7]DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., et al. (2024). DeepSeek-V3 Technical Report. https://arxiv.org/abs/2412.19437
  8. [8]Wang, L., Gao, H., Zhao, C., Sun, X., & Dai, D. (2024). Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. https://arxiv.org/abs/2408.15664
  9. [9]Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Morrison, J., Min, S., Shi, W., Walsh, P., et al. (2025). OLMoE: Open Mixture-of-Experts Language Models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2409.02060
  10. [10]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., et al. (2025). Qwen3 Technical Report. https://arxiv.org/abs/2505.09388
  11. [11]Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 4895–4901. https://doi.org/10.18653/v1/2023.emnlp-main.298
  12. [12]Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568, 127063. https://doi.org/10.1016/j.neucom.2023.127063
  13. [13]Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems, 32. https://arxiv.org/abs/1910.07467
  14. [14]Henry, A., Dachapally, P. R., Pawar, S. S., & Chen, Y. (2020). Query-Key Normalization for Transformers. Findings of the Association for Computational Linguistics: EMNLP 2020, 4246–4253. https://doi.org/10.18653/v1/2020.findings-emnlp.379
  15. [15]Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., et al. (2023). Scaling Vision Transformers to 22 Billion Parameters. Proceedings of the 40th International Conference on Machine Learning (ICML), 202, 7480–7512. https://arxiv.org/abs/2302.05442
  16. [16]Shazeer, N. (2020). GLU Variants Improve Transformer. https://arxiv.org/abs/2002.05202
  17. [17]Press, O., & Wolf, L. (2017). Using the Output Embedding to Improve Language Models. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 157–163. https://doi.org/10.18653/v1/E17-2025
  18. [18]Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., et al. (2023). PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 24(240), 1–113. https://arxiv.org/abs/2204.02311
  19. [19]Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., et al. (2024). Scaling Laws for Fine-Grained Mixture of Experts. Proceedings of the 41st International Conference on Machine Learning (ICML), 235, 33270–33288. https://arxiv.org/abs/2402.07871
  20. [20]Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., et al. (2024). MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. Proceedings of the 41st International Conference on Machine Learning (ICML), 235, 32431–32454. https://arxiv.org/abs/2402.14905
  21. [21]Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., & Bernstein, J. (2024). Muon: An optimizer for hidden layers in neural networks. In https://kellerjordan.github.io/posts/muon/. https://kellerjordan.github.io/posts/muon/kellerjordan.github.io/posts/muon
  22. [22]Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y., Qin, Y., Xu, W., et al. (2025). Muon is Scalable for LLM Training. https://arxiv.org/abs/2502.16982
  23. [23]Amsel, N., Persson, D., Musco, C., & Gower, R. M. (2026). The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2505.16932
  24. [24]Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1711.05101
  25. [25]Jordan, K., Bernstein, J., Rappazzo, B., @fernbear.bsky.social, Vlado, B., Jiacheng, Y., Cesista, F., Koszarsky, B., & @Grad62304977. (2024). modded-nanogpt: Speedrunning the NanoGPT baseline. In https://github.com/KellerJordan/modded-nanogpt. https://github.com/KellerJordan/modded-nanogptgithub.com/KellerJordan/modded-nanogpt
  26. [26]Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., & Wolf, T. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. Advances in Neural Information Processing Systems 37 (Datasets and Benchmarks Track), 30811–30849. https://doi.org/10.52202/079017-0970
  27. [27]Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., et al. (2024). MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. https://arxiv.org/abs/2404.06395
  28. [28]Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., & Chintala, S. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training. Proceedings of the VLDB Endowment, 13(12), 3005–3018. https://doi.org/10.14778/3415478.3415530
  29. [29]Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24), 929–947. https://doi.org/10.1145/3620665.3640366
  30. [30]Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., et al. (2024). DataComp-LM: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems 37 (Datasets and Benchmarks Track). https://doi.org/10.52202/079017-0455
  31. [31]Ben Allal, L., Lozhkov, A., Bakouch, E., Martín Blázquez, G., Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., et al. (2025). SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model. https://arxiv.org/abs/2502.02737
  32. [32]CodeParrot. (2021). codeparrot-clean. In https://huggingface.co/datasets/codeparrot/codeparrot-clean. https://huggingface.co/datasets/codeparrot/codeparrot-cleanhuggingface.co/datasets/codeparrot/codeparrot-clean
  33. [33]Ling Team, Li, A., Liu, B., Hu, B., Li, B., Zeng, B., Ye, B., Tang, C., et al. (2025). Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation. https://arxiv.org/abs/2510.22115
  34. [34]Gerganov, G., & ggml-org contributors. (2023). llama.cpp: LLM inference in C/C++. In https://github.com/ggml-org/llama.cpp. https://github.com/ggml-org/llama.cppgithub.com/ggml-org/llama.cpp
  35. [35]Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., et al. (2024). The Language Model Evaluation Harness. In https://doi.org/10.5281/zenodo.5371628. https://doi.org/10.5281/zenodo.5371628
  36. [36]Zhang, P., Zeng, G., Wang, T., & Lu, W. (2024). TinyLlama: An Open-Source Small Language Model. https://arxiv.org/abs/2401.02385
  37. [37]keeeeenw. (2024). MicroLlama. In https://huggingface.co/keeeeenw/MicroLlama. https://huggingface.co/keeeeenw/MicroLlamahuggingface.co/keeeeenw/MicroLlama
  38. [38]Glaive AI. (2023). glaive-function-calling-v2. In https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2. https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2huggingface.co/datasets/glaiveai/glaive-function-calling-v2
  39. [39]NVIDIA. (2026). Nemotron-SFT-Agentic-v2. In https://huggingface.co/datasets/nvidia/Nemotron-SFT-Agentic-v2. https://huggingface.co/datasets/nvidia/Nemotron-SFT-Agentic-v2huggingface.co/datasets/nvidia/Nemotron-SFT-Agentic-v2
  40. [40]Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. https://arxiv.org/abs/2405.14782
  41. [41]Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 4791–4800. https://arxiv.org/abs/1905.07830
  42. [42]Bisk, Y., Zellers, R., Le Bras, R., Gao, J., & Choi, Y. (2020). PIQA: Reasoning about Physical Commonsense in Natural Language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 7432–7439. https://doi.org/10.1609/aaai.v34i05.6239
  43. [43]Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020). WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05). https://doi.org/10.1609/aaai.v34i05.6399
  44. [44]Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. https://arxiv.org/abs/1803.05457
  45. [45]Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018). Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2381–2391. https://arxiv.org/abs/1809.02789
  46. [46]Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019). BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2924–2936. https://arxiv.org/abs/1905.10044
  47. [47]Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernández, R. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), 1525–1534. https://doi.org/10.18653/v1/P16-1144
  48. [48]Welbl, J., Liu, N. F., & Gardner, M. (2017). Crowdsourcing Multiple Choice Science Questions. Proceedings of the 3rd Workshop on Noisy User-generated Text (W-NUT), 94–106. https://arxiv.org/abs/1707.06209
  49. [49]Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., et al. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. Proceedings of the 40th International Conference on Machine Learning (ICML), 202, 2397–2430. https://arxiv.org/abs/2304.01373
  50. [50]Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., et al. (2022). OPT: Open Pre-trained Transformer Language Models. https://arxiv.org/abs/2205.01068
  51. [51]Gemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., et al. (2025). Gemma 3 Technical Report. https://arxiv.org/abs/2503.19786
  52. [52]Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual Instruction Tuning. Advances in Neural Information Processing Systems (NeurIPS), 36. https://arxiv.org/abs/2304.08485
  53. [53]Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024). Improved Baselines with Visual Instruction Tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 26296–26306. https://arxiv.org/abs/2310.03744
  54. [54]Zhou, B., Hu, Y., Weng, X., Jia, J., Luo, J., Liu, X., Wu, J., & Huang, L. (2024). TinyLLaVA: A Framework of Small-scale Large Multimodal Models. https://arxiv.org/abs/2402.14289
  55. [55]Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid Loss for Language Image Pre-Training. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 11975–11986. https://arxiv.org/abs/2303.15343
  56. [56]Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. https://arxiv.org/abs/2502.14786
  57. [57]Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of the 40th International Conference on Machine Learning (ICML), 202, 28492–28518. https://arxiv.org/abs/2212.04356
  58. [58]Wu, Y., Chen, K., Zhang, T., Hui, Y., Nezhurina, M., Berg-Kirkpatrick, T., & Dubnov, S. (2023). Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP49357.2023.10095969
  59. [60]Drossos, K., Lipping, S., & Virtanen, T. (2020). Clotho: An Audio Captioning Dataset. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 736–740. https://doi.org/10.1109/ICASSP40776.2020.9052990
  60. [61]Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: An ASR corpus based on public domain audio books. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5206–5210. https://doi.org/10.1109/ICASSP.2015.7178964
  61. [62]Xie, Z., & Wu, C. (2024). Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming. https://arxiv.org/abs/2408.16725
  62. [63]Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014). Microsoft COCO: Common Objects in Context. European Conference on Computer Vision (ECCV), 740–755. https://doi.org/10.1007/978-3-319-10602-1_48
  63. [64]Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., & Wen, J.-R. (2023). Evaluating Object Hallucination in Large Vision-Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 292–305. https://arxiv.org/abs/2305.10355
  64. [65]Hudson, D. A., & Manning, C. D. (2019). GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://arxiv.org/abs/1902.09506
  65. [66]Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017). Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://arxiv.org/abs/1612.00837
  66. [67]Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J. A., Hu, K., Liu, S., Zhang, Y., et al. (2025). LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models. Findings of the Association for Computational Linguistics: NAACL 2025, 881–916. https://doi.org/10.18653/v1/2025.findings-naacl.51
  67. [68]Yang, Y., Wang, Y., Li, D., Luo, Z., Chen, B., Huang, C., & Li, J. (2025). Aria-UI: Visual Grounding for GUI Instructions. Findings of the Association for Computational Linguistics: ACL 2025, 22418–22433. https://doi.org/10.18653/v1/2025.findings-acl.1152
  68. [69]Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems (NeurIPS), 36. https://arxiv.org/abs/2305.18290
  69. [70]Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., et al. (2024). UltraFeedback: Boosting Language Models with Scaled AI Feedback. Proceedings of the 41st International Conference on Machine Learning (ICML), 235, 9722–9744. https://arxiv.org/abs/2310.01377
  70. [71]Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., & Gonzalez, J. E. (2025). The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. Proceedings of the 42nd International Conference on Machine Learning (ICML), 267, 48371–48392.
  71. [72]NVIDIA. (2025). Nemotron-Agentic-v1. In https://huggingface.co/datasets/nvidia/Nemotron-Agentic-v1. https://huggingface.co/datasets/nvidia/Nemotron-Agentic-v1huggingface.co/datasets/nvidia/Nemotron-Agentic-v1
  72. [75]BitAgent. (2024). BitAgent/tool_calling. In https://huggingface.co/datasets/BitAgent/tool_calling. https://huggingface.co/datasets/BitAgent/tool_callinghuggingface.co/datasets/BitAgent/tool_calling
  73. [76]interstellarninja. (2024). tool-calls-singleturn. In https://huggingface.co/datasets/interstellarninja/tool-calls-singleturn. https://huggingface.co/datasets/interstellarninja/tool-calls-singleturnhuggingface.co/datasets/interstellarninja/tool-calls-singleturn
  74. [77]Cactus Compute. (2026). Needle: Automation foundation model for tiny devices. In https://github.com/cactus-compute/needle. https://github.com/cactus-compute/needlegithub.com/cactus-compute/needle
  75. [78]Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., et al. (2023). Gemini: A Family of Highly Capable Multimodal Models. https://arxiv.org/abs/2312.11805
  76. [79]Microsoft. (2017). UI Automation Overview. In https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overview. https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overviewlearn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overview
  77. [80]Liu, H. (2023). LLaVA Visual Instruct Pretrain LCS-558K: a concept-balanced subset of LAION/CC/SBU with BLIP synthetic captions. In https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain. https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrainhuggingface.co/datasets/liuhaotian/LLaVA-Pretrain
  78. [81]Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., & Li, C. (2025). Large Language Diffusion Models. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2502.09992
  79. [82]Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., & Kuleshov, V. (2024). Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems (NeurIPS), 37. https://arxiv.org/abs/2406.07524
  80. [83]Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., & Xie, E. (2026). Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2505.22618
  81. [84]Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., et al. (2025). Scaling Diffusion Language Models via Adaptation from Autoregressive Models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2410.17891
  82. [85]Gao, L., Dupré la Tour, T., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., & Wu, J. (2025). Scaling and Evaluating Sparse Autoencoders. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2406.04093
  83. [86]Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 4195–4205. https://arxiv.org/abs/2212.09748
  84. [87]Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y., & Han, S. (2025). Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2410.10733
  85. [88]Xie, E., Chen, J., Chen, J., Cai, H., Tang, H., Lin, Y., Zhang, Z., Li, M., et al. (2025). SANA: Efficient High-Resolution Text-to-Image Synthesis with Linear Diffusion Transformers. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2410.10629
  86. [89]MIT HAN Lab. (2025). DC-AE f32c32 (SANA 1.1, Diffusers format). In https://huggingface.co/mit-han-lab/dc-ae-f32c32-sana-1.1-diffusers. https://huggingface.co/mit-han-lab/dc-ae-f32c32-sana-1.1-diffusershuggingface.co/mit-han-lab/dc-ae-f32c32-sana-1.1-diffusers
  87. [90]Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning (ICML), 139, 8748–8763. https://arxiv.org/abs/2103.00020
  88. [91]Liu, X., Gong, C., & Liu, Q. (2023). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2209.03003
  89. [92]Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2023). Flow Matching for Generative Modeling. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2210.02747
  90. [93]Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. Proceedings of the 41st International Conference on Machine Learning (ICML), 235, 12606–12633. https://arxiv.org/abs/2403.03206
  91. [94]Ho, J., & Salimans, T. (2022). Classifier-Free Diffusion Guidance. https://arxiv.org/abs/2207.12598
  92. [95]Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018). Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), 2556–2565. https://doi.org/10.18653/v1/P18-1238
  93. [96]Changpinyo, S., Sharma, P., Ding, N., & Soricut, R. (2021). Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3558–3568. https://arxiv.org/abs/2102.08981
  94. [97]Zhang, K., Mo, L., Chen, W., Sun, H., & Su, Y. (2023). MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 36. https://arxiv.org/abs/2306.10012
  95. [98]Wei, C., Xiong, Z., Ren, W., Du, X., Zhang, G., & Chen, W. (2025). OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2411.07199
  96. [99]Zhao, H., Ma, X., Chen, L., Si, S., Wu, R., An, K., Yu, P., Zhang, M., Li, Q., & Chang, B. (2024). UltraEdit: Instruction-based Fine-Grained Image Editing at Scale. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 37. https://arxiv.org/abs/2407.05282
  97. [100]Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A Large-Scale Hierarchical Image Database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 248–255. https://doi.org/10.1109/CVPR.2009.5206848
  98. [101]Fang, R., Yu, A., Duan, C., Huang, L., Bai, S., Cai, Y., Wang, K., Liu, S., Liu, X., & Li, H. (2025). FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark. https://arxiv.org/abs/2509.09680
  99. [102]Photoroom. (2026). midjourney-v6-recap. In https://huggingface.co/datasets/Photoroom/midjourney-v6-recap. https://huggingface.co/datasets/Photoroom/midjourney-v6-recaphuggingface.co/datasets/Photoroom/midjourney-v6-recap
  100. [103]Black Forest Labs. (2024). FLUX.1. In https://github.com/black-forest-labs/flux. https://github.com/black-forest-labs/fluxgithub.com/black-forest-labs/flux
  101. [104]Hendrycks, D., & Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). https://arxiv.org/abs/1606.08415
  102. [105]Tauri Contributors. (2019). Tauri. In https://github.com/tauri-apps/tauri. https://github.com/tauri-apps/taurigithub.com/tauri-apps/tauri
  103. [106]Hugging Face. (2023). Candle: Minimalist ML Framework for Rust. In https://github.com/huggingface/candle. https://github.com/huggingface/candlegithub.com/huggingface/candle
  104. [107]ggml contributors. (2023). GGUF File Format Specification. In https://github.com/ggml-org/ggml/blob/master/docs/gguf.md. https://github.com/ggml-org/ggml/blob/master/docs/gguf.mdgithub.com/ggml-org/ggml/blob/master/docs/gguf.md
  105. [108]Karpathy, A. (2024). fineweb-edu-100B-gpt2-token-shards. In https://huggingface.co/datasets/karpathy/fineweb-edu-100B-gpt2-token-shards. https://huggingface.co/datasets/karpathy/fineweb-edu-100B-gpt2-token-shardshuggingface.co/datasets/karpathy/fineweb-edu-100B-gpt2-token-shards
  106. [109]kjj0. (2024). fineweb10B-gpt2. In https://huggingface.co/datasets/kjj0/fineweb10B-gpt2. https://huggingface.co/datasets/kjj0/fineweb10B-gpt2huggingface.co/datasets/kjj0/fineweb10B-gpt2
  107. [110]Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., & von Werra, L. (2024). SmolLM-Corpus. In https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus. https://huggingface.co/datasets/HuggingFaceTB/smollm-corpushuggingface.co/datasets/HuggingFaceTB/smollm-corpus
  108. [111]Liu, Z., Hoang, T., Zhang, J., Zhu, M., Lan, T., Kokane, S., Tan, J., Yao, W., Liu, Z., Feng, Y., Murthy, R., Yang, L., Savarese, S., Niebles, J. C., Wang, H., Heinecke, S., & Xiong, C. (2024). APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. Advances in Neural Information Processing Systems (Datasets and Benchmarks Track). https://arxiv.org/abs/2406.18518
  109. [112]Liu, W., Huang, X., Zeng, X., Hao, X., Yu, S., Li, D., Wang, S., Gan, W., Liu, Z., Yu, Y., Wang, Z., Wang, Y., Ning, W., Hou, Y., Wang, B., Wu, C., Wang, X., Liu, Y., Wang, Y., … Chen, E. (2025). ToolACE: Winning the Points of LLM Function Calling. International Conference on Learning Representations. https://arxiv.org/abs/2409.00920

Paper details

vlnc:W_26n6tn3w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Gemini (multiple versions), Claude Code, Codex

View statement
Looking for related papers…
Contents