WorksComputer Science
PBit1: One Programmable P-Bit per Parameter - A Finite-Control Representation of Pretrained Networks
Abstract
We introduce PBit1, a finite-control representation of pretrained neural networks that assigns one programmable p-bit to each learned parameter. A 12-bit command sets its probability, a shared affine range maps that probability to the weight domain, and temporal observations reconstruct the weight for inference. Operator-aware range grouping and sensitivity-based sampling allocation adapt the representation to the network. Across five independent weight draws at 32,768 observations per scalar, selected configurations preserve task quality in GPT-2, DistilBERT, and LFM3-1.7B with every learned scalar converted and no fine-tuning. GPT-2 and LFM3 perplexity increase by approximately 0.66% and 0.26%, respectively, while DistilBERT accuracy changes from 87.51% to 87.49%. A sensitivity-based policy reduces GPT-2 reconstruction observations by 41.7% while retaining the quality criterion. Complete-model evaluations reuse reconstructed weights in BF16 inference, and THRML operator experiments examine native sampling and spatial-temporal tradeoffs. Together, these results establish PBit1 as a concrete construction for studying finite-control weight representation and temporal reconstruction in pretrained networks.
Code Research archive
Figure 1. PBit1 separates stochastic-element count from reconstruction precision. The left panel shows how finite probability commands program one logical p-bit per learned scalar. Shared ranges and temporal counts reconstruct weights for reuse in BF16 inference. The right panel shows that selected configurations meet the predefined 2% perplexity-ratio criterion on three complete models at . Points and intervals show five-draw means in log-loss space and conditional 95% intervals. DistilBERT uses categorical perplexity and also meets its accuracy criterion.
Introduction
A programmable probabilistic bit has a binary instantaneous state and a tunable stationary probability. Repeated observations estimate that probability, allowing additional time to improve precision with the same stochastic-element count. This suggests a representation for pretrained networks in which each learned parameter is assigned one logical programmable p-bit. The central question is how faithfully a network can be reconstructed from these elements under finite probability controls and sampling budgets.
We introduce PBit1, which combines one programmable p-bit per learned parameter with shared affine ranges and temporal reconstruction. Encoding sets the probability command, and decoding maps an observed count back to a weight. The shared range determines the estimation scale, while the sampling policy determines the temporal budget. Together, these components convert an existing checkpoint into reconstructed weights without fine-tuning. We denote the one-element-per-scalar assignment by K1.
PBit1 builds on probability-based representations and progressive stochastic computation [7, 16]. Its finite-control construction covers every learned scalar and combines operator-aware shared ranges with sensitivity-based sampling. We evaluate how these choices affect task fidelity, storage, and the number of observations required for reconstruction.
We examine three research questions.
Does the representation preserve the pretrained function? Complete-model evaluations test task quality on GPT-2, DistilBERT, and LFM3-1.7B with every learned scalar converted.
What affects the required sampling budget? Range grouping and sensitivity-based allocation determine how reconstruction work is distributed across the network.
What resources and tradeoffs does the representation introduce? Storage and sampling costs are evaluated alongside native THRML comparisons of one-element and multi-element representations.
The GPT-2 layout study connects these questions by showing how range grouping changes reconstruction quality at a fixed sampling budget.
Complete-model evaluations reconstruct weights once and reuse them in BF16 inference. This makes it possible to study the representation at network scale while controlling the reconstruction process. Direct decoding of the programmed mean provides a reference for the error introduced by temporal sampling, and native THRML experiments examine the corresponding behavior at operator scale.
Background and Related Work
Stochastic representations. Stochastic computing represents numerical values through the statistics of bit streams, with arithmetic and accuracy shaped by stream length and correlation [1]. Bernoulli bit-stream representations also have a long history in neural computation [16]. Probabilistic binary networks use stochastic weight distributions and ensembles [13], while progressive stochastic binarization supports pretrained-network conversion, precision through accumulation, and adaptive computational allocation [7]. PBit1 brings these ideas into a specified finite-control construction with one programmable stochastic element per pretrained scalar, shared affine ranges, and temporal weight reconstruction. Its evaluation follows the complete conversion of three pretrained checkpoints and the effect of range and sampling policies on their task quality.
Post-training quantization. GPTQ uses post-training reconstruction for transformer quantization [6], and AWQ uses activations to protect important weight directions [8]. AdaRound adapts rounding decisions to a data-dependent reconstruction objective [12], while SmoothQuant redistributes quantization difficulty between activations and weights through an equivalent scaling transformation [17]. PBit1's grouping policy also uses activation-dependent sensitivity. Temporal sampling introduces a variance term and an observation budget in addition to the encoding approximation, linking the choice of local range to the time required for reconstruction.
Probabilistic hardware. Programmable p-bits provide tunable stochastic binary units for probabilistic circuits [3]. Magnetic-tunnel-junction implementations have demonstrated connected p-bit networks for integer factorization [2]. Extropic's Z1T work studies sparse transformer-like computation with multi-p-bit representations and temporally accumulated statistics [4]. We use the official THRML sampler [5] to compare one-element and four-element constructions on a pretrained operator bank. These experiments complement the full-model reconstruction study with direct measurements of native sampling behavior.
Method
Affine stationary-probability encoding
For parameter in group , let be the center and the radius. The ideal representation is
With independent samples, the temporal estimate is
For , the worst-case standard deviation is . The corresponding error-based precision measure, , grows as . Here, effective precision describes the sampling error of the scalar estimate.
Finite probability programming
The implementation stores the group minimum and span as FP32 values. This is algebraically equivalent to and , up to floating-point evaluation. For a nonconstant group,
Decoding evaluates the field in FP32 and the sigmoid in FP64, with operation order fixed in the source. Constant groups use and , so sampling does not change their value. For nonconstant groups, finite fields and quantized controls introduce bias, including at the endpoints. Increasing reduces variance around the programmed mean , with the encoding bias set by the finite command.
For a deterministic target and independent samples,
The finite commands and shared ranges are stored alongside the stochastic states. § 6 accounts for these components.
Temporal readout and execution
We compare two execution regimes. R1 reconstructs sampled weights before deterministic inference, and R2 accumulates linear-operator outputs with sampled weights. All complete-model results use R1. We draw and reconstruct , which has the same marginal distribution as averaging independent Bernoulli observations. This avoids generating each observation explicitly.
For fixed inputs and identical weight samples, linearity gives
Native operator tests verify this identity numerically. For fixed inputs and identical samples, R1 and R2 therefore have the same linear output and sampling variance. In R1, the reconstructed matrix is reused across the dataset, so examples share the same weight draw. Nonlinear operations remain outside this averaging identity.
Range and sampling policies
A PBit1 configuration consists of a fixed grouping of the pretrained scalars, their finite probability commands, shared minima and spans, and a temporal budget for each tensor. Changing a group’s membership changes its span and the resulting reconstruction error. Changing its sample budget changes the variance without changing its programmed mean. Both choices configure the representation around the pretrained weights. The experiments below evaluate operator-aware grouping and a gradient-based temporal allocation as configuration policies for this mechanism.
Experimental Setup
Models, data, and numerical execution
Table 1 lists the evaluated checkpoints and tasks. GPT-2 [14] and LFM3 [9] are evaluated on WikiText-2 [11], using their respective tokenizers and fixed article/window layouts. The SST-2-finetuned DistilBERT checkpoint [15] is evaluated on IMDb [10]; its BF16 reference uses the same transfer between tasks. We convert every learned scalar, including biases and normalization parameters. Tied input embeddings and output heads use one shared representation and are counted once.
| Model | Unique parameters | Scored cases | Task |
| GPT-2 | 124,439,808 | 283,225 tokens | WikiText-2 |
| DistilBERT | 66,955,010 | 25,000 reviews | IMDb classification |
| LFM3-1.7B | 1,728,801,536 | 294,442 tokens | WikiText-2 |
Table 1. Complete evaluated checkpoints and task coverage. Token counts denote scored prediction targets.
Full-model inference uses BF16 weights and computation with deterministic settings. GPT-2 scores 586 windows across 62 articles; LFM3 scores 607 windows across the same articles. DistilBERT truncates inputs at 256 tokens and uses batches of 32. Runs execute locally on an NVIDIA DGX Spark/GB10; LFM3 uses CPU master weights and GPU offloading. The supplement specifies the runtime, loss accumulation, checkpoint revisions, and execution details.
Selection and statistical protocol
We use training activations to choose which matrix direction is grouped for each operator. Groups contain 128 consecutive entries after the matrix is flattened in the chosen direction; LFM3 uses groups of seven for depthwise convolution filters and sixteen for normalization tensors. The uniform policy uses 12-bit controls and samples per scalar. Each policy and seed list is fixed before its complete evaluation. The benchmarks had been inspected during earlier experiments, and the reported evaluations use fixed policies and seed lists on these datasets.
Let be the mean BF16 negative log likelihood and the value for independent weight draw . For five seeds, and
The predefined perplexity criterion requires the upper endpoint of to be at most 1.02. This interval measures Monte Carlo variability conditional on the fixed evaluation data. DistilBERT must also have an upper 95% bound on absolute accuracy decline of at most two percentage points. For DistilBERT, categorical perplexity is the exponential of classification cross-entropy. All five prespecified seeds enter the reported statistics. We independently rescore saved per-example or per-token outputs and use a separate SDK to reconstruct weights and replay prespecified neural windows. These checks test implementation reproducibility. The supplement also reports article/seed bootstrap intervals to characterize variability across the evaluated articles.
Results
Complete-model reconstruction feasibility
The selected GPT-2 and LFM3 layouts meet the predefined 2% criterion (Table 2). DistilBERT also meets both criteria, with mean accuracy of 87.49% against 87.51% for BF16 and an upper interval endpoint of 0.084 percentage points for the decline. Its categorical-perplexity ratio is 0.99966 [0.99676, 1.00256]. GPT-2’s original layout exceeds the quality margin at the same sample count, motivating the range-grouping analysis below.
| Condition | BF16 PPL | K1 PPL | PPL ratio [95% interval] |
| GPT-2, original layout | 25.50149 | 27.02231 | 1.05964 [1.04254, 1.07701] |
| GPT-2, selected layout | 25.50149 | 25.66953 | 1.00659 [0.99927, 1.01397] |
| LFM3, selected layout | 18.93015 | 18.97918 | 1.00259 [0.98751, 1.01790] |
Table 2. Five-seed complete-model evaluations at . Perplexities for stochastic language models are exponentials of mean seed NLL. Brackets give conditional Monte Carlo 95% intervals for the ratio.
Deterministic decoding reference. Directly decoding the stored commands gives the finite-controller mean . Table 3 compares this model with BF16 and sampled reconstruction. The controller mean captures the encoding approximation, while the sampled result also includes temporal estimation error and BF16 rounding. For LFM3, the supplementary joint seed/article bootstrap gives a sampled ratio interval of [0.99188, 1.01065].
| Condition | GPT-2 | LFM3 |
| BF16 reference | 25.50149 | 18.93015 |
| Controller mean | 25.47279 | 18.95199 |
| Sampled, | 25.66953 | 18.97918 |
Table 3. Perplexity under deterministic decoding and sampled reconstruction using selected layouts. Sampled values exponentiate mean five-seed NLL; deterministic values are single decoded models.
Configuring reconstruction error
Operator-aware range grouping
For a fixed input and independently sampled weights, the variance of output coordinate is
(11) shows how range grouping affects the operator. Reconstruction variance is weighted by the input activations, so the same weight error can have different effects in different parts of the network. Row-wise and column-wise grouping assign different sets of weights to each range, changing both the programmed means and sampling variances. Orientation selection uses this structure to reduce error in sensitive directions at a fixed element count.
In GPT-2, changing the range orientation brings the reconstruction within the quality margin at (Table 2). The output use of the tied embedding matrix is especially sensitive. On a fixed training diagnostic, restoring that use to BF16 reduces PPL from 58.98 to 46.10, near the 45.80 reference. Restoring the input use alone gives 59.66. Reorienting the shared stochastic matrix gives 45.27 in the same draw, while retaining a single representation for both uses. With clean hidden activations held fixed, the predicted head-logit variance falls from 0.614 to 0.00270, a reduction of approximately 228-fold.
The preferred orientation varies across operators. The same reorientation increases predicted variance by approximately 3.2 times in one examined DistilBERT feed-forward matrix. LFM3 also shows an interaction between local and network-wide sensitivity. The block that is most harmful when substituted into the clean network differs from the block whose restoration helps the fully stochastic network most. The supplement develops these observations through interventions, signal-scale analysis, and conditional output-error covariance.
Sensitivity-based temporal allocation
Once a suitable range layout is chosen, sampling work can also be redistributed. Uniform sampling spends the same number of observations on parameters with different sensitivities. We instead use a first-order proxy that weights reconstruction variance by squared loss gradients. With variance coefficient and parameters in group , the continuous allocation problem is
whose solution has before bounds and rounding. This relationship defines a first-order sampling heuristic. The implemented policy chooses orientations and sample counts from training/calibration data, with groups of sixteen for small normalization and bias vectors. The supplement gives the discrete allocation procedure used with the fixed pretrained weights.
A fresh five-seed GPT-2 evaluation scores all 283,225 targets. The selected policy reduces nominal observations per reconstruction from 4,077,643,628,544 to 2,378,245,865,472, a reduction of 41.7%, while adding 53,088 bytes of range metadata. Every one of the 124,439,808 parameters retains a K1 assignment, and no sample count exceeds 32,768. The PPL ratio is 1.00780 [0.99907, 1.01660], meeting the same quality criterion (Figure 2).
Figure 2. Validated sensitivity-based allocation reduces nominal reconstruction work to 58.324% of the uniform budget. Vertical bars show independent five-seed Monte Carlo intervals. The sampled matrix is reused across inputs.
Spatial tradeoffs and nonideal controls
A stronger spatial baseline
We examine the tradeoff between stochastic-element count and reconstruction error using THRML on an
8 × 768 GPT-2 output-head bank and eight fixed hidden vectors. The K4 control encodes normalized using three clamped coarse bits, , and one probabilistic fractional bit with target . Decoding gives . The fractional command uses the same 12-bit field grid as K1.
At matched scheduled observations, K4 has lower operator error than K1 (Table 4), at the cost of more elements and coarse information. Scheduled observations include clamped nodes; free stochastic updates differ. The three rows use 201,326,592, 50,331,648, and 201,326,592 free updates, respectively. Counting controls and instantaneous states gives 13 bits per K1 scalar and 19 per K4 scalar before shared ranges. Clamped-node reads, holding, and programming complete the corresponding hardware cost model.
| Condition | Elements | All-node observations | Mean MSE | |
| K1 | 6,144 | 32,768 | 201,326,592 | |
| K4 coarse/fractional | 24,576 | 8,192 | 201,326,592 | |
| K4 coarse/fractional | 24,576 | 32,768 | 805,306,368 |
Table 4. Native THRML reoriented-head control, five draws. Scheduled observations count all nodes; free updates count only stochastic nodes. Output MSE is averaged over the fixed operator inputs.
Noise distribution and device mismatch
Gaussian perturbations matched to the finite-controller mean and estimator variance give similar GPT-2 training-diagnostic quality to Bernoulli reconstruction, with five-draw PPL ratios are 1.00406 [0.98855,
Figure 3. Native THRML comparison for the reoriented output-head bank. Bars show empirical five-draw mean output MSE, without confidence intervals. K4 at and K1 at have equal scheduled all-node observations; their free-node update counts differ (Table 4).
1.01981] and 0.99688 [0.98409, 1.00984], respectively. The matched-noise comparison relates the observed task quality to the scale of reconstruction error.
Second, simulated device gain and offset increase GPT-2 PPL relative to ideal reconstruction, with a paired ratio of 1.03229 [1.00038, 1.06522] on 1,024 training targets. An oracle correction using exact device parameters reduces mean squared bias by about 447-fold. Storing two FP32 calibration values per GPT-2 scalar would add approximately 996 MB. The supplement gives the mismatch model, calibration controls, and simulations of correlated samples and drift.
Resource Accounting and Implementation
The K1 representation stores probability commands and shared ranges alongside one stochastic state per parameter. For parameters and groups, packed commands and FP32 ranges require bits. Including one instantaneous state per parameter gives bits. The current implementation stores commands in uint16 arrays, requiring bits before states. At group size 128, the totals including states are 13.5 bits per parameter with packed controls and 17.5 with uint16 controls. Reconstructed BF16 weights and framework memory are additional.
For LFM3, packed commands and ranges would occupy 2,701,541,888 bytes; the implemented uint16 representation occupies 3,565,942,656 bytes. R1 also holds 3,457,603,072 bytes of reconstructed BF16 weights, and retaining the reference model adds another copy. With weight offloading, the maximum recorded GPU allocation is 824,188,928 bytes, in addition to the host-resident data.
A hardware implementation would distribute energy across switching, bias programming, retention, readout, accumulation, and recalibration. Capacity and correlation-adjusted sampling rate would determine how the logical representation is scheduled. If programming is reused over inferences, the per-inference energy can be organized as
R1 amortizes reconstruction across inputs. A native R2 implementation would instead accumulate sampled operator outputs during execution, giving a different schedule for readout and accumulation. The terms above provide a cost model for evaluating those implementations.
Reproducibility. Saved-output rescoring checks the evaluation results, and an independent SDK reproduces weight reconstruction and selected neural windows or batches. These prespecified replays match bitwise on the recorded runtime for GPT-2, DistilBERT, and LFM3. The supplement records their coverage, native checks, and evidence identifiers. The SDK provides finite-controller reconstruction with an optional pinned THRML backend. Research snapshots and validation records are available in the public research repository. Full reproduction also uses the separately stored checkpoints and arrays.
Discussion
PBit1 links reconstruction fidelity to two configuration choices. Shared ranges set the scale of the error, and temporal allocation determines how much sampling is spent reducing it. The GPT-2 results show the impact of both choices at network scale. The head intervention and fixed-activation variance calculation identify a sensitive operator, although the layout change affects encoding bias as well as sampling variance. The native K4 result adds a spatial tradeoff, achieving lower operator error at matched scheduled observations through additional elements and coarse information.
The evaluation covers three checkpoints on two previously inspected tasks. Its five-seed intervals describe reconstruction variability on those fixed datasets, and the focused diagnostics use limited training inputs. Extending the study to untouched tasks, long-context generation, and instruction-following would establish how broadly the selected configurations transfer. Within the finite-control format, additional sampling improves estimation around the programmed mean, while command precision determines the remaining encoding bias. The supplement develops this distinction through a collision example and a counting bound.
A frozen-configuration sampling-budget sweep would map the quality–observation relationship more fully. Comparisons with optimized post-training quantization would connect representation fidelity to deployment costs. Full-model native execution and physical measurements would then extend the current BF16 reconstruction and THRML operator results to complete stochastic implementations, including programming, readout, retention, calibration, and correlated sampling.
Conclusion
PBit1 introduces and validates a concrete finite-control representation of pretrained neural networks, with one logical programmable p-bit per learned scalar, shared affine ranges, and temporal reconstruction. Shared ranges set the reconstruction scale, while temporal observations set estimation precision. The complete-model results establish that selected configurations preserve predefined quality margins on the evaluated checkpoints and tasks without fine-tuning. Sensitivity-based allocation shows that the temporal budget can be reduced while retaining that criterion.
Within the demonstrated regime, PBit1 establishes a finite-control, all-parameter path from pretrained weights to stochastic reconstruction, with measurable spatial, temporal, and fidelity tradeoffs.
Code, data, and declarations. Package: https://github.com/oramadia/pbit1. Research code and experimental records: https://github.com/oramadia/pbit1-research. The manuscript and supplement are included in this package. The original project code is Apache-2.0; third-party model and dataset licenses remain separate. The THRML experiments use Extropic software in simulation. This work is independent of Extropic. See the companion supplement for evidence identifiers, exact protocols, and historical failures.
AI-assisted writing disclosure. ChatGPT assisted with language editing, manuscript organization, reference discovery, and LaTeX formatting. The author takes responsibility for the final manuscript and its scientific content.
References
- [1]A. Alaghi and J. P. Hayes. Survey of Stochastic Computing. ACM Transactions on Embedded Computing Systems, 12(2s):92:1–92:19, 2013.DOI
- [2]W. A. Borders, A. Z. Pervaiz, S. Fukami, K. Y. Camsari, H. Ohno, and S. Datta. Integer Factorization Using Stochastic Magnetic Tunnel Junctions. Nature, 573: 390–393, 2019.DOI
- [3]K. Y. Camsari, R. Faria, B. M. Sutton, and S. Datta. Stochastic p-Bits for Invertible Logic. Physical Review X, 7 (3):031014, 2017.DOI
- [4]Extropic. Z1T: Sparse Transformer-Like Models for Probabilistic Hardware. Technical article, September 4, September 2026.
- [5]Extropic. THRML: Thermodynamic Hypergraphical Models Library. Software and documentation, 2026.
- [6]E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.17323
- [7]D. Hartmann and M. Wand. Progressive Stochastic Binarization of Deep Networks. In Workshop on Energy Efficient Machine Learning and Cognitive Computing, co-located with NeurIPS, 2019.arxiv.org/abs/1904.02205
- [8]J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. In Proceedings of Machine Learning and Systems, volume 6, 2024.DOI
- [9]Liquid AI. LFM3-1.7B Model Card and Checkpoint. Hugging Face model repository, 2026.
- [10]A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, 2011.
- [11]S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer Sentinel Mixture Models, 2016.arxiv.org/abs/1609.07843
- [12]M. Nagel, R. A. Amjad, M. van Baalen, C. Louizos, and T. Blankevoort. Up or Down? Adaptive Rounding for Post-Training Quantization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7197–7206, 2020.arxiv.org/abs/2004.10568
- [13]J. W. T. Peters and M. Welling. Probabilistic Binary Neural Networks, 2018.arxiv.org/abs/1809.03368
- [14]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI technical report, 2019.
- [15]V. Sanh, L. Debut, J. Chaumond, and T. Wolf. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter, 2019.arxiv.org/abs/1910.01108
- [16]J. Shaw-Taylor and J. Zhao. Generalisation of a Class of Continuous Neural Networks. In Advances in Neural Information Processing Systems, volume 8, 1995.
- [17]G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 38087–38099, 2023.arxiv.org/abs/2211.10438

