WorksComputer Science
BeFit: Shifting Complexity from Weights to Bernstein Activations for Resource-Constrained Sensing
Abstract
Time-series sensing increasingly relies on neural networks, yet deployment on resource-constrained microcontrollers remains challenging. Aggressive model compression can reduce accuracy, while restricting weights to multiplication-free representations introduces further losses. We present BeFit, an end-to-end framework that shifts model capacity from weights to learnable Bernstein polynomial activations. BeFit (i) finds compact Bernstein-activated convolutional neural networks (BeCNNs) through hardware-aware neural architecture search and trains them via knowledge distillation; (ii) projects their weights to multiplication-free codes, freezes the codes, and recovers the resulting accuracy loss through the learnable activations; and (iii) generates model-specific kernels that exploit the projected weights for multiplication-free convolutions. On human activity recognition, BeCNNs are 2.75 percentage points (pp) more accurate on average than ReLU CNNs of identical architecture. On a TI CC3220SF (Cortex-M4), weight projection speeds BeCNNs up by 2.2x on average while losing 0.8 pp, compared with 2.4 pp for ReLU CNNs. Against the MLPerf Tiny DS-CNN reference model running on TensorFlow Lite Micro with CMSIS-NN, the unprojected BeCNN achieves 1.0 pp higher accuracy, 2.2x lower latency, and 6.4x lower flash, while the projected model remains within 1.0 pp of the reference accuracy and runs 5.9x faster. In emulation on a RISC-V core without a hardware multiplier, the instruction-count reduction relative to the DS-CNN increases from 5.2x on the Cortex-M4 to 147x. BeFit also extends to transformers and MLPs, and across all tasks the final models stay within 1.56 pp of their float32 teachers on average.
Introduction
Sensing applications such as human activity recognition (HAR) on wearables and keyword spotting (KWS) run neural networks continuously and close to the sensor, often on low-power microcontrollers (MCUs) with a few hundred kilobytes of memory and limited compute. Fitting these budgets requires shrinking the model and reducing costly multiplications, both of which lower accuracy. Recovering that accuracy within MCU budgets remains a challenge.
Existing work addresses parts of this problem. Runtimes and kernel libraries [11, 25], hardware-aware architecture searches [6, 29] and memory co-design [55] fit models to the MCU, but keep fixed activations and weights that must be multiplied. Ternary and power-of-two quantization [27, 56] removes the multiplications but keeps fixed activations, so the restricted weights must recover the lost accuracy on their own, which is hard in compact models. Existing MCU work on learnable activations, such as TinyKAN-HAR [26], keeps weights that still multiply. None of these approaches combines frozen multiplication-free weights with learnable polynomial activations to recover the resulting accuracy loss.
We present BeFit, an end-to-end framework that turns a large pretrained network into a compact MCU model with multiplication-free convolutions. BeFit replaces a CNN’s conventional activations with learnable Bernstein polynomials [9, 21], forming a BeCNN. Next, it finds a compact BeCNN with a hardware-aware architecture search and trains it via knowledge distillation (KD). It then projects the convolutional weights onto a multiplication-free set, freezes them and recovers the lost accuracy through the activations, and finally generates kernels that exploit the projected weights for deployment.
To our knowledge, BeFit is the first to use learnable Bernstein activations for sensing on MCUs, and the first to use them to recover the accuracy lost when weights are projected onto a multiplication-free set.
The contributions of this work are as follows:
An end-to-end framework for MCU deployment. BeFit, built around learnable Bernstein activations, turns a large pretrained network into an integer MCU image through hardware-aware architecture search, knowledge distillation, weight projection with recovery, and kernel generation with lookup-table activations.
BeCNNs: Bernstein activations for sensing on MCUs. We bring learnable Bernstein activations to compact CNNs for sensing on MCUs, replacing their conventional activations to form Bernstein-activated CNNs (BeCNNs). They restore the expressiveness compact models lose under strong compression, at almost no additional inference cost.
Weight projection with activation recovery. We project the convolutional weights to multiplication-free codes and freeze them, then exploit the learnable Bernstein activations to recover the model’s lost accuracy.
Projection-aware kernel generation. BeFit generates architecture-specific kernels that exploit the projected weights for multiplication-free convolutions, and compiles each activation into a lookup table for efficient inference.
Evaluation on six tasks and a real MCU. We show that BeCNNs are more accurate than ReLU CNNs under strong compression and lose little accuracy after projection. We evaluate BeFit end to end on the TI CC3220SF [45] against TensorFlow Lite Micro [11] with CMSIS-NN [25] and TinyEngine [29]. We also test it in emulation on RISC-V with no hardware multiplier, and extend it to transformers and multilayer perceptrons (MLPs).
Background and Related Work
MCU frameworks. TensorFlow Lite Micro (TFLM) [11] is an open-source MCU runtime used by MLPerf Tiny’s reference implementations [5]. Its interpreter executes a quantized model graph and calls a kernel per operator, taken from CMSIS-NN [25] on Arm Cortex-M. While TFLM runs a given model, hardware-aware architecture searches design models that fit the MCU: MCUNet [29] co-designs the search with its own inference engine, and MicroNAS [23] targets time-series classifiers under memory and latency limits. However, these works keep standard layers with fixed activations and multiplied weights. BeFit instead changes the building blocks themselves, with learnable Bernstein activations and multiplication-free weights, and builds the search, training and kernels around them.
Learnable activations. Padé Activation Units [33] learn rational activations end to end, over an unbounded input range. Kolmogorov–Arnold Networks (KANs) [31] replace each weight of a multilayer perceptron with a learnable one-dimensional function on its edge, and these functions can be compiled into lookup tables on MCUs [24]. TinyKAN-HAR [26] applies one learnable spline per neuron after each linear layer of a fully connected network, stored as a lookup table, and keeps int8 weights; it targets zero-shot recognition and explainability. Faggi et al. [14] pair a CNN backbone with a KAN classifier head, but the convolutions keep fixed activations. BeFit instead places one learnable Bernstein curve per filter or layer inside a standard CNN, compiles each into a lookup table, and uses these curves to recover the accuracy lost to compression and multiplication-free weights.
Bernstein polynomial networks. Khedr and Shoukry [21] introduce DeepBern-Nets, feed-forward networks in which standard activations are replaced by learnable Bernstein polynomials, and use their coefficient bounds to certify image classifiers. Albool et al. [2] prove that their approximation error decays exponentially in the network depth, against a polynomial rate for ReLU networks. Bern2Edge [12] later distills feed-forward networks into Bernstein networks for FPGA deployment, either as lookup tables or as symbolic rules. BeFit instead brings Bernstein-activated CNNs to time-series sensing on resource-constrained MCUs, and exploits them to recover the accuracy lost to multiplication-free convolutions.
Multiplication-free weights. Ternary () and power-of-two () weights need no multiplier. Ternary Weight Networks (TWN) [27] approximate each filter by a scale times ternary codes and train latent full-precision weights through the quantizer, and INQ [56] quantizes weights to powers of two or zero in steps, retraining the remaining weights after each step. DeepShift [13] trains power-of-two weights so that each multiply becomes a shift and a sign flip, and ShiftCNN [18] converts pretrained CNNs to power-of-two weights for custom accelerators. All keep fixed activations and recover accuracy, when needed, through the weights. ReActNet [30] binarizes weights and activations and adds learnable shifts and slopes to the sign and PReLU functions, but still trains the weights. BeFit instead freezes the projected codes and recovers accuracy through the learnable Bernstein curves. To our knowledge, it is the first to do so with learnable polynomial activations.
Sensing systems have also cut multiplications by leaving standard networks behind: Bakar et al. [4] run Tsetlin machines on batteryless MCUs, whose inference evaluates Boolean clauses and sums their votes without any multiplication, and Fast-Inf [10] reduces the multiply count within binary tree-based networks whose cost grows logarithmically with network size. Both, however, give up the CNN. BeFit instead keeps the convolutional backbone with multiplication-free weights.
Kernel generation. Generated kernels are usually specialized to the model’s structure. TinyEngine in MCUNet [29] generates kernels and memory schedules for each layer’s shape, and Lupe [53] converts network descriptions into code for ultra-low-power MCUs, but both still load and multiply the weights. Other work specializes the code to the weight values. On FPGAs, unrolled ternary networks [48] store the weights in the routing of pruned adder trees. On MCUs, ATAMAN [3] writes int8 weights into generated Cortex-M7 kernels, but still multiplies, and Neuro-C [40] fixes ternary connections on a Cortex-M0, but has no convolutions. In contrast, BeFit generates multiplication-free convolution kernels for MCUs directly from its projected weights (Section 3.5).
BeFit Framework
BeFit turns a large pretrained teacher into a compact integer model for MCUs. Its central idea is to shift model capacity from the weights to the activations: the weights are made fewer and then multiplication-free, while learnable Bernstein activation curves carry the expressiveness the weights give up. This trade suits MCUs because the activations are cheap where the weights are not. Each activation curve compiles into a fixed-size lookup table, so its inference cost is small and independent of its degree. The building block is the BeCNN (Section 3.1), a standard CNN whose fixed activations are replaced by learnable Bernstein curves (Fig. 2). Around it, BeFit runs four modules (Fig. 1).
(1) A hardware-aware NAS (Section 3.2) finds a compact BeCNN under the MCU’s memory limits, trading accuracy against instructions measured on the target’s own build. Because the polynomial degree does not change these costs, the search treats it as a free dimension and can raise the curve’s expressiveness at no additional deployed inference cost, although training cost grows (Section 4.1). (2) Knowledge distillation (Section 3.3) then trains the selected BeCNN in full from the teacher. (3) Projection and recovery (Section 3.4) remove the convolutions’ multiplications: the convolutional weights are projected onto multiplication-free codes and frozen, and the accuracy lost to projection is recovered by retraining the Bernstein curves rather than the weights. (4) Projection-aware kernel generation (Section 3.5) specializes the inference kernels to the multiplication-free codes. Each Bernstein activation becomes a single table load, and requantization a shift. The result is an integer model whose convolutions need no multiplier. Section 3.6 extends the same pipeline to transformers and MLPs.
BeCNNs: Bernstein Convolutional Networks
Bernstein polynomials. A Bernstein polynomial of degree , defined on the interval , is written as [9]:
where is the input to the polynomial function, is the normalized input, and are the polynomial coefficients. The Bernstein basis functions are defined as:
They are nonnegative and sum to one on . Hence, the polynomial is a convex combination of its coefficients over a bounded domain, with the coefficients directly controlling the activation shape.
BeCNNs. BeFit replaces a CNN’s fixed activations with learnable Bernstein polynomials, forming a BeCNN (Figure 2). A BeCNN keeps the convolutions, pooling and classifier of a standard CNN and learns one curve per filter, shared across all of the filter’s positions.
Figure 2. A BeCNN: a CNN whose activations are learnable Bernstein polynomials (Eq. (1)). Each activation normalizes its input to and outputs ; only the coefficients are learned, and the curve stays within the convex hull of its control points. Layer count is schematic.
Per-filter Activations. At each position, filter computes its pre-activation from its weights , bias and input , as in a standard CNN. The activation then normalizes to and evaluates the filter’s curve (Eq. (1)), with the filter’s own bounds and learned coefficients .
Why Bernstein activations. Two properties suit Bernstein activations to resource-constrained MCUs. (1) Parameter efficiency: for Lipschitz targets, the approximation error of a Bernstein network of degree and depth decays as , exponentially in the depth, against a polynomial rate for ReLU networks, and most tested feed-forward architectures need far fewer parameters than their ReLU equivalents [2]. (2) Deployability: each activation’s normalized input lies in the fixed range , so its curve can be precomputed offline into a lookup table whose size does not depend on the degree [12].
Hardware-Aware Neural Architecture Search (NAS)
The first step of BeFit (Figure 1) is a hardware-aware search for the right BeCNN. In a conventional CNN with fixed activations, added capacity costs instructions. In contrast, a BeCNN can increase its representational capacity by raising the Bernstein polynomial degree without increasing deployed inference cost. The search therefore chooses the architecture and the polynomial degree together, trading accuracy against computation under the target MCU’s memory limits.
Figure 1. BeFit pipeline. A hardware-aware NAS finds a compact BeCNN, and a high-accuracy teacher helps train it via KD. The BeCNN’s convolutional weights are projected to multiplication-free codes, the learnable activations recover the lost accuracy, and a generated kernel exploits these codes for MCU deployment. Weights and curves are illustrative.
Search objective. For an architecture with Bernstein degree and training hyperparameters , let be the validation accuracy of the trained int8 model and its instructions per inference, before projection. The search approximates the Pareto set of
where is flash (model data plus inference code), is RAM (static data plus peak stack), and and are the MCU’s limits. , and come from a cost table measured before the search: every architecture is compiled with the target’s build flags and run in the QEMU simulator with random weights, which is valid because the instruction count of the unprojected int8 model does not depend on its weight values.
Polynomial degree. The degree enters the accuracy objective of Eq. (3) but none of its costs, since every curve compiles into the same lookup table whatever its degree (§ 3.5.2). The search can therefore raise a curve’s expressiveness at no additional deployed inference cost (only training cost grows), and it samples jointly with the architecture, so each shape gets the degree that suits it.
Cheap training. To estimate how an architecture performs, each trial trains from scratch on labels only, without distillation, with one seed and for a short, fixed number of epochs, and is scored on its int8 model with every curve compiled to its deployed lookup table, so the score includes the loss from tabulating the curves.
Optimizer. We use the multi-objective tree-structured Parzen estimator [8, 37] in Optuna [1]. Flash and RAM are trial constraints: an architecture that violates them is not trained, and its recorded violation steers the sampler away. Multi-objective asynchronous successive halving [28, 41] prunes weak trials early. It ranks trials on both accuracy and instructions, so that cheap models stay on the Pareto front instead of being pruned for lower accuracy.
Selecting students. Because each trial uses one seed, the candidates of non-dominated rank (the Pareto front and the next front behind it) are retrained with two more seeds, and the front is taken on the three-seed validation mean. Under an instruction budget , we keep the candidate with the best mean among those with . This BeCNN becomes the student trained in BeFit’s next step (§ 3.3). § 4.3 gives the search spaces, the setup, the ReLU baseline and the results.
BeCNN Training via Knowledge Distillation
The second step of BeFit (Figure 1) trains the student BeCNN found by the NAS. Since the search only trains each candidate briefly and on labels alone, knowledge distillation (KD) from the pretrained teacher now trains the student in full, before any weight projection.
Knowledge distillation loss. The student objective follows the standard distillation loss [19]:
where is the cross-entropy loss between student predictions and ground-truth labels, is the softmax, is the Kullback–Leibler divergence between the softened teacher and student distributions, and and are the student and teacher logits. is the temperature controlling how soft the outputs are, and balances supervision between hard labels and teacher guidance. The teacher stays frozen, sees the same batches as the student, and is discarded at inference.
Bernstein bounds. A Bernstein polynomial is evaluated on , while a filter’s pre-activations can take any value, so each activation needs bounds that map its pre-activation onto this interval (Eq. (1)). BeCNN training starts with a short warm-up using fixed bounds. Each filter’s bounds are then calibrated from the 0.1st and 99.9th percentiles of its pre-activations on the training data and stay fixed for the rest of training [12]. Fixing the bounds stabilizes training and gives each lookup table a known input range (§ 3.5.2). Models with BatchNorm before the activation skip the calibration and keep the warm-up bounds.
Out-of-bounds penalty. Pre-activations outside the bounds are clamped, so the BeCNN adds an out-of-bounds penalty to its loss to discourage them:
where is the normalized pre-activation of Eq. (1) before clamping. The result is a compact, high-accuracy BeCNN, which we refer to as BeFit. These dense models are the starting point of weight projection (§ 3.4). § 4.4 compares them against equivalent ReLU networks under the same hardware budgets.
Projection and Recovery via Learnable Activations
The third step of BeFit (Figure 1) projects the trained BeCNN’s dense convolutional weights to multiplication-free codes and recovers the lost accuracy through the learnable Bernstein activations (Figure 3).
Figure 3. Projection, recovery and multiplication-free convolution. Convolutional weights are approximated as with a per-filter scale and codes . Recovery freezes and trains the Bernstein coefficients, biases, scales and classifier. At inference, nonzero codes are implemented using additions and shifts, so the convolution needs no multiplier. Matrices and curves are illustrative.
Projection to multiplication-free codes
Projection restricts each filter’s weights to a small set of values that need no multiplication. We first write the weights of filter as a vector and replace it with : one positive scale for the whole filter times a vector of integer codes from a code set . We choose between two code sets, ternary () and powers of two (). The scale adds no run-time multiply: requantization uses only a right shift, and the remaining scale is folded into the activation’s lookup table (§ 3.5.2). With the scale folded away, the convolution itself needs no multiplications: a nonzero code is one addition or subtraction of a shifted input (), and a zero costs nothing.
The share of zeroed weights is chosen to be either fixed, at 70, 50 or 30%, or free, in which case each weight simply takes the code with the lowest error, which may be zero. § 4.5 compares these sets.
Choosing the codes. Naively rounding each weight to its nearest code minimizes the weight error , treating all weights alike. However, the filter’s output is what actually matters: because its inputs are correlated, errors on different weights can cancel or add up. Data-aware projection therefore minimizes the output error directly, as in post-training quantization [15, 16, 34]. Let be the input samples under the filter’s kernel (its taps), collected from training windows, and let be their mean. A weight error changes the filter’s output by , and the variance of that change is , where is the covariance of the taps:
To minimize this output error, we solve for the scale and the codes in turn, since each depends on the other, repeating two steps until they stop changing. (1) With the scale fixed, assign each weight in turn the code in that minimizes . (2) With the codes fixed, set the scale to its least-squares solution . We run this from multiple initial scales and retain the solution with the lowest projection error.
With a fixed share of zeroed weights, we search greedily: the largest weights start as their signs () and the rest as zero. Each round applies the single change (moving a nonzero to a zero position or flipping a sign) that most lowers , then refits , until no change lowers the error. This gives the final ternary codes; for powers of two, steps (1) and (2) then choose the values at the nonzero positions.
Eq. (6) controls only the variance of the output change, but projection also shifts the filter’s mean output by , which moves relative to the activation’s fixed bounds (Eq. (1)) and can push it out of range. The bias is corrected to cancel this [35]:
Recovery through learnable Bernstein activations.
Projection lowers the model’s accuracy, and conventional methods recover it by retraining the weights [27, 56]. BeFit instead freezes the projected codes and trains the Bernstein coefficients, along with the convolutional biases, per-filter scales and classifier, in one training run. Because the coefficients directly control each activation’s shape (Eq. (1)), training them reshapes each filter’s response without changing its frozen weights, which recovers the accuracy lost to projection. § 4.5 shows how much of the recovery the polynomial coefficients carry alone.
Recovery trains on the deployed int8 model through fake-quantized forward passes with straight-through gradients (§ 3.5.2). It uses self-distillation with the loss of Eq. (4), with the unprojected BeCNN as the teacher, and keeps the epoch with the best validation accuracy.
The result is a network with multiplication-free convolutions: only the classifier stays int8 and multiplies. § 3.5 generates a kernel that exploits these weights for MCU deployment, and § 4.5 gives the ReLU baseline, the ablations and the results.
Projection-Aware Kernel Generation and Inference
The last step of BeFit (Figure 1) turns the trained BeCNN into integer code for the MCU. The projection-aware generated kernel (the generated kernel for short) runs projected models (BeFit), the dense kernel runs unprojected ones (BeFit), and every Bernstein activation becomes a lookup table.
Kernel generation.
Projection alone does not make inference cheaper: a generic kernel still loads and multiplies every weight, zeros included. The savings come from how the kernel stores and applies the fixed codes. BeFit supports three options: weights as code for speed, weights as data for flash savings, and a mix of the two (Figure 4).
Figure 4. A schematic four-filter chunk of projected weights, stored two ways with identical outputs. As code (panel 3a), each nonzero weight becomes an addition or subtraction: faster, more flash. As data (panel 3b), index lists drive one shared loop: slower, less flash. Panel 4 shows the Cortex-M4 optimizations.
Weights as code. The codes are known at compile time, so a generator writes them directly into the program (Figure 4, panel 3a). Each weight multiplies one tap, so the generator applies one rule per code in (panel 2): a zero emits no instruction, adds or subtracts the input, and adds or subtracts it shifted by . Each filter becomes straight-line code with no weight loads and no loop over weights. This is the fastest option, at the cost of more flash, since every nonzero code becomes instructions.
Weights as data. To save flash, the nonzero codes can instead be stored as index lists read by one shared loop (Figure 4, panel 3b). Each power of two has an add list and a subtract list: for every nonzero code, the loop loads its index, loads that input and adds or subtracts it, doubling the sum between powers. Zeros take no space and all filters share one loop, so this option needs the least flash, at the cost of higher latency from one extra index load per code. It suits large models where flash is the limit.
Mix. The mix sits between these two ends of the trade-off: weights are stored as code for the parts of the model that save the most time per byte, and the rest are stored as data. Section 4.6 compares the three options.
Architecture-aware. The generated kernel also exploits the Cortex-M4’s architecture in three ways. (1) One register, two positions. An int16 input buffer places the inputs of two consecutive output positions in the two halves of one 32-bit register, so each SADD16 or SSUB16 updates both. (2) Reuse across filters. With weights as code, one input load serves a chunk of up to four filters, instead of a separate load for each. (3) Power groups. SADD16 has no built-in shift, so shifting every input would cost extra instructions. Instead, each filter adds its inputs into one accumulator per power of two, then combines them once per output by doubling and adding (Figure 4, panel 4). For example, weights on inputs give , and , and output .
Lookup-table activations
The kernel’s sum then passes through the filter’s Bernstein activation. A polynomial is too costly to evaluate at run time on an MCU, but a Bernstein curve’s input lies in the fixed range (Eq. (1)), so it can be precomputed into a lookup table. Bern2Edge [12] introduced such tables for FPGAs, with linear interpolation that costs one multiply per activation. BeFit removes this multiply: each input maps directly to one table entry, so an activation is a single table load.
Table indexing. Each filter’s curve is stored in a table of int8 entries, and finding an entry at run time takes only an addition, a shift and a clamp. The normalization of Eq. (1) is computed ahead of time: its division and fold into the filter’s scale and integer bias , so , where includes for a projected model. A right shift by , chosen per filter at compile time as the smallest shift for which the entries cover all of , brings this sum down to a table index:
where sat clamps to , keeping in as in Eq. (1).
Scale-aware tables. A shift divides only by powers of two, so it cannot match the filter’s scale exactly. Correcting this at run time would take a multiply, so each table instead stores the curve at the inputs its entries actually receive, adapting scale-aware activation tables and shift-based scaling [43, 49]. The remaining constants also need no run-time work: the table’s output scale and offset fold into the classifier, and BatchNorm and the pooling average fold into adjacent layers.
Same cost at any degree. A curve of any degree compiles to the same -entry table, so the degree changes only the stored values, not the flash ( bytes per curve), RAM or instructions (one load per output). High-degree polynomials therefore add expressiveness under compression and capacity for recovery at no additional deployed inference cost.
The resulting convolution needs no multiplications: weights become additions, requantization a shift and the activation a table read. Only the int8 classifier still multiplies.
Dense kernel
The dense kernel runs the unprojected BeCNNs. It stores weights as data and multiplies each one, with three optimizations. (1) SIMD multiply-accumulates. SMLAD performs two per instruction, as in CMSIS-NN [25], and one pass computes two time positions of four filters in eight register accumulators. (2) Direct inputs. The input is widened once into a zero-padded int16 buffer, so filters read it in place instead of copying it per position. (3) Fusion. Each sum goes straight through the shift, the activation table of Eq. (8) and the pooling sum, without being stored. Section 4.6 compares this kernel with TFLM+CMSIS-NN and with the generated kernel.
| Task | Input | Classes | Train | Val | Test |
| UCI-HAR | 6 × 128 | 6 | 5,867 | 1,485 | 2,947 |
| HHAR | 6 × 128 | 6 | 30,850 | 5,374 | 10,755 |
| PAMAP2 | 18 × 128 | 12 | 10,982 | 2,107 | 1,934 |
| Speech Cmds v2 | 49 × 10 MFCC | 12 | 85,511 | 10,102 | 4,890 |
| Speech Cmds v2 (KWT) | 98 × 40 MFCC | 35 | 84,843 | 9,981 | 11,005 |
| UJIIndoorLoc | 520 RSSI | 3+5 | 13,955 | 5,982 | 1,111 |
Table 1. Tasks and splits.
Extensions to Other Architectures
BeFit is not specific to convolutions: any linear layer followed by a scalar activation can use the same pipeline. In a transformer, per-unit Bernstein curves replace the feed-forward network’s activation (GELU or Swish), and projection makes the linear layers multiplication-free; only the attention products, which multiply two activations, keep their multiplications. In an MLP, each hidden neuron gets one curve, and the hidden rows are projected to multiplication-free codes. Section 4.7 evaluates both.
Results
Experimental Setup
Terminology. Throughout, BeCNN denotes the model architecture, a CNN with learnable Bernstein activations, and BeFit the complete optimization and deployment framework. We call the trained BeCNN before weight projection BeFit, and the recovered model with frozen multiplication-free weight codes BeFit.
Datasets. Table 1 lists the tasks: three HAR datasets [38, 39, 42], split by person; Speech Commands v2 [51], with the MLPerf Tiny front end [5] for the depthwise-separable CNNs (DS-CNN) and all 35 words for the keyword transformers; and UJIIndoorLoc [46] for the MLPs. Appendix A gives the windowing and preprocessing.
Teacher models. Each task has one teacher, shared by its Bernstein and ReLU students. ResNet [50] (UCI-HAR), DeepSense [54] (HHAR) and DeepConvLSTM [36] (PAMAP2) are published architectures trained with their published optimizers and settings. HAR students distill from the logit average of three seeds, while reported teacher accuracies are per seed. The DS-CNN teacher is the released MLPerf Tiny reference model [5, 32]. The keyword transformers are distilled from AST [17] and the UJIIndoorLoc MLPs from the MLP of Kim et al. [22] (§ 4.7). Appendix A lists all teachers with their accuracies and training recipes.
Training hardware. All models are trained in PyTorch on NVIDIA A30, V100 and A100 GPUs. Against a ReLU student of the same shape, a Bernstein student’s curve coefficients add 20–178% to the convolution’s parameters, and an epoch takes 1.32× as long. Inference does not pay this cost, because each curve compiles into a fixed lookup table (§ 3.5.2).
Board and QEMU. The target is the TI CC3220SF LaunchPad: an 80 MHz Cortex-M4 with 256 KiB RAM, 1 MiB flash, no floating-point unit and no flash cache [44, 45]; code runs from flash. Instruction counts come from QEMU’s Cortex-M4 machine with the same build flags (Appendix § 8): the inference machine code is identical to the board’s, and a single-step count of one inference on the board equals QEMU’s. Every image is checked bit for bit against the integer reference.
Metrics. Accuracy is the test accuracy of the int8 network, whose outputs equal the MCU’s, as mean ± population SD over three seeds; accuracy differences are in percentage points (pp). Instructions are QEMU instructions for one inference. Latency is measured on the CC3220SF board for the seed-0 model: Cortex-M4 cycle-counter cycles around one inference, median over at least 100 test inputs, at 80 MHz. Flash counts the model and runtime without the test harness; RAM counts static data and peak stack.
DS-CNN activations. A DS-CNN has an activation after every layer, so one curve per filter would cost far more flash than the HAR BeCNNs. Its BeCNN students therefore share one Bernstein curve per layer, and BatchNorm gives each filter its own input scale and shift on that curve.
BeFit End-to-End Results
Table 2 presents BeFit’s end-to-end results, before (BeFit) and after (BeFit) projection, against the teacher baseline across all tasks. The teachers run on TensorFlow Lite Micro (TFLM) [11] with Arm’s CMSIS-NN kernels [25]. TFLM is also the runtime of MLPerf Tiny’s reference implementations [5], making this a strong, widely reproduced baseline. Appendix B compares TFLM’s plain C reference kernels, which run 2.5 to 6.1× slower.
Accuracy. Across the six tasks, the projected students lose 0.7 pp on average against their int8 teachers and 1.56 pp against their float32 teachers. They beat the int8 teachers on PAMAP2 and UJI-IndoorLoc.
Hardware. The HAR teachers are not built for microcontrollers, so the large reductions come mostly from compression. On UCI-HAR and HHAR, the projected students run 834 to 9,580× faster than the teachers, with 36 to 71× less flash and about 20 to 31× less RAM. DeepConvLSTM does not fit the board at all (a 290 KiB tensor arena against 256 KiB of RAM). AST does not fit either. These teacher-to-student ratios therefore measure the whole framework, dominated by compression, rather than the kernels alone. The same-architecture and dense-versus-projected comparisons of Sections 4.4–4.6 isolate the contributions of the Bernstein activations, projection and generated kernel.
The MLPerf Tiny DS-CNN, in contrast, is a strong model built for microcontrollers, and BeFit still beats it. BeFit is 1.0 pp more accurate and 2.2× faster, with 6.4× less flash and 2.0× less RAM. BeFit gives up 1.0 pp for 5.9× lower latency, 2.3× less flash and 1.5× less RAM. Its generated kernel trades flash for speed, turning the weights into instructions (§ 3.5).
Against the same model on TinyEngine [29], a code generator with no interpreter, BeFit wins on accuracy, latency, flash and RAM, while BeFit is 6.6× faster but needs 1.5× more flash and loses 1.0 pp of accuracy. TinyEngine’s outputs match TFLM’s, and it uses 3.5× less flash and 1.4× less RAM, but it is 13% slower. TinyEngine has no kernel for the model’s 10×14 first convolution and no int8 softmax, so these two layers use the CMSIS-NN kernels bundled with it, included in its latency and memory. TinyEngine also cannot run the HAR teachers: it has no kernels for their 1×k convolutions and no recurrent layers.
Without a hardware multiplier. Table 3 repeats the comparison on RV32EC [52], a RISC-V instruction set with no multiply instruction. The teachers therefore spend 83 to 93% of their instructions in a software routine, against 13 to 29% for BeFit. On the MLPerf Tiny reference, the reduction in instructions grows from 5.2× on the Cortex-M4 to 147×. As a control, the same RISC-V core with the multiply extension (RV32EMC) reaches only 7.9× (Appendix § 10), so the larger reduction comes from the missing multiplier, not from RISC-V.
Hardware-Aware Neural Architecture Search
Setup. The search of § 3.2 runs on two model families. HAR CNNs use the search space of Table 4. DS-CNNs for keyword spotting have at most five convolution layers, and the search covers the width (16–128), the first convolution’s kernel and stride, the curves per channel at the last activation, the learning rate and, for Bernstein, the degree (15, 21 or 31). Trials train for 60 epochs on labels only, are pruned at epochs 7 and 20, and are selected on validation accuracy. Bernstein studies receive more trials than ReLU studies (500 vs. 300 for HAR, 200 vs. 150 for DS-CNNs): their extra dimensions (degree and the coefficient learning-rate multiplier) change
| Dimension | Values |
| Dilation groups | 1, 2, 3 (dilations ) |
| Filters per group | 8, 12, 16, 24, 32, 48, 64, 96, 128 |
| Kernel size | 3, 5, 7, 9 |
| Stride (all groups) | 1, 2, 4, 8, 16, 32 |
| Normalization | none, BatchNorm |
| Learning rate | , , |
| Bernstein degree | 3, 5, 7, 11, 15, 23, 31 |
| Coefficient lr multiplier | 1, 3, 10 |
Table 4. HAR search space. The ReLU control omits the last two rows.
neither the compiled kernel nor its instructions, flash or RAM, so at equal trial counts Bernstein would cover fewer distinct architectures. Every architecture uses at most 16% of the CC3200SF’s flash and 13% of its RAM, so the memory constraints of Eq. (3) never bind and instructions alone shape the fronts (Appendix § 11). Instructions rank models as board cycles do (Spearman 0.9964 over 180 board images).
Results. Figure 5 applies the budget rule of § 3.2, which selects on the three-seed validation mean, at every instruction budget and plots the test-accuracy gap between the two picks. On HAR, the BeCNN is almost always ahead, over 94 to 99% of the log-instruction range. On Speech Commands, the BeCNN leads over 69% of the range, mostly below 1M, and in the intermediate budgets the two trade the lead within approximately 1 pp.
Figure 5. Int8 test accuracy of the BeCNN minus the ReLU model, in pp, for the models selected on validation accuracy at every instruction budget. Dashed lines: the budgets carried forward.
These gaps come from cheap training, so they rank architectures rather than give final accuracy. We carry three budgets per HAR dataset and four for Speech Commands through the rest of the BeFit pipeline (Table 5 and Table 6). On HAR, the BeCNN leads at all budgets by 2.31 to 5.94 pp, with its largest lead at each dataset’s smallest budget. On Speech Commands it leads at 0.5M and 2M,
| Task | Budget | Model | Shape | Instr. | Test (%) | ΔAcc. |
| UCI-HAR | 6k | BeCNN | 1×8, k7, s32, N5 | 5,530 | 83.33 ± 0.63 | +3.82 |
| ReLU | 1×8, k7, s32 | 5,497 | 79.50 ± 0.07 | |||
| 15k | BeCNN | 1×8, k9, s8, N31 | 12,334 | 91.81 ± 1.01 | +2.41 | |
| ReLU | 3×8, k3, s16 | 10,727 | 89.40 ± 0.65 | |||
| 25k | BeCNN | 3×12, k3, s8, BN, N31 | 22,841 | 94.23 ± 0.29 | +2.31 | |
| ReLU | 3×16, k5, s16 | 23,189 | 91.92 ± 0.15 | |||
| HHAR | 7k | BeCNN | 1×8, k5, s16, BN, N23 | 6,438 | 81.36 ± 1.12 | +5.94 |
| ReLU | 1×8, k5, s16, BN | 6,405 | 75.42 ± 2.26 | |||
| 15k | BeCNN | 2×8, k7, s16, BN, N23 | 11,536 | 86.71 ± 1.23 | +3.44 | |
| ReLU | 3×8, k7, s32, BN | 10,547 | 83.27 ± 0.90 | |||
| 60k | BeCNN | 3×16, k9, s8, N11 | 58,688 | 91.75 ± 0.68 | +3.05 | |
| ReLU | 3×24, k9, s16, BN | 48,371 | 88.69 ± 0.25 | |||
| PAMAP2 | 12k | BeCNN | 1×8, k3, s32, BN, N11 | 11,424 | 79.15 ± 1.22 | +4.58 |
| ReLU | 1×8, k3, s32 | 11,321 | 74.56 ± 1.83 | |||
| 20k | BeCNN | 1×8, k3, s8, BN, N11 | 17,352 | 85.09 ± 1.49 | +2.76 | |
| ReLU | 1×16, k3, s16 | 18,035 | 82.33 ± 0.90 | |||
| 40k | BeCNN | 1×16, k5, s8, BN, N31 | 35,112 | 87.18 ± 1.88 | +2.45 | |
| ReLU | 1×8, k9, s8, BN | 30,449 | 84.73 ± 0.51 |
Table 5. Best model per instruction budget (HAR). Shape: dilation groups×filters per group, kernel size (k), stride (s), BatchNorm (BN), Bernstein degree (N). ΔAcc.: BeCNN minus ReLU, in pp. Models are selected on validation accuracy; Test: int8 test accuracy, three seeds.
| Budget | Model | Shape | Instr. | Test (%) | ΔAcc. |
| 0.5M | BeCNN | w16, k6, s4, h4, N15 | 423,591 | 79.14 ± 1.46 | +1.58 |
| ReLU | w16, k6, s4, h4 | 438,713 | 77.56 ± 1.03 | ||
| 1M | BeCNN | w32, k6, s4, h1, N15 | 872,678 | 86.00 ± 0.11 | −1.69 |
| ReLU | w32, k6, s4, h4 | 953,880 | 87.69 ± 0.52 | ||
| 2M | BeCNN | w48, k6, s3, h1, N15 | 1,934,222 | 89.86 ± 0.58 | +0.68 |
| ReLU | w48, k10, s4, h1 | 1,595,564 | 89.18 ± 0.21 | ||
| 4M | BeCNN | w64, k6, s3, h4, N21 | 3,023,118 | 91.64 ± 0.23 | +0.04 |
| ReLU | w48, k10, s2, h1 | 3,077,451 | 91.60 ± 0.57 |
Table 6. Best DS-CNN per instruction budget. Shape: width (w), first-convolution kernel (k×4) and stride (s×2), curves per channel at the last activation (h), Bernstein degree (N). Models are selected on validation accuracy; Test: int8 test accuracy, three seeds.
ReLU leads at 1M, and the two tie at 4M. Full training and projection change these gaps (§ 4.4 and § 4.5).
BeCNN Training via Knowledge Distillation and Dense Deployment
Setup. The budget models of Table 5 and Table 6 are distilled from their task’s teacher, with the same recipe for both activations. We use , (: labels only), AdamW with a cosine learning-rate schedule, and early stopping on validation accuracy. For each model we report the with the best three-seed accuracy mean.
Results. At the largest budgets, the distilled BeCNNs come within 1.3 and 1.8 pp, respectively, of the int8 teachers on UCI-HAR and HHAR, and beat them on PAMAP2 and Speech Commands
(Table 7 and Table 8). The BeCNN also beats its ReLU counterpart at all nine HAR budgets, by 0.2 to 4.5 pp, and at three of four Speech Commands budgets. Its lead is largest at the smallest budget on most tasks, showing that the learned activations help most when a network has little capacity left. Every student fits the CC3200SF, with at most 20.9 KiB of flash and 12.3 KiB of RAM.
| Task | Budget | Model | Test (%) | ΔAcc. | Flash | RAM | Lat. |
| UCI-HAR | 6k | BeCNN | 9.1 | 2.4 | 0.221 | ||
| UCI-HAR | 6k | ReLU | 8.2 | 2.4 | 0.215 | ||
| UCI-HAR | 15k | BeCNN | 9.3 | 2.5 | 0.511 | ||
| UCI-HAR | 15k | ReLU | 8.7 | 2.4 | 0.418 | ||
| UCI-HAR | 25k | BeCNN | 11.9 | 2.5 | 0.919 | ||
| UCI-HAR | 25k | ReLU | 11.0 | 2.5 | 0.926 | ||
| HHAR | 7k | BeCNN | 8.9 | 2.4 | 0.257 | ||
| HHAR | 7k | ReLU | 8.0 | 2.4 | 0.249 | ||
| HHAR | 15k | BeCNN | 10.4 | 2.5 | 0.477 | ||
| HHAR | 15k | ReLU | 9.9 | 2.6 | 0.426 | ||
| HHAR | 60k | BeCNN | 16.6 | 2.7 | 2.491 | ||
| HHAR | 60k | ReLU | 16.0 | 2.7 | 1.983 | ||
| PAMAP2 | 12k | BeCNN | 10.4 | 5.4 | 0.448 | ||
| PAMAP2 | 12k | ReLU | 9.5 | 5.4 | 0.446 | ||
| PAMAP2 | 20k | BeCNN | 10.4 | 5.4 | 0.715 | ||
| PAMAP2 | 20k | ReLU | 10.5 | 5.4 | 0.754 | ||
| PAMAP2 | 40k | BeCNN | 13.0 | 5.5 | 1.541 | ||
| PAMAP2 | 40k | ReLU | 11.2 | 5.6 | 1.360 |
Table 7. HAR NAS models (Table 5) after KD. Test: int8, three seeds. Flash and RAM (KiB) and latency (ms) on the CC3220SF. ΔAcc.: BeCNN minus ReLU (pp).
| Budget | Model | Test (%) | ΔAcc. | Flash | RAM | Lat. |
| 0.5M | BeCNN | 8.0 | 2.7 | 17.4 | ||
| 0.5M | ReLU | 7.8 | 2.7 | 16.9 | ||
| 1M | BeCNN | 10.0 | 4.6 | 35.0 | ||
| 1M | ReLU | 11.3 | 5.0 | 36.5 | ||
| 2M | BeCNN | 13.8 | 8.6 | 76.8 | ||
| 2M | ReLU | 14.2 | 6.7 | 60.8 | ||
| 4M | BeCNN | 20.9 | 12.0 | 120.2 | ||
| 4M | ReLU | 14.2 | 12.3 | 117.1 |
Table 8. DS-CNN NAS models (Table 6) after distillation; Test: int8, three seeds. Flash and RAM (KiB) and latency (ms) on the CC3220SF. ΔAcc.: BeCNN minus ReLU (pp).
Same-architecture control. To isolate the effect of the NAS and of KD from that of the Bernstein activation, we also train both activations on the same nine architectures, without KD (Table 9). The BeCNN leads on eight pairs and ties on the ninth, by 2.75 pp on average, showing that the advantage comes mainly from the activation itself. Its hardware overhead is small: 0.5 to 4.8% more latency and at most 3.4 KiB more flash (0.3% of the board’s 1 MiB).
| Dataset | Shape | Flash (KiB) BeCNN / ReLU | Flash (KiB) Δ | RAM (KiB) BeCNN / ReLU | RAM (KiB) Δ | Latency (ms) BeCNN / ReLU | Latency (ms) Δ | Test (%) BeCNN | Test (%) ReLU | Test (%) Δ |
| UCI-HAR | 1×8, k3, s32 | 8.7 / 7.8 | +11.5% | 2.4 / 2.4 | +1.2% | 0.185 / 0.178 | +3.6% | 80.00 ± 0.97 | 77.15 ± 0.51 | +2.85 |
| UCI-HAR | 1×8, k7, s16 | 9.1 / 8.2 | +11.0% | 2.4 / 2.4 | +1.1% | 0.294 / 0.286 | +2.8% | 88.34 ± 0.58 | 87.12 ± 1.25 | +1.22 |
| UCI-HAR | 1×8, k9, s8 | 9.3 / 8.4 | +10.8% | 2.5 / 2.4 | +1.1% | 0.511 / 0.500 | +2.3% | 89.91 ± 1.15 | 88.36 ± 0.86 | +1.55 |
| HHAR | 1×8, k3, s32, BN | 8.7 / 7.8 | +11.5% | 2.4 / 2.4 | +1.2% | 0.185 / 0.178 | +3.6% | 73.71 ± 1.27 | 70.00 ± 0.14 | +3.70 |
| HHAR | 3×8, k5, s32, BN | 11.2 / 9.3 | +20.4% | 2.5 / 2.5 | +1.1% | 0.389 / 0.372 | +4.8% | 85.19 ± 1.90 | 82.56 ± 0.79 | +2.63 |
| HHAR | 3×16, k7, s16, BN | 15.5 / 12.1 | +28.1% | 2.6 / 2.6 | +1.0% | 1.186 / 1.146 | +3.5% | 91.63 ± 1.27 | 86.30 ± 0.94 | +5.32 |
| PAMAP2 | 1×8, k3, s32 | 10.4 / 9.5 | +9.3% | 5.4 / 5.4 | +0.6% | 0.448 / 0.446 | +0.5% | 79.59 ± 0.28 | 74.75 ± 0.77 | +4.84 |
| PAMAP2 | 1×8, k3, s16, BN | 10.4 / 9.5 | +9.3% | 5.4 / 5.4 | +0.6% | 0.537 / 0.533 | +0.7% | 82.42 ± 1.49 | 79.78 ± 1.91 | +2.64 |
| PAMAP2 | 3×8, k3, s8, BN | 13.5 / 11.6 | +16.2% | 5.7 / 5.6 | +0.6% | 1.532 / 1.511 | +1.4% | 85.35 ± 1.15 | 85.38 ± 0.83 | −0.03 |
Table 9. BeCNN and ReLU CNN with identical architectures, trained without distillation. Hardware on the CC3220SF. Δ: BeCNN minus ReLU, relative to ReLU (%) for hardware and in pp for test acc.
Projection and Recovery
Projecting the distilled students. Table 10 projects each distilled student to powers of two, recovers it (§ 3.4; recipe in Appendix § 8) and runs it on the generated kernel (§ 3.5). ReLU uses conventional training in the style of TWN [27], which validation selects over our frozen-code recovery at eight of nine HAR models (Appendix § 12). We therefore keep this stronger baseline for ReLU.
| Dataset | Budget | Teacher Acc. (%) | BeFit (Ours) Acc. (%) Dense | BeFit (Ours) Acc. (%) Proj. | Δ | Δ_teacher | BeFit (Ours) Lat. (ms) Dense | BeFit (Ours) Lat. (ms) Proj. | Spd. | ReLU (TWN [27]) Acc. (%) Dense | ReLU (TWN [27]) Acc. (%) Proj. | Δ | Δ_teacher | ReLU (TWN [27]) Lat. (ms) Dense | ReLU (TWN [27]) Lat. (ms) Proj. | Spd. |
| UCI-HAR | 6k | 96.01 | 83.92 | 83.20 | −0.72 | −12.81 | 0.22 | 0.15 | 1.49× | 81.19 | 79.06 | −2.13 | −16.95 | 0.22 | 0.14 | 1.49× |
| UCI-HAR | 15k | 92.29 | 92.10 | −0.19 | −3.91 | 0.51 | 0.25 | 2.08× | 89.97 | 88.61 | −1.36 | −7.40 | 0.42 | 0.24 | 1.73× | |
| UCI-HAR | 25k | 94.70 | 93.96 | −0.74 | −2.05 | 0.92 | 0.48 | 1.93× | 93.51 | 92.22 | −1.29 | −3.79 | 0.93 | 0.46 | 2.02× | |
| HHAR | 7k | 94.72 | 81.55 | 81.15 | −0.40 | −13.57 | 0.26 | 0.15 | 1.66× | 77.02 | 70.08 | −6.94 | −24.64 | 0.25 | 0.15 | 1.66× |
| HHAR | 15k | 88.68 | 87.88 | −0.80 | −6.84 | 0.48 | 0.24 | 1.96× | 84.28 | 79.69 | −4.59 | −15.03 | 0.43 | 0.23 | 1.82× | |
| HHAR | 60k | 92.91 | 92.16 | −0.75 | −2.56 | 2.49 | 1.12 | 2.22× | 92.70 | 88.43 | −4.27 | −6.29 | 1.98 | 0.87 | 2.27× | |
| PAMAP2 | 12k | 84.13 | 79.99 | 79.82 | −0.17 | −4.31 | 0.45 | 0.34 | 1.32× | 77.42 | 77.84 | +0.42 | −6.29 | 0.45 | 0.34 | 1.32× |
| PAMAP2 | 20k | 85.92 | 85.35 | −0.57 | +1.22 | 0.71 | 0.44 | 1.62× | 85.18 | 82.70 | −2.48 | −1.43 | 0.75 | 0.47 | 1.61× | |
| PAMAP2 | 40k | 86.92 | 86.73 | −0.19 | +2.60 | 1.54 | 0.79 | 1.96× | 84.13 | 83.51 | −0.62 | −0.62 | 1.36 | 0.59 | 2.30× | |
| KWS | 0.5M | 91.62 | 81.07 | 79.59 | −1.48 | −12.03 | 17.4 | 4.9 | 3.53× | 78.98 | 76.37 | −2.61 | −15.25 | 16.9 | 4.7 | 3.63× |
| KWS | 1M | 86.41 | 84.51 | −1.90 | −7.11 | 35.0 | 11.1 | 3.16× | 87.23 | 84.66 | −2.57 | −6.96 | 36.5 | 11.9 | 3.08× | |
| KWS | 2M | 90.74 | 89.95 | −0.79 | −1.67 | 76.8 | 25.2 | 3.05× | 90.20 | 88.36 | −1.84 | −3.26 | 60.8 | 18.5 | 3.30× | |
| KWS | 4M | 92.60 | 90.62 | −1.98 | −1.00 | 120.2 | 44.3 | 2.71× | 91.66 | 90.15 | −1.51 | −1.47 | 117.1 | 35.9 | 3.26× | |
| Mean | −0.82 | −4.93 | 2.21× | −2.44 | −8.41 | 2.27× |
Table 10. Distilled students after powers-of-two projection, on the generated kernel, against each task’s int8 teacher. Δ: Proj. − Dense; Δteacher: Proj. − Teacher (pp). Spd.: Dense/Proj. Bold: smaller drop.
Accuracy. After projection, the BeCNN leads ReLU at 12 of 13 budgets, by up to 11.1 pp. It loses 0.8 pp of its dense accuracy on average, against 2.4 pp for ReLU.
Speed. Projection makes both BeCNN and ReLU models about 2.2× faster on the board, and up to 3.6× on the DS-CNNs, so the BeCNN keeps its accuracy lead without losing the speedup.
Choice of weight set. Table 11 compares the weight sets on one student. Ternary at 70% zeros is the fastest (2.61×) but loses 2.6 pp. Powers of two with a free share lose 0.7 pp and still run 1.93× faster.
| Weights | Test (%) | Instr. | Flash | RAM | Lat. | Speedup |
| Dense int8 | 94.70 | 22,819 | 11.9 | 2.5 | 0.919 | 1.00 |
| Ternary | ||||||
| 70% zeros | 92.06 | 9,121 | 6.9 | 2.4 | 0.352 | 2.61 |
| 50% zeros | 91.87 | 10,043 | 7.3 | 2.4 | 0.385 | 2.39 |
| 30% zeros | 90.74 | 10,989 | 7.7 | 2.4 | 0.418 | 2.20 |
| Free | 93.20 | 9,710 | 7.2 | 2.4 | 0.378 | 2.43 |
| Powers of two | ||||||
| 70% zeros | 92.74 | 10,281 | 7.3 | 2.4 | 0.379 | 2.42 |
| 50% zeros | 93.88 | 11,990 | 8.0 | 2.5 | 0.432 | 2.13 |
| 30% zeros | 93.42 | 13,538 | 8.8 | 2.5 | 0.484 | 1.90 |
| Free | 93.96 | 13,433 | 8.6 | 2.5 | 0.477 | 1.93 |
Table 11. UCI-HAR 25k BeCNN under each weight set. Flash and RAM in KiB, latency in ms. Speedup: against dense model.
Which parameters recover the projection loss. Table 12 isolates the Bernstein coefficients. At ternary codes, where projection loses the most, training only the coefficients gives 75 to 96% of the full recovery in seven of eight cases, and up to 17.8 pp at 85% zeros. The activations therefore carry the recovery; biases, scales and the head add the rest.
| Ternary codes | UCI-HAR | HHAR | ||||
| None | Coef. | Full | None | Coef. | Full | |
| 30% zeros | 87.69 | 88.99 | 89.31 | 76.84 | 78.20 | 80.79 |
| 50% zeros | 87.34 | 90.45 | 90.78 | 74.04 | 78.25 | 79.57 |
| 70% zeros | 85.21 | 90.07 | 90.76 | 79.82 | 82.83 | 82.96 |
| 85% zeros | 68.57 | 86.40 | 88.51 | 68.37 | 78.62 | 81.95 |
Table 12. Recovery ablation on 16-filter BeCNNs (dense: 92.00% on UCI-HAR, 89.53% on HHAR). None: no recovery; Coef.: Bernstein coefficients only; Full: coefficients, biases, scales and head. Test (%).
Kernel Evaluation
TFLM+CMSIS-NN vs. BeFit’s kernel. Table 13 checks that our ReLU and GELU baselines are strong by running the same trained
| Network | Model | Kernel | Test (%) | Instr. | Lat. (ms) | Flash (KiB) | RAM (KiB) | Speedup |
| HHAR CNN, 3×16, k7, s16 | ReLU | TFLM+CMSIS-NN | 84.50 ± 0.51 | 79,406 | 3.05 | 70.9 | 6.8 | – |
| ReLU | Ours | 86.30 ± 0.94 | 28,250 | 1.15 | 12.1 | 2.6 | 2.66× | |
| BeFit | Ours | 91.63 ± 1.27 | 28,184 | 1.19 | 15.5 | 2.6 | 2.57× | |
| DS-CNN, 0.5M | ReLU | TFLM+CMSIS-NN | 78.36 ± 0.77 | 674,407 | 21.33 | 84.0 | 9.1 | – |
| ReLU | Ours | 78.98 ± 0.54 | 438,713 | 16.90 | 7.8 | 2.7 | 1.26× | |
| BeFit | Ours | 81.07 ± 0.35 | 423,591 | 17.42 | 8.0 | 2.7 | 1.22× | |
| KWT, 3×256 | GELU | TFLM+CMSIS-NN | 91.82 ± 0.25 | 79.1 M | 2,698.1 | 282.0 | 89.4 | – |
| GELU | Ours | 92.43 ± 0.37 | 40.5 M | 1,464.5 | 213.7 | 60.5 | 1.84× | |
| BeFit | Ours | 94.83 ± 0.41 | 38.6 M | 1,414.7 | 257.1 | 60.5 | 1.91× |
Table 13. The same network on TFLM+CMSIS-NN [11, 25] and on our dense kernel (Ours), with BeFit of the same shape on ours. TFLM’s KWT uses one custom GELU operator. Speedup: against TFLM+CMSIS-NN.
network on TFLM with CMSIS-NN and on our dense kernel. TFLM’s int8 models come from the TFLite Converter, as for the teachers, so they share our float32 weights but not our int8 bits. Our kernel beats TFLM+CMSIS-NN on every metric: 0.6 to 1.8 pp more accurate, 1.26–2.66× faster, with up to 10.8× less flash and 3.4× less RAM. Our ReLU and GELU baselines are therefore stronger than the standard deployment, and BeFit’s lead over them does not come from a weak kernel. The Bernstein network of the same shape keeps that speedup (1.22–2.57×) and is up to 5.3 pp more accurate than the ReLU network on our kernel.
Projected codes: dense vs. generated kernel. On the dense kernel, projected codes take exactly the unprojected model’s cycles on all 13 BeCNNs, because it still multiplies every weight. Only the generated kernel turns them into speed, 1.3–3.5× (Figure 6), so projection and the generated kernel pay off only together.
Figure 6. Board speedup of the projected BeCNNs over their unprojected models. Projected weights only pay off on the generated kernel: on the dense kernel, the latency stays the same (1×).
Weights as code vs. as data. Table 14 stores the same codes three ways (Section 3.5.1). On the transformer, code is 1.79× faster than dense but needs 1.6× its flash, while data cuts the flash by a third and is barely faster (1.03×). The mix is 1.22× faster within the unprojected model’s flash. On the CNN, only code pays off (2.22× for 9% more flash). Data runs 2.5× slower: each nonzero code costs three instructions (an index load, an input load and an add), while one multiply-accumulate instruction of the dense kernel handles two weights.
| KWT | HHAR | ||||||
| Weights | Lat. (ms) | Speedup | Flash (KiB) | Lat. (ms) | Speedup | Flash (KiB) | |
| Dense | 1,859 | 1.00× | 249.5 | 2.49 | 1.00× | 16.6 | |
| Code | 1,036 | 1.79× | 391.3 | 1.12 | 2.22× | 18.1 | |
| Data | 1,810 | 1.03× | 167.9 | 6.14 | 0.41× | 10.2 | |
| Mix | 1,529 | 1.22× | 246.3 | 2.79 | 0.89× | 15.9 |
Table 14. A projected KWT (6×64) and BeCNN (HHAR 60k) with their codes stored three ways: Code, Data and Mix. Dense: the unprojected model on the dense kernel. Speedup: over Dense.
Beyond CNNs: Transformers and MLPs
The same pipeline carries over to two other model families: keyword-spotting transformers and Wi-Fi localization MLPs.
Keyword transformers
We replace the GELU feed-forward (FFN) activation of a KWT-1-style transformer [7] with one degree-15 Bernstein curve per hidden unit. Students have embedding width 64, one attention head, six or three blocks and FFN width , and are distilled () from the AST teacher (Section 4.1) through a separate distillation token and head, as in DeiT [47], with KWT’s training recipe. GELU becomes one 256-entry table shared by all units of a layer, and each Bernstein unit its own 64-entry table (Section 3.5.2); both run on BeFit’s kernels.
Results. Table 15 compares GELU with Bernstein curves, before and after projection to powers of two with 70% zeros. BeFit beats GELU at every shape, and the gain grows with compression, up to 2.40 pp at three blocks. It needs 1.9–4.8% fewer instructions and runs 1.2–3.5% faster at the same RAM, for 9–22% more flash. Bernstein activations also let the FFN halve its width: BeFit at beats GELU at with 25% lower latency and 16% less flash.
| Shape | Model | Test (%) | Instr. (M) | Lat. (ms) | Flash (KiB) | RAM (KiB) |
| 12×3072 | AST* | 97.10 | 95,738 | — | 83,797.0 | 2,353.9 |
| 6×256 | GELU | 96.06 ± 0.13 | 80.4 | 2,908 | 386.0 | 60.5 |
| BeFit | 96.48 ± 0.31 | 76.5 | 2,807 | 471.2 | 60.5 | |
| BeFit | 95.90 ± 0.21 | 51.3 | 1,995 | 468.0 | 62.5 | |
| 6×128 | GELU | 95.33 ± 0.18 | 62.1 | 2,224 | 281.0 | 60.5 |
| BeFit | 96.21 ± 0.13 | 60.2 | 2,176 | 323.8 | 60.5 | |
| BeFit | 95.34 ± 0.23 | 43.6 | 1,655 | 320.0 | 62.5 | |
| 6×64 | GELU | 94.77 ± 0.13 | 52.9 | 1,882 | 228.7 | 60.5 |
| BeFit | 95.68 ± 0.11 | 51.9 | 1,859 | 249.5 | 60.5 | |
| BeFit | 95.44 ± 0.21 | 40.6 | 1,529 | 246.3 | 62.5 | |
| 3×256 | GELU | 92.43 ± 0.37 | 40.5 | 1,464 | 213.7 | 60.5 |
| BeFit | 94.83 ± 0.41 | 38.6 | 1,415 | 257.1 | 60.5 | |
| BeFit | 94.51 ± 0.08 | 26.5 | 1,053 | 242.2 | 62.5 |
Table 15. KWTs on Speech Commands (35 words). Shape: blocks × FFN width ℎ. *Does not fit the board: measured in QEMU on TFLM’s reference kernels, so it has no latency.
Projection costs BeFit at most 0.9 pp and speeds it up 1.2–1.4×. The attention products limit the speedup: they multiply two activations, have no weights to project, and make up 20–34% of the unprojected model’s multiply-accumulates.
MLPs for Wi-Fi localization
UJIIndoorLoc [46] maps Wi-Fi RSSI values to a building and a floor; we report joint building-and-floor accuracy, the share of samples whose building and floor are both correct. The teacher is Kim et al.’s SAE-pretrained ReLU MLP [22], 520–256–128–64–128, with two softmax heads instead of their multi-label output; it reaches 92.02 ± 0.57% float32 joint accuracy, against the published 91.18%. The students have a single Bernstein hidden layer of 128, 64 or 32 units, are distilled and then projected to ternary weights with 70% zeros.
Results. One Bernstein hidden layer replaces the teacher’s four (Table 16): the 64-unit student reaches 92.59% joint accuracy against 92.02%. Ternary projection with 70% zeros costs at most 0.09 pp and makes the students 12 to 48× faster than the teacher on TFLM, with up to 4.9× less flash and 2.7× less RAM.
Limitations and Future Work
Remaining multiplications. The int8 classifier still multiplies. In the keyword transformers, the attention products and multiply two activations, so there are no weights to project. These products are 20–34% of the unprojected model’s multiply-accumulates, which limits the speedup. Multiplication-free or lookup-based attention is a natural next step.
On-device adaptation. The projected codes are frozen, and training only the Bernstein coefficients recovers most of the projection loss (§ 4.5). This suggests updating only the curves on the device, for example to adapt to a new user or to sensor drift, while the weight code stays fixed in flash.
Energy and hardware coverage. We report latency, flash and RAM on a single Cortex-M4 board. The RISC-V results without a hardware multiplier come from emulation. Future work will measure energy per inference on the board. It will also run BeFit on multiplier-less RISC-V silicon and on Cortex-M cores with vector extensions, where the trade-off between projected and dense weights may change.
Conclusion
We presented BeFit, an end-to-end framework that shifts model capacity from the weights to learnable Bernstein activations, so that compact MCU models keep their accuracy while their convolutions become multiplication-free. Across six sensing tasks, BeCNNs outperform ReLU CNNs of identical architecture, lose little accuracy after weight projection, and run faster than strong MCU baselines, with gains that grow on cores without a hardware multiplier. These results suggest that learnable activations are an effective place to put model capacity when weights must be small and cheap.
Acknowledgments
This material is based upon work supported by the U.S. National Science Foundation (NSF) under Grant No. 2504809, and by the NSF Graduate Research Fellowship Program under Grant No. DGE-2235784. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.
| Shape | Model | Test (%) | Lat. (ms) | Flash (KiB) | RAM (KiB) |
| 520–256–128–64–128–8 | Teacher | 92.02 ± 0.64 | 14.68 | 211.4 | 2.9 |
| 520–128–8 | BeFit | 92.41 ± 0.40 | 6.66 (↓2.21×) | 162.2 (↓1.30×) | 2.4 (↓1.19×) |
| BeFit | 92.41 ± 0.52 | 1.18 (↓12.5×) | 108.8 (↓1.94×) | 1.5 (↓1.98×) | |
| 520–64–8 | BeFit | 92.59 ± 0.36 | 3.34 (↓4.40×) | 91.9 (↓2.30×) | 2.4 (↓1.19×) |
| BeFit | 92.56 ± 0.04 | 0.60 (↓24.4×) | 65.4 (↓3.23×) | 1.2 (↓2.41×) | |
| 520–32–8 | BeFit | 91.39 ± 0.40 | 1.68 (↓8.74×) | 56.7 (↓3.73×) | 2.4 (↓1.19×) |
| BeFit | 91.30 ± 1.42 | 0.31 (↓47.6×) | 43.4 (↓4.87×) | 1.1 (↓2.74×) |
Table 16. MLPs on UJIIndoorLoc: the teacher on TFLM with CMSIS-NN against BeFit students; (↓, ×): teacher / student. Test: joint building-and-floor accuracy.
Dataset and Build Details
Dataset preprocessing. HAR windows are 128 samples at 50 Hz (2.56 s, 50% overlap), lie within single-label runs, and are standardized per channel on training statistics. UCI-HAR keeps total acceleration and body gyroscope. HHAR uses the accelerometer and gyroscope of the six phones that record both, resampled to 50 Hz, without the null label. PAMAP2 uses the g accelerometer and gyroscope of its three IMUs, with missing samples interpolated, resampled from 100 Hz, without the transient label 0. The DS-CNN’s 12 Speech Commands classes are ten words, silence and unknown, with 30 ms frames, 20 ms hop, 40 mel bins and 10 coefficients [5]. UJIndoorLoc splits its official training file 70:30 [22] and tests on its official validation file.
Teacher models. Table 17 lists every teacher. ResNet, DeepSense and DeepConvLSTM are trained on our splits with their published recipes: Adam at , halved on plateau, batch 64 (ResNet [20, 50]); Adam at , , L2 , batch 64 (DeepSense [54]); RMSProp at , dropout 0.5, batch 100 (DeepConvLSTM [36]). On top of these, they use the students’ augmentation and train for at most 500 epochs, with early stopping on validation accuracy (patience 100). DeepSense splits each 128-sample window into 8 intervals of 16 samples, because our windows are shorter than the published 5 s. The DS-CNN is the released MLPerf Tiny model [5, 32], and its int8 network is the released TFLite file. AST is the released checkpoint MIT/ast-finetuned-speech-commands-v2 [17], which we convert to int8 with the TFLite Converter on 500 training clips. The UJIIndoorLoc teacher follows Kim et al. [22]: a stacked autoencoder (256–128–256) whose encoder feeds classifier layers 64–128, trained with Adam, dropout 0.2, batch 10 for 20 epochs.
Table 17: Teachers. Float32 and int8: test accuracy (%); UJIIndoorLoc: joint building-and-floor accuracy. ±: population SD over three seeds; no ±: one released model.
Table 17.
Build and measurement. Images are compiled with GNU Arm GCC 15.2.1 (-O2, -mcpu=cortex-m4, soft-float ABI); TFLite Micro with CMSIS-NN [11, 25] uses its release build. Images are loaded over JTAG, and test inputs are streamed over UART.
Post-projection recovery training. Recovery after projection uses Adam with a cosine schedule, batch size 256, and , in Eq. (4), for 50 epochs on HAR and 100 on the DS-CNNs. BeCNNs with BatchNorm keep it live during recovery, add the out-of-bounds penalty of Eq. (5), and fold it into the codes’ scales and biases at the end.
TFLM Reference Kernels
Table 18 repeats the int8 teachers on TFLM’s default reference kernels, in plain C, with no optimized kernels. They run 2.5 to 6.1× slower than with CMSIS-NN, with 5 to 31% less flash and about the same RAM. AST has no CMSIS-NN build and does not fit the board, so Table 2 reports it on these reference kernels in QEMU, and it is not listed here.
| Task | Model | Acc. (%) | Lat. (ms) | Flash (KiB) | RAM (KiB) |
| UCI-HAR | ResNet | 96.01 ± 0.32 (96.52) | 4,569 | 610.9 | 77.6 |
| BeFit | 94.70 ± 0.09 | 0.92 (↓4,971×) | 11.9 (↓51.2×) | 2.5 (↓31.6×) | |
| BeFit | 93.96 ± 0.79 | 0.48 (↓9,580×) | 8.6 (↓70.7×) | 2.5 (↓31.4×) | |
| HHAR | DeepSense | 94.72 ± 0.57 (95.08) | 934.4 | 660.1 | 52.9 |
| BeFit | 92.91 ± 0.25 | 2.49 (↓375×) | 16.6 (↓39.8×) | 2.7 (↓19.3×) | |
| BeFit | 92.16 ± 0.53 | 1.12 (↓834×) | 18.1 (↓36.4×) | 2.7 (↓19.7×) | |
| PAMAP2 | DeepConvLSTM* | 84.13 ± 1.86 (86.99) | — | ≥849.5 | >256.0 |
| BeFit | 86.92 ± 0.97 | 1.54 (↓≥65.5×) | 13.0 (↓>46.4×) | 5.5 | |
| BeFit | 86.73 ± 0.86 | 0.79 (↓≥88.4×) | 9.6 (↓>43.5×) | 5.9 | |
| KWS | MLPerf Tiny | 91.62 (92.17) | 259.6 | 134.8 | 24.1 |
| On TinyEngine | 91.62 | 294.1 | 38.3 | 16.8 | |
| BeFit | 92.60 ± 0.86 | 120.2 (↓2.16×) | 20.9 (↓6.44×) | 12.0 (↓2.00×) | |
| BeFit | 90.62 ± 1.01 | 44.3 (↓5.86×) | 58.3 (↓2.31×) | 15.9 (↓1.51×) | |
| KWT | AST* | 97.10 (98.03) | — | 83,797.0 | 2,353.9 |
| BeFit | 95.68 ± 0.11 | 1,859 | 249.5 (↓336×) | 60.5 (↓38.9×) | |
| BeFit | 95.44 ± 0.21 | 1,529 | 246.3 (↓340×) | 62.5 (↓37.7×) | |
| UJIIndoorLoc | MLP | 92.02 ± 0.64 (92.02) | 14.7 | 211.4 | 2.88 |
| BeFit | 92.59 ± 0.36 | 3.34 (↓4.40×) | 91.9 (↓2.30×) | 2.41 (↓1.19×) | |
| BeFit | 92.56 ± 0.04 | 0.60 (↓24.4×) | 65.4 (↓3.23×) | 1.20 (↓2.41×) |
Table 2. Each task’s int8 teacher on TFLM with CMSIS-NN against BeFit. Parentheses: teacher’s float32 accuracy; (↓ ×): TFLM teacher / BeFit. UJIIndoorLoc: joint building-and-floor accuracy. *Does not fit the board: AST measured in QEMU on TFLM’s reference kernels, so it has no latency; DeepConvLSTM has no Cortex-M build, so its values are lower bounds.
| Teacher | Lat. (ms) | Slower | Flash (KiB) | RAM (KiB) |
| ResNet | 22,205 | 4.9× | 576.1 | 77.1 |
| DeepSense | 2,373 | 2.5× | 625.3 | 51.0 |
| MLPerf Tiny DS-CNN | 1,591 | 6.1× | 92.9 | 23.5 |
| UJIIndoorLoc MLP | 60.8 | 4.1× | 200.3 | 2.38 |
Table 18.
Table 18: Int8 teachers on TFLM’s reference kernels against TFLM with CMSIS-NN, seed 0.
Multiply-Extension Control
Table 19 repeats the instruction counts of Table 3 on the same RISC-V core with the multiply extension (RV32EMC). With a hardware multiplier, the reductions return to the same order of magnitude as on the Cortex-M4, so the larger reductions on RV32EC come from the missing multiplier, not from RISC-V.
| Task | Model | Acc. (%) | Instr. (M) M4 | Instr. (M) RV32EC | Flash (KiB) | RAM (KiB) |
| UCI-HAR | ResNet | 96.01 ± 0.32 | 120.4 | 5,103.6 | 621.6 | 77.4 |
| BeFit | 93.96 ± 0.79 | 0.013 (↓8,962×) | 0.093 (↓54,798×) | 25.8 (↓24.1×) | 3.2 (↓24.4×) | |
| HHAR | DeepSense | 94.72 ± 0.57 | 24.8 | 790.7 | 682.9 | 53.3 |
| BeFit | 92.16 ± 0.53 | 0.032 (↓767×) | 0.274 (↓2,890×) | 82.0 (↓8.3×) | 3.5 (↓15.4×) | |
| PAMAP2 | DeepConvLSTM | 84.13 ± 1.86 | — | 25,770.9 | 941.5 | 291.8 |
| BeFit | 86.73 ± 0.86 | 0.022 (↓154,680×) | 0.167 (↓19.3×) | 48.7 (↓19.3×) | 8.2 (↓35.4×) | |
| KWS | MLPerf Tiny | 91.62 | 7.6 | 439.3 | 148.3 | 23.9 |
| On TinyEngine | 91.62 | 8.6 | 251.5 | 42.8 | 16.5 | |
| BeFit | 90.62 ± 1.01 | 1.460 (↓5.2×) | 2.996 (↓147×) | 88.5 (↓1.7×) | 18.5 (↓1.3×) |
Table 3. Teachers against BeFit: QEMU instructions per inference on the Cortex-M4 and RV32EC, with RV32EC flash and RAM; (↓ ×): TFLM teacher / student. — = no Cortex-M build. On RV32EC, teachers use CMSIS-NN’s portable C code, with TinyEngine’s Arm DSP instructions rewritten in plain C.
Table 19: Teacher and projected student on RV32EMC: QEMU instructions per inference, seed 0. Reduction: teacher / student, with the Cortex-M4 and RV32EC reductions of Table 3 for comparison.
| Task | Teacher | Student | M4 | RV32EMC | RV32EC |
| UCI-HAR | 374.5 | 0.065 | 8,962 | 5,760 | 54,798 |
| HHAR | 68.9 | 0.237 | 767 | 291 | 2,890 |
| PAMAP2 | 1,225.1 | 0.138 | — | 8,879 | 154,680 |
| KWS | 18.9 | 2.398 | 5.2 | 7.9 | 147 |
Table 19.
Neural Architecture Search Details
The HAR and Speech Commands searches took 11.1 and 33.0 GPU-hours, including pruned trials. Candidates of non-dominated rank ≤ 2 are evaluated over three seeds: 163 BeCNN and 166 ReLU models.
Figure 7 shows the resulting fronts. On all three HAR datasets, the BeCNN envelope lies above the ReLU envelope at most instruction counts, with the largest gaps at the smallest models. It peaks at 94.93% against 93.29% on UCI-HAR, 93.22% against 91.44% on HHAR and 87.18% against 85.85% on PAMAP2. On HHAR and PAMAP2 the BeCNN peak needs fewer instructions. On Speech Commands, the BeCNN leads almost everywhere below 0.9M instructions, by up to 6.13 pp. Between 0.9M and 3M the two alternate, and above 3M the BeCNN leads by at most 1.09 pp.

Figure 7. Pareto fronts: candidates are selected on three-seed validation accuracy and plotted at their int8 test accuracy. Markers: front models; step line: best test accuracy up to each instruction count; faint dots: all three-seed candidates.
ReLU Projection Baseline
Table 20 compares two ways to recover the ReLU students after projection: conventional TWN-style training, which retrains the weights, and our frozen-code pass, which trains only the scales, biases and head. We choose the method on validation data. The table reports int8 validation accuracy, and its numbers differ from the test accuracies of Table 10. Conventional training wins at eight of nine budgets, so the ReLU baseline uses it.
Table 20: ReLU students of Table 10 at powers of two {0, ±1, ±2, ±4} with a free share of zeros: TWN-style training [27] (Conventional) against our frozen-code single pass (BeFit). int8 validation accuracy (%), three-seed mean. Bold: better of the pair.
| Dataset | Budget | TWN [27] | BeFit |
| UCI-HAR | 6k | 87.23 | 87.14 |
| 15k | 93.45 | 93.54 | |
| 25k | 95.13 | 93.78 | |
| HHAR | 7k | 80.15 | 78.02 |
| 15k | 85.27 | 84.86 | |
| 60k | 94.21 | 91.59 | |
| PAMAP2 | 12k | 77.90 | 76.62 |
| 20k | 83.86 | 82.61 | |
| 40k | 83.23 | 80.24 |
Table 20.
References
- [1]Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2623–2631. doi:10.1145/3292500.3330701DOI
- [2]Ibrahim Alboul, Malak Gamal El-Din, Salma Elmalaki, and Yasser Shoukry. 2026. Exponential Approximation Rates and Parameter Efficiency of Learnable Bernstein Activations. arXiv preprint arXiv:2602.04264 (2026).
- [3]Giorgos Armeniakos, Georgios Mentzos, and Dimitrios Soudris. 2024. Accelerating TinyML Inference on Microcontrollers Through Approximate Kernels. In IEEE International Conference on Electronics, Circuits and Systems (ICECS). 1–4. doi:10.1109/ICECS61496.2024.10848979DOI
- [4]Abu Bakar, Tousif Rahman, Rishad Shafik, Fahim Kawsar, and Alessandro Montanari. 2022. Adaptive Intelligence for Batteryless Sensors Using Software-Accelerated Tsetlin Machines. In Proc. ACM SenSys. 236–249. doi:10.1145/3560905.3568512DOI
- [5]Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Nat Jeffries, Csaba Kiraly, Jeremy Holleman, Pietro Montino, David Kanter, Pete Warden, Danilo Pau, Urmish Thakker, Antonio Torrini, Jay Cordaro, Giuseppe Di Guglielmo, Javier Duarte, Hanson Tran, Nhan Tran, Niu Wenxu, and Xu Xuesong. 2021. MLPerf Tiny Benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Joaquin Vanschoren and Sai-Kit Yeung (Eds.), Vol. 1. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/da4fb56ce93e74d3df8527599f62642-Abstract-round1.htmlarxiv.org/abs/2106.07597
- [6]Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas Navarro, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul N. Whatmough. 2021. MicroNet: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers. In Proceedings of Machine Learning and Systems, Vol. 3. https://arxiv.org/abs/2010.11267
- [7]Axel Berg, Mark O’Connor, and Miguel Tairum Cruz. 2021. Keyword Transformer: A Self-Attention Model for Keyword Spotting. In Proceedings of Interspeech.DOI
- [8]James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems, Vol. 24. 2546–2554.
- [9]Serge Bernstein. 1912. Démonstration du théorème de Weierstrass fondée sur le calcul des probabilités. Communications de la Société mathématique de Kharkov, Series 2, 13 (1), 1–2. https://www.mathnet.ru/eng/khom107
- [10]Leonardo Lucio Custode, Pietro Farina, Eren Yildiz, Renan Beran Kilic, Kasim Sinan Yildirim, and Giovanni Iacca. 2024. Fast-Inf: Ultra-Fast Embedded Intelligence on the Batteryless Edge. In Proc. ACM SenSys. 239–252. doi:10.1145/3666025.3699335DOI
- [11]Robert David, Jared Dukhan, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Napier, Meghna Natraj, Shlomi Regev, Rocky Rhodes, Tiezhen Wang, and Pete Warden. 2021. TensorFlow Lite Micro: Embedded Machine Learning for TinyML Systems. In Proceedings of Machine Learning and Systems (MLSys), Vol. 3. 800–811. https://proceedings.mlsys.org/paper_files/paper/2021/hash/6c44dc73014d66ba49b28d483a8f8b0d-Abstract.htmlarxiv.org/abs/2010.08678
- [12]Malak Gamal El-Din, Yifan Zhang, Yasser Shoukry, Sitao Huang, and Salma Elmalaki. 2026. Bern2Edge: A Neurosymbolic Compiler for Edge Deployment via Bernstein Polynomial Networks. arXiv:2608.20497 doi:10.48550/arXiv.2608.20497
- [13]Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Henry Tian, and Joey Yiwei Li. 2021. Deep Shift: Towards Multiplication-Less Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2359–2368. doi:10.1109/CVPRW53098.2021.00268
- [14]Daniele Faggi, Oleksandr Kuznetsov, Stefano Mereu, Alessandro Galdelli, Emanuele Frontoni, and Marco Arnesano. 2026. Hybrid CNN-KAN Architectures for Ultra-Compact Person Detection on IoT Devices: A Systematic Evaluation of Quantization Strategies. IEEE Access 14 (2026), 81696–81712. doi:10.1109/ACCESS.2026.3693881DOI
- [15]Elias Frantar and Dan Alistarh. 2022. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35.arxiv.org/abs/2208.11580
- [16]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2210.17323
- [17]Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio Spectrogram Transformer. In Proceedings of Interspeech.
- [18]Denis A. Gudovskiy and Luca Rigazio. 2017. ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks. arXiv:1706.02393arxiv.org/abs/1706.02393
- [19]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. doi:10.48550/arXiv.1503.02531DOI
- [20]Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019. Deep Learning for Time Series Classification: A Review. Data Mining and Knowledge Discovery 33, 4 (2019), 917–963. doi:10.1007/s10618-019-00619-1DOI
- [21]Hisham Khedr and Yasser Shoukry. 2024. DeepBern-Nets: Taming the Complexity of Certifying Neural Networks Using Bernstein Polynomial Activations and Precise Bound Propagation. Proceedings of the AAAI Conference on Artificial Intelligence 38, 19 (2024), 21232–21240. doi:10.1609/aaai.v38i19.30117DOI
- [22]Kyeong Soo Kim, Sanghyuk Lee, and Kaizhu Huang. 2018. A scalable deep neural network architecture for multi-building and multi-floor indoor localization based on Wi-Fi fingerprinting. Big Data Analytics 3, 1, Article 4 (2018). doi:10.1186/s41044-018-0031-2
- [23]Tobias King, Yexu Zhou, Tobias Roddiger, and Michael Beigl. 2025. MicroNAS for Memory and Latency Constrained Hardware Aware Neural Architecture Search in Time Series Classification on Microcontrollers. Scientific Reports 15 (2025), 7575. doi:10.1038/s41598-025-90764-z
- [24]Oleksandr Kuznetsov. 2026. LUT-KAN: Look-up Table Acceleration of Kolmogorov–Arnold Networks on Resource-Constrained Microcontrollers for Embedded Edge Inference. Journal of Systems Architecture 179, Article 103923 (2026). doi:10.1016/j.sysarc.2026.103923DOI
- [25]Liangzhen Lai, Naveen Suda, and Vikas Chandra. 2018. CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs. doi:10.48550/arXiv.1801.06601DOI
- [26]Ismail Lamaakal, Chaymae Yahyati, Yassine Maleh, Khalid El Makkaoui, and Ibrahim Ouahbi. 2026. Explainable Kolmogorov–Arnold Networks for Zero-Shot Human Activity Recognition on TinyML Edge Devices. Machine Learning and Knowledge Extraction 8, 3, Article 55 (2026). doi:10.3390/make8030055DOI
- [27]Fengfu Li, Bin Liu, Xiaoxiang Wang, Bo Zhang, and Junchi Yan. 2016. Ternary weight networks. arXiv preprint arXiv:1605.04711 (2016).arxiv.org/abs/1605.04711
- [28]Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. 2020. A System for Massively Parallel Hyperparameter Tuning. In Proceedings of Machine Learning and Systems (MLSys), Vol. 2. 230–246.
- [29]Ji Lin, Wei-Ming Chen, Xuejin Lin, John Cohn, Chuang Gan, and Song Han. 2020. MCUNet: Tiny Deep Learning on IoT Devices. In Advances in Neural Information Processing Systems, Vol. 33. 11711–11722. https://arxiv.org/abs/2007.10319
- [30]Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwong-Ting Cheng. 2020. ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions. In European Conference on Computer Vision (ECCV) (Lecture Notes in Computer Science, Vol. 12359). 143–159. doi:10.1007/978-3-030-58568-6_9DOI
- [31]Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. 2025. KAN: Kolmogorov–Arnold Networks. In International Conference on Learning Representations (ICLR).
- [32]MLCommons. 2026. MLPerf Tiny: Released Keyword-Spotting Training Reference. https://github.com/mlcommons/tiny/tree/master/benchmark/training/keyword_spotting Source snapshot commit 4adddf used in this study.
- [33]Alejandro Molina, Patrick Schramowski, and Kristian Kersting. 2020. Padé Activation Units: End-to-End Learning of Flexible Activation Functions in Deep Networks. In International Conference on Learning Representations (ICLR).arxiv.org/abs/1907.06732
- [34]Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or Down? Adaptive Rounding for Post-Training Quantization. In Proceedings of the 37th International Conference on Machine Learning (ICML) (PMLR, Vol. 119). 7197–7206.arxiv.org/abs/2004.10568
- [35]Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data-Free Quantization Through Weight Equalization and Bias Correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 1325–1334. doi:10.1109/ICCV.2019.00141DOI
- [36]Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors 16, 1 (2016), 115. doi:10.3390/s16010115DOI
- [37]Yoshihiko Ozaki, Yuki Tanigaki, Shuhei Watanabe, and Masaki Onishi. 2020. Multiobjective Tree-Structure Parzen Estimator for Computationally Expensive Optimization Problems. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO). 533–541. doi:10.1145/3377930.3389817
- [38]Attila Reiss. 2012. PAMP2 Physical Activity Monitorinq. UCI Machine Learning Repository. doi:10.24432/C5NW2H
- [39]Jorge Reyes-Ortiz, Davide Anguita, Alessandro Ghio, Luca Oneto, and Xavier Parra. 2012. Human Activity Recognition Using Smartphones. UCI Machine Learning Repository. doi:10.24432/C54S4KDOI
- [40]Diletta Romano, Luca Mottola, and Thiemo Voigt. 2026. Neuro-C: Neural Inference Shaped by Hardware Limits. In European Conference on Computer Systems (EuroSys). ACM, 2126–2140. doi:10.1145/3767295.3769380DOI
- [41]Robin Schmucker, Michael Dominik, Muhammed Bilal Zafar, David Salinas, and Cédric Archambeau. 2021. Multi-objective Asynchronous Successive Halving. arXiv preprint arXiv:2106.12639 (2021).arxiv.org/abs/2106.12639
- [42]Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Amin Dey, Tobias Sonne, and Mads Møller Jensen. 2015. Heterogeneity Activity Recognition. UCI Machine Learning Repository. https://archive.ics.uci.edu/dataset/344/heterogeneity+activity+recognitionDOI
- [43]TensorFlow Authors. 2024. TensorFlow Lite: Scale-Aware Activation Lookup-Table Construction and Evaluation. Source code, release v2.18.0, tensor-flow/lite/kernels/internal/common. https://github.com/tensorflow/tensorflow/blob/v2.18.0/tensorflow/lite/kernels/internal/common.h Functions LUTPopu-lateInt8, LUTPopulate, and LUTLookup; accessed October 28, 2026.
- [44]Texas Instruments. 2017. CC3220 SimpleLink Wi-Fi and Internet of Things Technical Reference Manual. Texas Instruments. https://www.ti.com/lit/ds/symlink/cc3220s.pdf
- [45]Texas Instruments. 2021. CC3220R, CC3220S, and CC3220SF SimpleLink Wi-Fi Single-Chip Wireless MCU Solutions. Texas Instruments. https://www.ti.com/lit/ds/symlink/cc3220sf.pdf Datasheet, revised May 2021.
- [46]Joaquín Torres-Sospedra, Raúl Montoliu, Adolfo Martínez-Úsó, Joan P. Avariento, Tomás J. Arnau, Mauri Benedito-Bordonau, and Joaquín Huer-ta. 2014. UJIIndoorLoc: A new multi-building and multi-floor database for WLAN fingerprint-based indoor localization problems. In 2014 International Conference on Indoor Position-ing and Indoor Navigation (IPIN). IEEE, 261–270. doi:10.1109/IPIN.2014.7275492
- [47]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training Data-Efficient Image Transform-ers & Distillation through Attention. In Proceedings of the 38th International Conference on Machine Learning (ICML).arxiv.org/abs/2012.12877
- [48]Stephen Trigell, Martin Kumm, Martin Hardieck, David Boland, Duncan Moss, Peter Zipf, and Philip H. W. Leong. 2019. Learning Ternary Neural Networks. ACM Transactions on Reconfigurable Technology and Systems 12, 4, Article 22 (2019), 23 pages. doi:10.1145/3359983
- [49]Maarten Vandersteegen, Kristof Van Beeck, and Toon Goedemé. 2021. Integrally-Connected CNNs with Bit Weights and Bit-Shift Quantization Scales at Full-Precision Accuracy. Electronics 10, 22, Article 2823 (2021). doi:10.3390/electronics10222823
- [50]Zhiguang Wang, Weizhong Yan, and Tim Oates. 2017. Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline. In International Joint Conference on Neural Networks (IJCNN). 1578–1585. doi:10.1109/IJCNN.2017.7966039DOI
- [51]Pete Warden. 2018. Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. doi:10.48550/arXiv.1804.03209
- [52]Andrew Waterman and Krste Asanović (Eds.). 2019. The RISC-V Instruction Set Manual, Volume I: Unprivileged ISA. RISC-V International. Document version 20191213.
- [53]Mingyuan Xiang, Pouya Mahdi Gholami, and Henry Hoffmann. 2025. Lupe: Integrating the Top-down Approach with DNN Execution on Ultra-Low-Power Devices. In Proceedings of the ACM Conference on Embedded Networked Sensor Systems (SenSys). ACM, Irvine, CA, USA, 357–370. doi:10.1145/3715014.3722059DOI
- [54]Shuochao Yao, Shaohan Hu, Jiran Zhao, Aston Zhang, and Tarek Abdelzaher. 2017. DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing. In Proceedings of the 26th International Conference on World Wide Web (WWW). 351–360. doi:10.1145/3038912.3052577DOI
- [55]Size Zheng, Renze Chen, Meng Li, Zihao Ye, Luis Ceze, and Yun Liang. 2024. vMCU: Coordinated Memory Management and Kernel Optimiza-tion for DNN Inference on MCUs. In Proceedings of Machine Learning and Systems, Vol. 6. https://proceedings.mlsys.org/paper_files/paper/2024/hash/d5a655b8b373737b4f2aea8f78e5e754-Abstract-Conference.htmlarxiv.org/abs/2406.06542
- [56]Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. 2017. Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights. In International Conference on Learning Representations. https://arxiv.org/abs/1702.03044





