te-comp: A Complete MLIR Compiler Generating Self-Contained C Inference Engines for MCUs

Introduction

Microcontrollers (MCUs) power billions of embedded devices, from wearable health monitors and keyword-spotting earbuds to industrial sensors and autonomous vehicles. Running deep learning inference directly on these devices eliminates cloud round-trips, reduces latency to microseconds, and preserves user privacy. However, MCUs operate under extreme resource constraints: typical ARM Cortex-M4/M7 targets offer 128–512 KB of SRAM, 512 KB–4 MB of Flash, and clock speeds of 160–400 MHz with no GPU, no virtual memory, and no dynamic allocation. Existing deployment approaches fall into the following categories, each with fundamental limitations.

Interpreter runtimes (TFLite Micro David et al. [5], ExecuTorch Meta AI [13]) load a serialized model and dispatch operations through a generic interpreter loop. This is portable but incurs 20–50 KB of SRAM overhead from dispatch tables, op resolvers, and runtime buffer management. Latency suffers from indirect function calls on every operator. Neither framework supports large language model (LLM) inference on MCUs.

Library-based code generators (MCUNet Lin et al. [12, 11]) use a Python script to generate a model-specific dispatch function that calls into a hand-written C kernel library (33 source files for MCUNet). This achieves near-optimal latency through carefully tuned SMLAD-based kernels, but adding a new operator requires writing new C files, and the generated code depends on the entire kernel library at link time.

General-purpose ML compilers (Apache TVM Chen et al. [4], IREE Google [7], Glow Rotem et al. [1]) use compiler IRs to optimize models, but none produces deployable bare-metal code for Cortex-M microcontrollers. TVM’s Cortex-M support is immature and still requires a runtime. IREE and Glow target GPUs and Cortex-A mobile processors.

None of these approaches produces self-contained, dependency-free C code specialized to a specific input model. We believe that this is the right abstraction for deployment: the output should be plain C code that firmware engineers just drop into any existing project and call with simple APIs without an interpreter, kernel library, or framework dependencies.

We present te-comp, the first complete MLIR-based compiler that achieves this goal. Given a standard model in .tflite for int8 CNNs or .safetensors for transformers, it produces:

• {model}_engine.c/.h: The complete inference engine with all kernels as nested loops, weight arrays in .rodata, and the activation arena in .bss.

• inference_task.c / inference_api.h: Integration layer (FreeRTOS or bare-metal).

• CMakeLists.txt: Drop-in CMake library.

Every loop bound, buffer offset, and arithmetic constant is a compile-time literal specialized to the input model (easy to optimize for machine code). The generated code has zero external function calls, not even memcpy. It achieves this through a multi-level MLIR compilation pipeline. A custom te dialect captures 34 quantized NN operations with all metadata and composable optimization passes progressively lower and optimize the IR before emitting C code. Our contributions are:

1. Self-contained C code. Given any input model, te-comp emits all C artifacts needed for deployment: the inference engine, RTOS glue code, a public API header, and a CMake library target with zero runtime dependencies. No other tool produces this complete output.

2. LLM inference on Cortex-M7. te-comp compiles an 8.3M-parameter GPT-Neo (TinyStories-v2) to a self-contained C engine using 249 KB SRAM and 3.62 MB Flash (measured on hardware), achieving $\sim1$ token/s interactive generation. No existing MCU framework supports this capability.

3. Outperforms MCUNet. On all 10 CNN benchmarks, te-comp outperforms for both peak SRAM (avg 0.62$\times$, up to 5.9$\times$ less) and Flash (avg 0.86$\times$, up to 32% less).

4. Numerical accuracy. te-comp matches TFLite on 10 int8 models with 50-70+ layers by generating identical ground-truth label accuracy.

Background

MLIR and TOSA

MLIR (Multi-Level Intermediate Representation) [10] is a compiler infrastructure that organizes IR into dialects — named collections of operations, types, and attributes. A compilation pipeline progressively lowers operations from high-level dialects to lower-level ones (e.g., linalg $\to$ scf $\to$ arith $\to$ LLVM), with each level exposing different optimization opportunities.

TOSA (Tensor Operator Set Architecture) [2] is an MLIR dialect that provides a portable, quantized tensor operation set. It serves as the entry point for te-comp: a Python converter translates .tflite int8 models into TOSA MLIR, preserving quantization parameters (zero points, scales) as tensor operands. TOSA represents operations like tosa.conv2d with explicit quantization semantics, making it suitable as a stable input format for quantized model compilation.

Cortex-M DSP Instructions

ARM Cortex-M4 and M7 processors include DSP extensions with packed-arithmetic instructions. The SMLAD (Signed Multiply Accumulate Dual) instruction performs two 16-bit multiplications and accumulates both results into a 32-bit accumulator in a single cycle:

where $a_{\mathrm{lo/hi}}$ and $b_{\mathrm{lo/hi}}$ are the lower and upper 16-bit halves of the 32-bit registers. Combined with SXT B16 (sign-extend two bytes to halfwords) and PKHBT (pack halfwords) instructions, this enables efficient int8 MAC (Multiply-Accumulate) operations, two MACs per cycle instead of one. Standard C compilers (GCC, LLVM) do not emit SMLAD from idiomatic for-loop code. Auto-vectorization targets NEON (Cortex-A) or MVE (Cortex-M55+), not the DSP extension. Generating SMLAD-based kernels requires either inline assembly, compiler intrinsics, or, as in te-comp, a compiler IR that directly models these patterns.

MCUNet TinyEngine

MCUNet ([12, 11]) is a co-designed neural architecture search and inference engine for microcontrollers. The TinyEngine inference library uses a hybrid code generation approach:

• A Python code generator emits a model-specific dispatch function that calls kernel routines in a fixed order.

• Kernel implementations are a hand-written C library (33 source files) selected by convolution type and shape (pointwise 1$\times$1, depthwise, general).

• Memory management uses a static arena with a memory scheduler that plans buffer offsets at compile time.

TinyEngine achieves state-of-the-art inference latency on Cortex-M through partial im2col with SMLAD-based kernels, in-place depthwise convolution (MCUNetV2), and operator specialization. However, the kernel library is not generated (it must be linked in its entirety), and adding new operators requires writing new C files. The code generator uses Python string templates rather than a compiler IR, making optimizations difficult to compose and verify.

te-comp adopts MCUNet’s key algorithmic innovations (in-place depthwise, partial im2col, operator specialization) and re-implements them as composable MLIR passes that generate all kernel code from IR. Thus, it eliminates the need for a pre-written library.

Compiler Design

Figure 1 shows the te-comp compilation pipeline. The compiler processes both CNN (.tflite) and transformer (.safetensors) models through a shared multi-level IR, with different optimization pass configurations selected automatically based on model type.

The te Dialect

The te dialect defines 34 operations for quantized neural network inference, organized into four categories (Table 1). The Appendix provides the complete list of operations and their descriptions. Each operation carries all metadata required for code generation as MLIR attributes:

Category	Operations
CNN (22 ops)	conv2d, depthwise_conv2d, transpose_conv2d, fully_connected, avg_pool2d, max_pool2d, add, mul, max, min, clamp, rescale, softmax, sigmoid, tanh, hard_swish, leaky_relu, reshape, transpose, concat, pad, upsample
RNN (2)	svdf, lstm
LLM (9 ops)	matmul_q4, gather, layer_norm, gelu, rope, kv_cache_write, kv_cache_attention, kv_slide, pos_embed
Special	se_block (squeeze-and-excitation)

• Quantization: input/weight/output zero points, per-channel multiplier and shift arrays, output clamp bounds.

• Layout: strides, padding, dilation, kernel dimensions.

• Memory: arena offset that is populated by the memory planner.

• Specialization hints: kernel variant (pointwise, depthwise, general), unrolling factors, in-place flag.

A key design decision is introducing a custom dialect rather than lowering TOSA directly to loops. TOSA operations are too coarse for optimization — a tosa.conv2d followed by tosa.rescale and tosa.clamp must be fused into a single te.conv2d before memory planning can reason about activation lifetimes. The te dialect provides a level where optimization passes can attach annotations (arena offsets, specialization hints, in-place flags) that are consumed by the kernel lowering pass.

Operation Fusion

te-comp fuses multi-op TOSA patterns into single te operations, reducing both memory traffic and intermediate buffer allocations. Three classes of fusion are applied:

• Quantized convolution: A tosa.conv2d followed by tosa.rescale (per-channel requantization) and tosa.clamp (activation bounds) fuses into a single te.conv2d with multiplier/shift arrays and clamp bounds as attributes. The same pattern applies to depthwise conv2d and fully-connected.

• Five-op normalization: The converter decomposes LayerNorm(x, \gamma, \beta) into 11 TOSA operations since TOSA has no native LayerNorm. te-comp fuses them into a single te.layer_norm. With $2L+1$ normalizations in an $L$-layer transformer ($L=8$ for TinyStories), this eliminates $17 \times10 = 170$ intermediate tensors from the memory plan.

• Six-op activation: The GELU activation decomposes into a 9-TOSA-operation sequence. te-comp fuses this entire sequence into a single te.gelu, eliminating eight intermediate buffers. With one GELU per transformer layer, the $L=8$ model saves $8 \times8 = 64$ intermediate tensors.

All fusion patterns are expressed as MLIR rewrite rules and verified by the MLIR type system, unlike the Python string templates (MCUNet) or hand-written kernel selection tables (CMSIS-NN) used by prior work.

Static Arena Memory Planner

The memory planner maps every activation tensor including model I/O to offsets within a byte array at compile time. Operating on the te IR where all shapes are statically known, it produces a deterministic, zero-overhead layout with no runtime allocator. The algorithm proceeds in five phases:

• Lifetime analysis: Assign each tensor a $[def, last\_use]$ interval and byte size.

• In-place aliasing: Tensors from in-place depthwise ops or zero-copy reshapes inherit their input’s offset; stride-2 ops further shrink the root buffer after execution, freeing the tail.

• Patch-size reduction: Substitute smaller patched sizes when all consumers are patch-inference operations (§ 4.3).

• FirstFit-decreasing packing: Sort tensors by size, place each at the lowest non-conflicting offset.

• Annotation: Attach arena_offset attributes to every operation result and record te.peak_mem attribute on the function.

Following the MCUNet design ([12]), model I/O lives inside the arena; the engine exposes zero-copy access APIs so the caller reads and writes directly at the planned offsets. In addition to the arena, the planner computes three scratch buffers: im2col (two i16 columns for the SMLAD kernel), depthwise (one padded spatial plane), and Q8 activation (for te.matmul_q4) whose sizes are recorded as function attributes. Total peak SRAM equals arena + scratch.

CNN and LLM Pipelines

Both model types enter the compiler as TOSA MLIR but follow different optimization pass configurations:

• CNN: tosa-to-te → flash-weights → in-place-DW → specialize → memory-plan → te-to-loops → arm-dsp-to-llvm → emit-c.

• LLM: flash-weights → kv-cache → quantize-weights → parallel-matmul → tosa-to-te → memory-plan → te-to-loops → emit-c.

The model type is auto-detected from the input file format: .tflite triggers the CNN pipeline, .safetensors with a transformer config triggers the LLM pipeline. The same te-comp binary handles both.

Optimization Passes

We now describe the CNN-specific and LLM-specific passes. Each is a standalone MLIR transformation that can be composed, reordered, or disabled independently. The Appendix provides the complete list of passes implemented by te-comp and their descriptions.

Flash-Resident Weights

The --te-flash-weights pass marks all weight constant tensors as flash-resident by annotating them with a flash_resident attribute. During C emission, these tensors are emitted as static const arrays in .rodata, which the linker places in Flash memory. This removes weights from SRAM entirely. Thus, for MCUNet models with 0.37–1.73M parameters, this saves 370 KB–1.73 MB of SRAM that would otherwise be consumed at startup.

In-Place Depthwise Convolution

Depthwise separable convolutions process each input channel independently. When stride = 1 with same-padding, the output has the same spatial dimensions as the input. The --te-in-place-depthwise pass sets an in_place flag on qualifying depthwise operations, instructing the memory planner to alias the output tensor to the same arena offset as the input.

This optimization, first proposed in MCUNetV2 [11], halves the activation memory for every depthwise layer. In te-comp, it is implemented as a composable MLIR pass that modifies tensor lifetime annotations—the memory planner extends the input’s lifetime to cover the output’s last use, and the kernel lowering pass generates an in-place write loop.

Operator Specialization

The --te-operator-specialize pass annotates convolution operations with kernel variant hints consumed by the lowering pass:

• Pointwise 1×1: Skips the $k_h,k_w$ loops entirely (2 fewer nested loops per operation), converting the convolution to a matrix multiply over spatial positions.

• Depthwise: Annotated for a channel-first loop structure that enables in-place operation and avoids im2col.

• Channel unrolling: Operations with small channel counts receive unrolling hints to eliminate loop overhead.

Memory Planning Interaction

The --te-memory-plan pass invokes the static arena memory planner (§ 3.3). It is the downstream consumer of the CNN optimization passes above. In-place depthwise aliasing, I/O-in-arena packing, and FirstFit-decreasing bin packing combine to reduce peak SRAM by 27–83% on MCUNet models compared to naive sequential allocation (Table 2). During C emission, the arena is realized as a single static int8_t buffer[N] array in .bss, where $N$ is the peak arena size reported by the planner.

Model	MACs	#Params	MCU SRAM (KB)	te SRAM (KB)	SRAM ratio	MCU Flash (KB)	te Flash (KB)	Flash ratio
mcunet-in0	6.4M	0.75M	266	45	0.17×	889	823	0.93×
mcunet-in1	12.8M	0.64M	307	90	0.29×	992	704	0.71×
mcunet-in2	67.3M	0.73M	242	200	0.83×	878	816	0.93×
mcunet-in3	81.8M	0.74M	293	242	0.83×	897	816	0.91×
mcunet-in4	125.9M	1.73M	456	400	0.88×	1876	1823	0.97×
mbv2-w0.35	23.5M	0.75M	308	283	0.92×	862	820	0.95×
proxyless-w0.3	38.3M	0.75M	292	242	0.83×	892	824	0.92×
mcunet-vww0	6.0M	0.37M	146	56	0.38×	617	423	0.69×
mcunet-vww1	11.6M	0.43M	162	87	0.54×	689	496	0.72×
mcunet-vww2	55.8M	0.64M	311	162	0.52×	897	734	0.82×
Average					0.62×			0.85×

LLM-Specific Passes

For transformer models such as TinyStories-v2, te-comp provides the following additional passes.

Q4.0 weight quantization: The --te-quantize-weights pass quantizes fp32 weight matrices to 4-bit using a block-wise format (32-element blocks, 4.5 bits/weight effective). Thus, it achieves ∼7× Flash compression. The pass reports per-layer RMSE, SNR, and maximum error at compile time, warning if any layer drops below a configurable SNR threshold.

Int4 KV cache: The --te-kv-cache pass inserts quantization and dequantization operations around KV cache writes and reads, reducing SRAM usage by 8× compared to fp32 storage. The cache format (int4/int8/fp32) is configurable.

Vocabulary pruning: The --prune-vocab pass prunes the embedding table and LM head to a corpus-observed token set. For TinyStories-v2 with 14.6K kept tokens, the embedding shrinks from 50,257×128 to 14,632×128, saving ∼71% of embedding Flash.

Parallel matmul: The --te-parallel-matmul pass splits matrix multiplications across multiple cores via row partitioning, generating per-core loop bounds.

Automated Optimization Suggestion

An AOT (Ahead-of-Time) analysis pass is included in te-comp, --te-suggest-optimizations. It inspects the TOSA IR and produces a prioritized recommendation report. It extracts the input model signature (op types, weight formats, attention dimensions), estimates memory requirements, and cross-references against user-supplied target constraints (SRAM, Flash, core count) to produce a prioritized list of applicable passes (HIGH, MEDIUM, LOW, SKIP) with estimated savings and accuracy impact. An --auto-optimize flag applies all non-SKIP recommendations automatically.

Code Generation

After optimization, te-comp lowers the annotated te IR to loops and emits complete C source files with the --emit-c backend. This section describes the kernel lowering strategy and the output structure.

Kernel Lowering

Each te operation is lowered to nested scf.for loops operating on memref buffers. For te.conv2d, the lowering produces a 5-deep loop nest (OH, OW, OC, KH, KW). The IC dimension is folded into the SMLAD paired MAC and the actual output includes bounds checking and per-channel

// Conv2D: N=1, OH=48, OW=48, OC=32, KH=3, KW=3, IC=3
for (int oh = 0; oh < 48; oh++) {
  for (int ow = 0; ow < 48; ow++) {
    for (int oc = 0; oc < 32; oc += 2) {
      int32_t acc0 = 0, acc1 = 0;
      for (int kh = 0; kh < 3; kh++) {
        for (int kw = 0; kw < 3; kw++) {
          // Partial im2col: pack 3×2 inputs
          // into int16 pairs, then SMLAD
          int32_t a = SXTB16(input[...]);
          int32_t b = SXTB16(weight[...]);
          acc0 = SMLAD(a, b, acc0);
          int32_t b = SXTB16(weight[...]);
          // ... (2nd output channel)
        }
      }
      // Per-channel requantize + clamp
      out[oh][ow][oc] = clamp(
        (acc0 * mul[oc] + round) >> shift[oc]
        + out_zp, -128, 127);
    }
  }
}

requantize.

For pointwise 1 × 1 convolutions, the $k_h$ and $k_w$ loops are eliminated entirely (§ 4.3). Every loop bound is a compile-time integer literal because te-comp knows the exact model shape. This makes the generated code easy to optimize for standard C compilers.

The two-column SMLAD GEMM processes 2 output pixels × 2 output channels per inner iteration, achieving 2 MAC operations per SMLAD instruction on ARM Cortex-M.

C Code Emission

The EmitC backend produces the following files:

• Inference engine source: The complete engine with all kernels as nested loops, weight arrays in .rodata section and activation arena in .bss section. The arena is hoisted to file scope so that the accessor functions can return pointers into it.

• Integration layer source: The complete set of APIs to initialize, run, de-initialize the inference engine, set model input and get model output with zero copy pattern.

• Manifest: A .json file with machine-readable metadata such as arena size, weights, layer details, and MACs.

Integration Layer

The compiler generates integration layer code from templates in two configurations: RTOS task or bare-metal.

• inference_task.c: This implements a FreeRTOS task or bare-metal wrapper. It contains zero static I/O buffers because the wrapper delegates directly to the

engine’s arena accessor functions.

• inference_api.h: This declares APIs to interface with the inference engine with zero-copy pattern.

• CMakeLists.txt: This defines a drop-in CMake library target.

Integration into an existing firmware project requires three lines if the project uses a CMake build system.

// CMakeLists.txt
add_subdirectory(te_generated)
target_link_libraries(firmware PRIVATE te_generated)

// Application code (zero-copy)
inference_init(NULL);
memcpy(inference_get_input(), sensor_data,
    INPUT_SIZE_BYTES); // write into arena
inference_run(); // in-place, no args
const int8_t *out = inference_get_output();
// read from arena

This matches MCUNet’s I/O APIs with zero-copy pattern. The caller’s sensor data goes directly into the arena; the result is read in-place without any memcpy inside the engine. The entire I/O and activation storage lives in the engine’s single arena buffer.

The generated code compiles with any ARM cross-compiler (arm-none-eabi-gcc, armclang, IAR). No LLVM, no Python, no framework headers are needed at the firmware build stage.

Evaluation

All benchmark scripts, model files, test inputs, and instructions to reproduce every result in this section are publicly available at [14].

Experimental Setup

Models. We evaluate on all 10 MCUNet benchmark CNN models (7 ImageNet, 3 Visual Wake Words). The MCUNet TFLite files are byte-identical to the official releases from the MCUNet repository [12]. For LLM evaluation, we compile TinyStories-v2, GPT-Neo, 8.3M parameters [6].

Metrics. SRAM is the peak memory reported by the memory planner: arena (which includes model input, activations, and output via FirstFit packing) plus scratch buffers (im2col, depthwise pad). With zero-copy I/O, the engine allocates no staging buffers (the caller writes input directly into the arena and reads output in-place) so the arena is the total SRAM footprint. This is directly comparable to MCUNet which also packs I/O inside the allocator and exposes zero-copy APIs. Flash is the .text section (code + weights in .rodata). Both verified with arm-none-eabi-size on the final ELF binary. For the LLM deployment, memory is measured on the complete deployed firmware (engine + tokenizer + vocabulary remap + driver + startup code).

Baselines. MCUNet TinyEngine numbers are from the official MCUNet GitHub repository [12]. SRAM and Flash are profiled with their tooling.

Compile time. te-comp compiles CNN models in 1–5 seconds and LLM models in 5–15 seconds on a modern x86 workstation (single-threaded, including Python converter, MLIR passes, and C emission). This is fast enough for interactive development with an IDE plugin.

CNN: te-comp vs. MCUNet TinyEngine

Table 2 presents the full comparison on all 10 MCUNet models for both peak SRAM and Flash.

SRAM. te-comp uses less SRAM than MCUNet on all 10 models, with an average ratio of 0.62× (38% less). The largest improvement is on mcunet-in0: 266 KB → 45 KB (5.9× reduction). The gains are largest on small-input models (mcunet-in0, vww0, vww1, and vww2) where the memory planner can more aggressively overlap activation buffers, and in-place depthwise convolution eliminates a larger fraction of peak memory.

Flash. te-comp uses less Flash on all 10 models, with an average ratio of 0.85× (15% less). The savings come from generating only the kernel code needed for the specific model, whereas MCUNet links its entire 33-file kernel library. The largest saving is on mcunet-vww0: 617 KB → 423 KB (31% less).

Numerical Accuracy

We verify accuracy by comparing te-comp’s generated C engine against the TFLite reference interpreter (v2.14) on all 10 int8 models, using 10 real test samples per model from Imagenette [8], MCUNet’s VWW minival [12], and standard test sets. The generated engine is compiled with gcc -O2 on the host with target=generic and compared element-wise against TFLite. During development, we identified and fixed five numerical bugs in the TFLite-to-TOSA conversion (requantize rounding, Pad+Conv spatial mapping, negative per-channel scales, softmax approximation, average-pool rounding). Table 3 presents the results after these corrections.

Model	Out	Tens.	Max LSB	RMSE	Elem. err	TFL→GT	Eng→GT
Deep mobile architectures (50–70+ layers)
mcunet-320kb-vww	2	353	2	1.118	65.0%	9/10	9/10
mcunet-10fps-vww	2	225	3	1.449	75.0%	9/10	9/10
mcunet-5fps-vww	2	241	3	1.449	85.0%	9/10	9/10
mbv2-w0.35-r144	1000	278	7	1.403	70.5%	10/10	10/10
mcunet-256kb-in	1000	289	7	1.608	74.0%	10/10	10/10
mcunet-5fps-in	1000	225	7	1.625	75.0%	10/10	10/10
proxyless-w0.3-r176	1000	326	7	1.519	73.6%	10/10	10/10
mcunet-10fps-in	1000	241	9	1.928	78.2%	9/10	9/10
mcunet-320kb-in	1000	289	9	1.412	69.6%	10/10	10/10
mcunet-512kb-in	1000	273	9	1.643	75.2%	10/10	10/10

The ten deep mobile architectures (50–70+ layers) exhibit 2–9 LSB maximum error from accumulated rounding through $O(50)$ requantize operations. Each operation is within tolerance of 1–2 LSB but compounding across the network. Despite element-level differences, ground-truth label accuracy is identical to TFLite on all 10 MCUNet models (9/10 or 10/10 on both engines for every model).

LLM: TinyStories-v2 on Cortex-M7

Memory. te-comp compiles TinyStories-v2 (GPT-Neo, 8.3M parameters, 8 layers, $d_{\mathrm{model}} = 128$, 16 attention heads) to a self-contained C inference engine for a single-core ARM Cortex-M7 at 160 MHz (NXP S32K344: 512 KB SRAM, 4 MB Flash, $25 automotive MCU).

Table 4 shows the memory breakdown measured on the final ELF binary deployed to the S32K3X4EVB-T172 evaluation board. The complete firmware including model weights, tokenizer, vocabulary remap table, inference engine, NXP RTD drivers, and startup code uses 3.62 MB of Flash (90.5% of 4 MB) and 249 KB of SRAM (48.5% of 512 KB), leaving headroom for application logic.

Component	Size	Section
Q4_0 weights + pruned embedding	3.02 MB	.rodata
BPE tokenizer blob	230 KB	.rodata
Generated engine + firmware code	382 KB	.text
Total Flash (.text)	3.62 MB	Fits 4 MB
Int4 KV cache + scales	108 KB	.bss
Activation arena	58 KB	.bss
Tokenizer + sampler + UART buffers	83 KB	.bss
Total SRAM (.bss)	249 KB	Fits 512 KB

Throughput. On the S32K3X4EVB-T172 board (single Cortex-M7 core at 160 MHz), the generated engine achieves approximately 1 token/s for autoregressive text generation (a single inference run generating one token takes approximately 1150ms). Each token requires ∼16.6M Q4 multiply-accumulate operations through 8 transformer layers. The engine runs bare-metal with no OS. A video demonstration of interactive multi-turn conversation on the physical board is available at [15].

KV Cache. The compiler’s --te-kv-cache pass detects the attention pattern in the TOSA IR (Q/K/V projections → reshape to heads → matmul → softmax → matmul) and replaces it with stateful te.kv_cache_write and te.kv_cache_attention operations. These lower to C code that maintains a persistent int4-quantized KV cache with a runtime position counter, enabling autoregressive generation where each call processes one token and attends over all previous tokens. The position counter auto-increments in engine run function and resets in engine initialization function, providing correct multi-turn conversation.

Related Work

MCUNet TinyEngine ([12, 11]). Closest to our work. It uses neural architecture search (NAS) to find MCU-friendly models and a Python code generator + hand-written C kernel library for deployment. te-comp replaces both the Python generator and the kernel library with a proper MLIR compiler. We adopt MCUNet’s key optimizations (in-place depthwise, partial im2col, operator specialization) and implement them as composable, verifiable MLIR passes. Our evaluation shows te-comp outperforms MCUNet on all 10 benchmark models for both SRAM and Flash usage.

TFLite Micro ([5]). The standard interpreter-based MCU runtime. It supports a wide range of operators via an op resolver that dispatches to pre-registered kernels. The interpreter overhead (dispatch tables, tensor arena management, op resolver) adds 20–50 KB SRAM. te-comp eliminates this overhead entirely by generating model-specific code with no interpreter loop.

CMSIS-NN ([9]). ARM’s hand-optimized kernel library for Cortex-M, commonly used as a backend for TFLite Micro. It provides efficient SMLAD-based convolution kernels. te-comp generates equivalent kernels from IR rather than linking a pre-written library, enabling per-model specialization.

ExecuTorch ([13]). Meta’s PyTorch-native edge deployment framework. It supports MCUs via an instruction-list interpreter (13–26 KB runtime) with pluggable kernel backends. MCU support is demonstrated only on MNIST with CMSIS-NN delegation. In this architecture, te-comp could serve as an ExecuTorch backend delegate for MCUs.

Apache TVM / microTVM ([4]). Compiler-based ML framework with emerging Cortex-M support. It still requires a TVM C runtime with operator dispatch. Autotuning (AutoTVM, MetaSchedule) targets Cortex-A NEON or GPU architectures, not Cortex-M DSP instructions.

IREE ([7]). Google’s production MLIR compiler for ML workloads. It targets GPUs, CPUs (x86, ARM Cortex-A), and accelerators. It requires an OS with threading and memory allocation; MCU targets are not supported.

Glow ([1]). Facebook’s AOT neural network compiler. It supports AOT compilation to C as te-comp does and targets Cortex-A and Cortex-M, but via the support of CMSIS-NN.

cardputer-ai (cardputer-ai contributors, 2024). A hand-written C++ LLM inference engine for the ESP32-S3 (Xtensa LX7, 8 MB PSRAM). It demonstrates that LLM inference is feasible on MCUs, but the engine is hand-tuned for a single architecture. te-comp automates this: the same compiler generates LLM engines for multiple architectures from a standard model file.

Conclusion

We presented te-comp, the first complete MLIR-based compiler that generates self-contained, dependency-free C inference engines for MCUs. The approach is demonstrated with the following results:

Self-contained C code generation. For each input model, te-comp emits all C artifacts needed for deployment: the inference engine with all kernels and weights, the RTOS integration layer, a public API header, and a CMake library target. The output compiles with any cross-compiler and integrates into existing firmware in a few lines of code. No interpreter, no kernel library, no framework dependencies. We believe this is the right abstraction for MCU deployment: a complete, model-specific C library that a firmware engineer can drop into the project.

LLM inference on a single-core Cortex-M7. te-comp compiles an 8.3M-parameter GPT-Neo (TinyStories-v2) to a self-contained C engine that fits in 249 KB SRAM + 3.62 MB Flash and achieves ~1 token/s interactive text generation on a $25$ NXP S32K344 automotive MCU. The generated engine includes a stateful int4 KV cache with a 96-token context window, enabling multi-turn conversation. To our knowledge, no existing MCU inference framework supports this capability.

Performance. On 10 MCUNet CNN benchmarks, te-comp uses 38% less SRAM and 14% less Flash than MCUNet TinyEngine, with identical ground-truth label accuracy to TFLite on all models.

The key insight is that MLIR’s multi-level dialect abstraction provides the right infrastructure for MCU inference compilation: high-level enough to express fusion patterns and memory planning, low-level enough to target DSP instructions, and structured enough to emit complete, self-contained C code rather than relying on pre-written kernel libraries.

Limitations and future work. Testing and debugging te-comp with a larger set of CNN and LLM models are in progress. Lowering for Xtensa (ESP32-S3), AURIX Tricore, and their on-device performance measurements are left as future work.

The residual 2–9 LSB error on deep networks with real inputs could potentially be reduced further by adopting TF/IREE’s approach of keeping tosa.pad as a separate op (avoiding the Pad+Conv folding complexity) or by verifying rounding at each intermediate layer.

Supplementary Material

This appendix provides usage examples for the te-comp compiler (Sections A.1–A.2), followed by a complete reference of every MLIR operation (Sections A.3–A.4) and compilation pass (Section A.5) implemented by te-comp.

Compilation Pipeline Commands

te-comp accepts .tflite, .safetensors, and .mlir files directly. The examples below show the recommended pass ordering for CNN and LLM models.

CNNs (int8 TFLite model, ARM Cortex-M target).

te-comp model.tflite \
  --convert-tosa-to-te \
  --te-flash-weights \
  --te-in-place-depthwise \
  --te-operator-specialize \
  --te-weight-prepack \
  --te-patch-inference \
  --te-memory-plan \
  --convert-te-to-loops \
  --convert-arm-dsp-to-llvm \
  --emit-c --rtos baremetal \
  -o output/

This produces a self-contained C inference engine in output/ directory:

• The engine source (engine.c/.h)

• RTOS integration (inference_task.c, inference_api.h)

• A CMake target (CMakeLists.txt)

• A machine-readable manifest (manifest.json)

LLMs (SafeTensors model, ESP32-S3 dual-core target).

te-comp model.safetensors \
  --sf-config config.json \
  --sf-seq-len 1 \
  --te-flash-weights \
  --te-kv-cache \
  --te-quantize-weights \
  --te-parallel-matmul \
  --convert-tosa-to-te \
  --te-memory-plan \
  --convert-te-to-loops \
  --target=xtensa \
  --convert-xtensa-to-llvm \
  --emit-c --rtos freertos \
  -o output/

The LLM-specific passes (te-kv-cache, te-quantize-weights, te-parallel-matmul) must precede convert-tosa-to-te. In particular, te-kv-cache must run before te-quantize-weights so that the position-embedding gather pattern is recognized before weight quantization transforms it.

Auto-optimize (constraint-driven).

te-comp model.tflite \
  --auto-optimize \
  --sram-limit 128K \
  --flash-limit 512K \
  --emit-c -o output/

This produces a self-contained C inference engine in output/ directory:

--auto-optimize runs the AOT analysis, then applies all non-SKIP recommended passes automatically before proceeding to code generation.

End-to-end pipeline summary. Table 5 lists the 14 compilation steps in recommended order, indicating which steps are CNN- or LLM-specific.

#	Step / Flag	Summary
1	Input conversion	.tflite/.safetensors → TOSA MLIR (Python converters)
2	--te-flash-weights	Mark large constants as flash-resident (zero SRAM)
3	--te-kv-cache L	Insert stateful KV cache; must precede weight quantization
4	--te-quantize-weights L	Q4.0 or int8 weight quantization
5	--te-parallel-matmul L	Multi-core row splitting
6	--convert-tosa-to-te	TOSA → te with op fusion
7	--te-in-place-depthwise C	Alias DW conv output to input
8	--te-operator-specialize C	Kernel variant hints
9	--te-weight-prepack C	Im2col layout + tiling
10	--te-patch-inference C	Patch-based spatial splitting
11	--te-memory-plan	Static arena allocation
12	--convert-te-to-loops	te → scf.for + intrinsics
13	--convert-*-to-llvm	Target intrinsics → LLVM/arith
14	--emit-c	C code emission (+ RTOS glue)

Automated Optimization Suggestion

The --te-suggest-optimizations flag runs an AOT analysis that inspects the TOSA IR and produces a prioritized recommendation report. Example invocation and abbreviated output for the MCUNet-10fps-ImageNet model on a NUCLEO-G474RE board (128 KB SRAM, 512 KB Flash):

te-comp model.tflite \
  --te-suggest-optimizations \
  --sram-limit 128K --flash-limit 512K

Model type:     CNN (int8)
Total ops:      506 TOSA ops
Conv2d:         31 (30 pointwise 1x1, 1 spatial)
Depthwise:      15

Est. peak SRAM:        187.5 KB (budget: 128.0 KB -> OVER)
Est. weight flash:     787.1 KB (budget: 512.0 KB -> OVER)

Recommended optimizations:

[HIGH] --te-in-place-depthwise
       15 depthwise convs; peak SRAM over budget
       Saving: 1.4 MB | Accuracy loss: 0pp

[HIGH] --te-memory-plan
       static arena allocation reduces peak SRAM
       Accuracy loss: 0pp

[MED]  --te-operator-specialize
       30 pointwise 1x1 convs, 31 total conv2d
       Accuracy loss: 0pp

[MED]  --te-patch-inference
       largest activation exceeds half SRAM budget
       Saving: 93.8 KB | Accuracy loss: ~0.2pp

[HIGH] --te-flash-weights
       weights: 805988 B, >70% of flash budget
       Accuracy loss: 0pp

[SKIP] --te-kv-cache (no attention patterns)
[SKIP] --te-quantize-weights (already int8)
[SKIP] --te-parallel-matmul (single-core)

Aggregate accuracy loss: ~0.2pp

The analysis runs three phases: (1) model signature extraction (op inventory, model type detection, attention patterns, weight types), (2) resource estimation (peak activation SRAM, weight Flash, scratch buffers), and (3) cross-referencing the signature and estimates against the target constraints to produce prioritized recommendations (HIGH/MED/LOW/SKIP) with estimated savings and accuracy impact.

The complete pipeline ordering is summarized in Table 5.

te Dialect Operations

Table 6 provides a brief description of each of the 34 operations in the te dialect.

Category	#	Operation	Description
CNN (22 ops)	1	conv2d	Quantized 2D conv with fused bias, per-channel requantize (mult+shift), and activation clamp.
CNN (22 ops)	2	depthwise.conv2d	Depthwise 2D conv (channel-wise filter). Supports in-place flag for output-aliased execution.
CNN (22 ops)	3	transpose.conv2d	Quantized transposed (deconvolution) 2D conv with fused requantize and clamp.
CNN (22 ops)	4	fully.connected	Quantized dense layer: clamp(requant($xW^T + b$)).
CNN (22 ops)	5	avg.pool2d	Quantized average pooling with optional fused rescale.
CNN (22 ops)	6	max.pool2d	Max pooling (operates directly on int8 values; no rescale).
CNN (22 ops)	7	add	Quantized element-wise add with per-input rescale (handles differing scales).
CNN (22 ops)	8	mul	Quantized element-wise multiply with output rescale.
CNN (22 ops)	9	max	Element-wise maximum of two tensors.
CNN (22 ops)	10	min	Element-wise minimum of two tensors.
CNN (22 ops)	11	clamp	Clamp to [min, max] (implements ReLU, ReLU6, etc.).
CNN (22 ops)	12	rescale	Per-channel or per-tensor requantization (mult+shift+zero-point).
CNN (22 ops)	13	softmax	Quantized softmax via LUT with configurable $\beta$ mult/shift.
CNN (22 ops)	14	sigmoid	Quantized sigmoid (logistic) via LUT.
CNN (22 ops)	15	tanh	Quantized tanh via LUT.
CNN (22 ops)	16	hard.swish	Quantized hard-swish: $x \cdot\operatorname{relu6}(x + 3)/6$.
CNN (22 ops)	17	leaky.relu	Quantized leaky ReLU with configurable $\alpha$ (mult/shift).
CNN (22 ops)	18	reshape	Zero-copy reshape (reinterprets buffer layout).
CNN (22 ops)	19	transpose	Dimension permutation.
CNN (22 ops)	20	concat	Concatenation of variadic inputs along a given axis.
CNN (22 ops)	21	pad	Constant-value padding (spatial H/W).
CNN (22 ops)	22	upsample	Nearest-neighbor upsampling by integer scale factors.
RNN (2)	23	svdf	SVD Filter: decomposes dense layer into feature and time components (KWS).
RNN (2)	24	lstm	Standard LSTM cell with forget, input, cell, and output gates.
LLM (9)	25	matmul.q4	Q4.0 quantized matmul (4-bit weights, bf16 per-block scales).
LLM (9)	26	gather	Token embedding lookup; supports Q4.0 flash-resident tables with on-the-fly dequant.
LLM (9)	27	layer.norm	Fused single-pass LayerNorm ($\gamma(x-\mu)/\sigma+\beta$).
LLM (9)	28	gelu	Fused GELU activation (tanh approx.), replaces 6-op TOSA decomposition.
LLM (9)	29	rope	Rotary Position Embedding (LLaMA-family).
LLM (9)	30	kv.cache.write	Quantize and store K/V into persistent KV cache (int4/int8/fp32).
LLM (9)	31	kv.cache.attention	Causal multi-head attention reading quantized KV cache [0, pos].
LLM (9)	32	kv.slide	Sliding-window KV cache eviction for unbounded generation.
LLM (9)	33	pos.embed	Runtime position embedding lookup via KV cache position counter.
Special (1)	34	se.block	Fused squeeze-and-excite (avg-pool$\rightarrow$FC$_1\rightarrow$ReLU$\rightarrow$FC$_2\rightarrow$sigmoid$\rightarrow$mul).

Target Intrinsic Dialect Operations

Each target architecture has a dedicated MLIR dialect with thin wrappers over hardware-specific instructions. The convert-te-to-loops pass emits the matching dialect’s operations based on the --target flag. A subsequent convert-<target>-to-llvm pass lowers these to LLVM IR or pure arith ops.

Compilation and Optimization Passes

Table 8 lists all 16 registered MLIR passes plus the AOT analysis system. The passes are listed in recommended pipeline order: LLM-specific pre-processing, TOSA→te conversion, MCUNet optimizations, memory planning, kernel lowering, target lowering.

#	Pass flag	Category	Description
	LLM pre-processing passes (run before TOSA→te conversion)
1	--te-flash-weights	LLM / CNN	Marks constants above a size threshold with te.flash_resident; memory planner skips them (zero SRAM), C emitter places them in .rodata.
2	--te-kv-cache	LLM	Detects attention patterns (reshape $[B,S,D] \to[B,S,n_h,d_h]$) and inserts te.kv_cache_write / te.kv_cache_attention ops for stateful autoregressive generation. Replaces static wpe with te.pos_embed.
3	--te-quantize-weights	LLM	Quantizes fp32 weight matrices to Q4_0 (4-bit, bf16 scales, ∼7× compression) or int8 (∼4×). Reports per-layer RMSE/SNR at compile time.
4	--te-parallel-matmul	LLM	Annotates large matmuls with row-split attributes (te.parallel_split, te.core_ranges) for multi-core execution. Supports equal and asymmetric (core-weights) splitting.
	Dialect conversion
5	--convert-tosa-to-te	Conversion	Lowers TOSA ops to te ops with pattern-based fusion: conv+rescale+clamp → te.conv2d; 5-op normalization → te.layer_norm; 6-op activation → te.gelu; residual add detection.
	MCUNet optimization passes
6	--te-in-place-depthwise	MCUNet	Sets in_place on depthwise conv2d ops where stride = 1 and same spatial dims; memory planner aliases output to input.
7	--te-operator-specialize	MCUNet	Annotates conv/FC ops with kernel variant hints: pointwise 1×1, IC/OC unrolling factors, small depthwise kernel.
8	--te-weight-prepack	MCUNet	Marks conv2d for im2col-based lowering and computes tiling hints (te.tile_oh, te.tile_ow) to fit L1 cache.
9	--te-patch-inference	MCUNet	Splits large conv2d inputs into overlapping spatial patches to reduce peak activation SRAM (patch-based inference).
10	--te-bias-offset-fusion	MCUNet	Folds the input zero-point×weight-sum offset into the bias tensor at compile time, eliminating a runtime subtraction per output element.
	Memory planning
11	--te-memory-plan	Transform	Static arena allocation: lifetime analysis → in-place aliasing → FirstFit-decreasing bin packing. Annotates each op with arena_offset. Packs model I/O into the same arena.
	Kernel lowering
12	--convert-te-to-loops	Lowering	Lowers all te ops to scf.for loops on memref buffers, emitting target-specific intrinsic ops based on --target={arm,xtensa,tricore,generic}.
	Target-specific lowering to LLVM / arith
13	--convert-arm-dsp-to-llvm	Lowering	Lowers arm_dsp.* ops to LLVM intrinsics (llvm.arm.smlad, llvm.arm.sxtb16, etc.) or pure arith patterns.
14	--convert-xtensa-to-llvm	Lowering	Lowers xtensa.* ops to Xtensa PIE assembly or library calls.
15	--convert-tricore-to-llvm	Lowering	Lowers tricore.* ops to arith patterns that the Tri-Core backend selects as MADD/PACK instructions.
16	--convert-generic-to-llvm	Lowering	Lowers generic.* ops to pure arith ops (portable fallback).
	Analysis (non-transforming)
–	--te-suggest-optimizations	Analysis	AOT model analysis: extracts model signature (op types, weight formats, attention dimensions), estimates memory, and produces a prioritized recommendation report (HIGH-MED/LOW/SKIP) with estimated savings.
–	--auto-optimize	Analysis	Runs the suggestion analysis, then applies all non-SKIP recommended passes automatically.

C Code Emission

In addition to the MLIR passes above, te-comp includes a C code emission backend (--emit-c) that translates the fully-lowered MLIR (after te-to-loops and target-to-LLVM passes) into standalone C source files. The emitter handles: arith.* → C arithmetic; scf.for/if → C loops/conditionals; memref.load/store → flat array indexing; memref.alloca/view → local arrays and arena pointer offsets; memref.get_global → .rodata constant references. An optional --rtos={freertos,baremetal} flag invokes a Python template generator that produces RTOS glue code (task wrapper, API header, CMake target) from the manifest JSON.

Dialect	Operation	Description / HW Instruction
arm.dsp	smlad	Dual 16×16 signed MAC (SMLAD)
arm.dsp	sxtb16	Sign-extend bytes 0,2 to halfwords (SXTB16)
arm.dsp	sadd16	SIMD add of two halfword pairs (SADD16)
arm.dsp	pkhbt	Pack bottom/top halfwords (PKHBT)
arm.dsp	requantize	Per-channel requant (SMULL + shift)
xtensa	ee.vmulas.s8	16× int8 vector MAC (PIE)
xtensa	mac.i32	Scalar i32 MAC (remainder lanes)
xtensa	requantize	Per-channel requant
xtensa	ee.vld.bc8	Broadcast-load byte to 16 lanes
xtensa	ee.vld.128.ip	128-bit vector load with post-incr.
xtensa	ee.vsr.32	Shift right 32-bit lanes by SAR
xtensa	ee.andq	Bitwise AND two Q registers
xtensa	ee.vsub.s8	Vector saturating subtract int8
xtensa	ee.zero.accx	Zero the ACCX accumulator
xtensa	rur.accx.0	Read ACCX low 32 bits
xtensa	_bf16.to.fp32	bf16 → fp32 (shift left 16)
tricore	madd	32×32 MAC with 64-bit intermediate
tricore	pack	Pack two 16-bit values into 32-bit
tricore	requantize	Per-channel requant (MUL.U/MADD)
generic	mac.i32	Scalar i32 MAC (arith.muli + arith.addi)
generic	requantize	Per-channel requant (pure i64 arith)