betaFeedback

WorksComputer Science

Silent Degradation during Auto-Vectorization of bf16/fp16 on RISC-V Under LLVM

Abstract

Low-precision numeric types such as 8-bit integers, bfloat16, and half-precision floating point (fp16) are often used in AI inference workloads to reduce memory bandwidth, and RISC-V's openly specified Vector extension (RVV) is a natural target for compiling such workloads with auto-vectorization. This paper examines whether LLVM's RISC-V auto-vectorizer handles these narrow types equally well, using a multiply-accumulate loop written for fp32, int8, bf16, and fp16. Under a common baseline target (-march=rv64gcv), int8 and fp32 vectorize cleanly, while bf16 and fp16 degrade to increasingly scalar-heavy fallback code, with fp16 falling back to per-element software conversion calls, unless additional target-feature extensions (zvfbfmin/zvfbfwma, zvfhmin) are enabled. Under QEMU user-mode emulation, the baseline bf16 and fp16 kernels execute 17.0×17.0\times and 75.9×75.9\times as many dynamic instructions as their extension-enabled counterparts. Compiler diagnostics do report this degradation, but do not distinguish, in their wording, between a limitation that a compiler flag resolves entirely and one reflecting an inherent conversion cost, leaving no direct signal toward the fix. These results are supported by assembly-level instruction analysis, by direct inspection of LLVM's optimization remarks, and by dynamic instruction counts collected under emulation.

Introduction

Using lower-precision numeric types, such as 8-bit integers and 16-bit floating-point formats (e.g., bfloat16 and IEEE half-precision fp16), in place of 32-bit floating point is a growing trend as developers try to reduce memory bandwidth and increase throughput in AI inference workloads [1], [2]. Model weights are commonly distributed in bf16 or fp16, and a developer who keeps them in that format would reasonably expect the inner loops that use them to run at least as fast as their fp32 versions, since they move half as many bytes. Among these narrow types, integer and floating-point formats are not vectorized equally well by default: as shown below, an 8-bit integer type vectorizes as cleanly as standard 32-bit floating point, while 16-bit floating-point types do not, unless additional target-feature flags are enabled.

RISC-V has also become a common choice for AI edge inference, partly because of its openly specified Vector extension (RVV) [3], which lets compilers auto-vectorize loops without hand-written SIMD intrinsics. Inference runtimes such as llama.cpp now include RVV kernels [4], and recent work on RISC-V LLM inference reports that the choice of compiler, and its support for ISA extensions, changes throughput [5].

Developers targeting RISC-V with a mainstream compiler may want to know whether auto-vectorization works equally well across the low-precision types AI workloads depend on, or whether some types are handled worse than others without any clear sign in the console output.

This paper examines that question with LLVM’s auto-vectorizer, by compiling versions of a common multiply-accumulate loop (an operation of the form dst+=a∗bdst += a * b, computed in one step) for the same RVV target, changing only the numeric type.

The compiler does not treat these types the same way: integer narrowing vectorizes cleanly by default, while both floating-point narrow types degrade to scalar-heavy fallback code unless additional target-feature flags (zvfbfmin/zvfbfwma for bf16, zvfhmin for fp16) are enabled, and the two floating-point types degrade in different, increasingly severe ways. The degradation is silent in a practical sense: the build succeeds, the output is numerically correct, and no warning is printed unless optimization remarks are requested. The compiler’s remarks, while accurate, do not distinguish between a degradation that a compiler flag fixes and one that reflects an inherent cost of the operation, so nothing in the text points toward the fix.

The extensions and their effects are documented, actively developed LLVM and RISC-V features [6], [7], [8]. This study describes, at the assembly and diagnostic level, how vectorization quality degrades across three AI-relevant narrow types under the same method, measures the resulting difference in executed instructions under emulation, traces the degradation to the stage of the LLVM pipeline where it is decided, and identifies a specific gap in how compiler remarks report that degradation to developers.

Background

Narrow types in AI inference

bf16 keeps fp32’s 8-bit exponent and truncates the mantissa to 7 bits, so an fp32 checkpoint can be converted to bf16 without rescaling; a bf16 value is exactly the upper 16 bits of the corresponding fp32 bit pattern [1]. fp16 uses a 5-bit exponent and a 10-bit mantissa, giving more precision over a narrower range. int8 is used for quantized inference, where fp32 weights are scaled into a small integer range [2].

In all three cases, inference kernels usually keep the narrow type only for storage and accumulate in a wider type: 16-bit floating-point inputs are widened to fp32 before the multiply-accumulate [1], and int8 products are accumulated in int32 [2].

Why the multiply-accumulate loop matters

Most of the arithmetic in a neural network comes from a few layer types. A fully connected layer computes each output as a weighted sum of its inputs, yj=∑iWjixiy_j = \sum_i W_{ji}x_i. A convolution computes the same kind of sum over a small window of its input, and an attention layer computes it when comparing queries with keys and when weighting values. All of these are carried out as matrix multiplication, and matrix multiplication is made of one repeated step: multiply two numbers and add the product to a running total. This step is the multiply-accumulate.

Multiplying an M×KM \times K matrix by a K×NK \times N matrix performs this step MKNMKN times, so the inner loop that performs it accounts for nearly all of the kernel’s running time. Any extra work inside that loop, such as converting each operand from bf16 to fp32 one element at a time, is repeated just as many times. The classifier layer of MobileNetV2, whose weights are used in Section IV-G, maps 1280 inputs to 1001 outputs and so performs about 1.28 million multiply-accumulates for each image.

The loop studied here is this step in its simplest form: it loads two narrow operands, widens them to the accumulator type, multiplies them, and adds the result into a wider destination. Optimized matrix multiplication kernels arrange their loops differently, splitting the work into tiles that fit in registers and cache, but the innermost loop of each tile performs the same load, widen, multiply, and add sequence over a vector of elements. Whether the compiler can vectorize this sequence for a given type therefore decides whether the larger kernel can be vectorized well for that type. The loop is also small enough that its full assembly can be read and counted by hand, which is not practical for a complete matrix multiplication kernel.

These loops are usually not written by the person training or deploying the model; they are part of inference runtimes and math libraries compiled from C or C++ source for the target device [4], and the -march string used for that build decides which of the code paths shown in this paper is taken.

The RISC-V Vector extension (RVV)

RVV is RISC-V’s standard extension for vector (SIMD-style) computation [3]. Unlike fixed-width SIMD extensions such as x86’s AVX, where the vector register width is fixed and known at compile time, RVV is scalable: the actual number of elements processed per instruction is determined by the hardware at runtime via the vsetvli instruction, which the compiler emits at the start of a vectorized loop to query and configure the active vector length.

Two parameters control how a given operation is vectorized: SEW (Selected Element Width), the bit-width of each individual element (8, 16, 32, or 64 bits), and LMUL (vector register group multiplier), which controls how many physical vector registers are grouped together to act as one larger logical vector register. A higher LMUL allows more data to be processed per instruction at the cost of using up more of the register file, reducing how many independent vector values can be live at once. The compiler’s cost model must estimate the tradeoff correctly to make good vectorization decisions, particularly for operations such as widening multiply-accumulate that produce a wider result type than their inputs.

Target-feature extensions for narrow types

RVV’s base vector extension (v) guarantees vector support for standard integer and floating-point types, and integer narrowing operations (e.g. int8 inputs widening to an int32 accumulator) are covered by this base support. Narrow floating-point formats are handled differently: because not all RISC-V implementations support native bf16 or fp16 arithmetic, RVV specifies them as separate, optional extensions layered on top of v: Zvfbfmin (minimal bf16 vector load/convert support) and Zvfbfwma (bf16 widening multiply-add) for bfloat16 [6], and Zvfhmin for fp16. A compiler targeting only the base v extension cannot assume these are available, and must fall back to other ways of generating code for loops that use these types. The RVA23U64 application profile makes Zvfhmin mandatory but leaves Zvfbfmin and Zvfbfwma optional [9].

Compiler auto-vectorization and diagnostics

LLVM’s Loop Vectorizer analyses each loop and, using a target-specific cost model, decides whether vectorizing is likely to be profitable, and at what width. Clang reports these decisions through optimization remarks: -Rpass=loop-vectorize reports successful vectorization and its width, -Rpass-missed reports loops that failed to vectorize, and -Rpass-analysis reports the cost-model reasoning behind these decisions. These remarks are the main way a developer, without reading the generated assembly, can learn how the compiler handled a given loop.

Where the decision is made

Figure 1 shows the stages a kernel passes through. Clang parses the source into an AST, in which each (float) cast is a floating-point conversion node, and lowers it to LLVM IR as a narrow load, an fpext to float, and an llvm.fmuladd.f32 call. The Loop Vectorizer then chooses a vector width. Before choosing a scalable width, it asks the target, through RISCVTargetTransformInfo, whether every element type in the loop is legal in a scalable vector. On RISC-V, half is legal only when Zvfhmin is enabled and bfloat only when Zvfbfmin is enabled. If any type is rejected, scalable vectorization is ruled out, the remark “Scalable vectorization is not supported for all element types found in this loop” is emitted, and the vectorizer falls back to a fixed width (here 4). Treating half and bfloat as legal under Zvfhmin and Zvfbfmin alone, rather than only under the full Zvfh extension, was added to LLVM in November 2024 [7].

Path of fma_fp16 through Clang and LLVM

Figure 1. Path of fma_fp16 through Clang and LLVM. The Loop Vectorizer queries RISCVTargetTransformInfo for whether each element type is legal in a scalable vector and for the cost of each operation. Without Zvfhmin, half is rejected, the vectorizer falls back to a fixed width, and instruction selection later expands each conversion into a library call. bf16 follows the same path, with Zvfbfmin in place of Zvfhmin and integer shifts in place of calls.

The fixed-width vector IR still contains a vector fpext from 16-bit to 32-bit elements. Instruction selection has no single RISC-V instruction for this conversion without the extension, so it splits the operation into scalar conversions. For bf16, each scalar conversion is an integer shift that places the 16 bits in the upper half of an fp32 register, followed by reinsertion into a vector. For fp16, the base target also lacks scalar half-precision support, so each scalar conversion becomes a call to the runtime library routine __extendhfsf2. With the extension enabled, the same IR lowers to vfwcvt.f.f.v for fp16, and the widening and multiply-accumulate combine into vfwmaccbf16.vv for bf16.

Related Work

Prior work has studied LLVM’s RISC-V auto-vectorization support using general-purpose benchmark suites. A 2025 study by researchers at LLNL evaluated vectorization coverage across the Test Suite for Vectorizing Compilers (TSVC), comparing LLVM and GCC across RVV and ARM SVE targets [10]. A separate 2025 study evaluated auto-vectorization performance on physical RVV-capable RISC-V hardware, using TSVC loops alongside several real-world applications, and found cases where LLVM failed to vectorize loops that GCC handled successfully [11]. More recently, a 2026 study compared GCC 15 and Clang 21 code generation across scientific and ML workloads, including SGEMM/DGEMM, and found that default LMUL selection is close to optimal in most cases, identifying predication overhead and strided memory access, rather than LMUL choice, as the primary sources of remaining performance gaps [12].

These studies evaluate general-purpose or full-precision numeric workloads and do not specifically isolate the behavior of low-precision, AI-relevant types such as bf16, fp16, or int8. Separately, LLVM’s own RISC-V maintainers have described ongoing work to support vectorized code generation for the Zvfbfmin/Zvfbfwma and Zvfhmin extensions, noting that widened bf16/fp16 operations need extra care in the cost model because of their effect on LMUL and register pressure [8]. This shows that the mechanism examined in this study is a known, active area of compiler development, but its effect on generated code quality, and on the clarity of the resulting compiler diagnostics, has not been measured in the studies above. The change that made half and bfloat legal for scalable vectorization under the minimal extensions was tested by compiling TSVC with bf16 element types [7], but that work concerns correctness of the extension-enabled path rather than the cost of the baseline one.

On the inference side, a 2025 study porting llama.cpp to the many-core SG2042 RISC-V CPU found that Clang 19 outperformed GCC 13.2 by 34% in token generation, and attributed the gap in part to ISA extension support [5]. llama.cpp’s bf16 RVV kernels using Zvfbfwma have so far been tested only under QEMU [4]. These reports show that toolchain configuration affects inference performance on RISC-V, but do not isolate the per-type effect measured here.

Research on compiler diagnostics has found that developers do read compiler error messages, and that messages which are harder to read lead to worse task performance [13]. That work concerns error messages; this study applies the same concern to optimization remarks.

This study differs from prior work in two ways: it focuses on narrow, AI-relevant types rather than general-purpose benchmark suites, and it evaluates not only whether vectorization succeeds but how clearly the compiler explains the result when it does not succeed cleanly.

Methodology

All kernels were compiled with Clang 22 (22.1.8) for riscv64-unknown-linux-gnu, the same version used for the measured binaries in Section IV-G. The analysis was also repeated with Clang 23.1.3, the current release at the time of writing, and with an LLVM 24 development build from August 2026. Both produce the same remarks and the same fallback code; under Clang 23.1.3, every vector loop body reported below has the same instruction count except the baseline fp16 loop, which is one instruction shorter. Common flags across all runs were -O3 -mabi=lp64d, plus diagnostic flags -Rpass=loop-vectorize -Rpass-missed=loop-vectorize -Rpass-analysis=loop-vectorize to show the vectorizer’s cost-model reasoning. Each kernel was compiled under a baseline target (-march=rv64gcv) and, where applicable, again with a narrow-type extension added, to isolate that extension’s effect on generated code.

Methodology flowchart showing MAC loop source, baseline and extended compilation, assembly output, instruction mix, and optimization remarks

Figure 2. Methodology: source variants, baseline vs. extended target-feature compilation, and the two static analysis channels. Execution measurement is described in § 4-G.

This study evaluates a single loop shape, an elementwise fused multiply-accumulate over an array, instantiated across four input types, along with one reduction-shaped kernel (dot_int8, discussed below). The elementwise shape was chosen because it matches the innermost loop of common AI operations, as described in Section II-B, while staying small enough to read the full assembly by hand.

Base loop

Listing 1. Base FMA loop template, instantiated per type below

void fma_<TYPE>(<ACC_TYPE> *dst, const <TYPE>
*a, const <TYPE> *b, int n) {
    for (int i = 0; i < n; i++) {
        dst[i] += (<ACC_TYPE>)a[i] *
                  (<ACC_TYPE>)b[i];
    }
}

fp32

fp32 (TYPE = float, ACC_TYPE = float), reference case

Listing 2. fp32 instantiation, used as a reference for standard-width vectorization

void fma_f32(float *dst, const float *a,
             const float *b, int n) {
    for (int i = 0; i < n; i++) {
        dst[i] += a[i] * b[i];
    }
}

Flags: -march=rv64gcv (no additional extension evaluated, since this type is fully supported by the base vector extension and serves as a reference point for standard-width vectorization quality).

int8

int8 (TYPE = signed char, ACC_TYPE = int)

Two int8 kernels were evaluated. The first, dot_int8, is structured as a reduction (accumulating into a scalar sum) rather than an elementwise array update, since integer dot products are the usual form for int8 quantized inference (e.g., in quantized GEMM/dot-product microkernels):

Listing 3. int8 instantiation, as a dot product (reduction)

int dot_int8(const signed char *a, const
             signed char *b, int n) {
    int sum = 0;
    for (int i = 0; i < n; i++) {
        sum += (int)a[i] * (int)b[i];
    }
    return sum;
}

Flags: -march=rv64gcv (no additional extension evaluated; see Results §V).

Because dot_int8’s reduction structure differs from the elementwise fma_* kernels used for the other three types, a second, elementwise-only int8 kernel, fma_int8, was also evaluated, matching the base loop template exactly:

Listing 4. int8 instantiation, elementwise (topology-matched to fma_f32/bf16/fp16)

void fma_int8(int *dst, const signed char *a,
              const signed char *b, int n) {
    for (int i = 0; i < n; i++) {
        dst[i] += (int)a[i] * (int)b[
        i];
    }
}

Flags: -march=rv64gcv.

dot_int8 is kept because it reflects the usual form of this operation in practice, while fma_int8 gives a direct, structure-matched comparison with the other three fma_* kernels.

bf16

D. bf16 (TYPE=__bf16\mathrm{TYPE} = \_\_bf16, ACC_TYPE=float\mathrm{ACC\_TYPE} = \mathrm{float})

Listing 5. bf16 instantiation

void fma_bf16(float *dst, const __bf16 *a,
              const __bf16 *b, int n) {
    for (int i = 0; i < n; i++) {
        dst[i] += (float)a[i] * (
                  (float)b[i];
    }
}

Flags: -march=rv64gcv (baseline) and -march=rv64gcv_zvfbfmin_zvfbfwma (extension added).

fp16

E. fp16 (TYPE=_Float16\mathrm{TYPE} = \_\mathrm{Float16}, ACC_TYPE=float\mathrm{ACC\_TYPE} = \mathrm{float})

Listing 6. fp16 instantiation

void fma_fp16(float *dst, const _Float16 *a,
              const _Float16 *b, int n) {
    for (int i = 0; i < n; i++) {
        dst[i] += (float)a[i] * (
                  (float)b[i];
    }
}

Flags: -march=rv64gcv (baseline) and -march=rv64gcv_zvfhmin (extension added).

Summary of runs

F. Summary of runs

KernelFlagsRole
dot_int8rv64gcvBaseline (reduction)
fma_int8rv64gcvBaseline (elementwise)
fma_f32rv64gcvBaseline (reference)
fma_bf16rv64gcvBaseline
fma_bf16rv64gcv_zvfbfmin_zvfbfwmaExtension added
fma_fp16rv64gcvBaseline
fma_fp16rv64gcv_zvfhminExtension added

Table 1. ALL COMPILATION RUNS PERFORMED

Execution measurement

G. Execution measurement

To measure the effect of the generated code rather than infer it from static listings, each of the seven configurations in Table I was compiled into a standalone benchmark and run under emulation. Binaries were built on an Ubuntu Linux host with Clang 22.1.8 for riscv64-unknown-linux-gnu, using the same -O3 -mabi=lp64d and -march strings as above, statically linked against the Ubuntu RISC-V cross glibc. They were executed with QEMU 9.1.0 [14] in user mode (qemu-riscv64), configured as an rv64 CPU with the vector extension, VLEN = 256, and ELEN = 64, with zvfbfmin and zvfbfwma, or zvfhmin, also enabled for the extension builds. Emulation is also how llama.cpp’s bf16 RVV kernels were tested, since their developers did not have hardware implementing Zvfbfwma [4].

Every kernel operates on arrays of N=147,456N = 147{,}456 elements (384×384384 \times384). Input values were taken from the largest weight tensor (the classifier weight matrix) of a pretrained MobileNetV2 [15] checkpoint and converted once from fp32 to each type: bf16 and fp16 by round-to-nearest conversion, and int8 by symmetric scaling to [−127-127, 127127]. All kernels therefore consume the same underlying values, and the second operand is a copy of the first. Each binary calls its kernel 100 times over the full array (14.7 million elements in total) and was run 5 times; medians are reported.

Two quantities were recorded. The primary one is the dynamic guest instruction count, collected with QEMU’s libinsn plugin, which counts every RISC-V instruction the program executes, including startup, file input, and initialization, which are common to all binaries and small relative to the kernel. The secondary one is host wall-clock time of the timed kernel region, measured with clock_gettime inside the guest. Instruction count is used for comparison because QEMU translates each guest instruction into host code or a software helper routine whose host cost varies by instruction and does not correspond to the cost of that instruction on any RISC-V core.

The RISC-V runtime library on the build host does not provide __extendhfsf2, which the baseline fp16 binary calls. A portable C implementation of the routine (bit manipulation on the 16-bit input, with handling for zero, subnormal, infinite, and NaN inputs) was linked into both fp16 binaries; only the baseline binary calls it.

Results

Determining vectorization outcomes

A. Determining vectorization outcomes

Vectorization outcomes are judged using two signals. First, Clang’s -Rpass=loop-vectorize remark directly reports whether a loop was vectorized and at what width. A fixed integer (e.g. 4) indicates compile-time-fixed vectorization, while vscale x N indicates scalable, vector-length-agnostic vectorization. This remark alone, however, does not capture quality: a loop may be classified as vectorized while still containing substantial scalar overhead. For this reason, the generated assembly is also inspected for the share of the loop body taken up by real vector instructions (vle*.v, vfmacc.vv, vwmacc.vv, etc.) versus scalar reconstruction work (per-element scalar loads, vslideup.vi sequences used to manually assemble vector operands, or calls to software conversion routines such as __extendhfsf2). This distinction is what allows fma_bf16 and fma_fp16 under baseline flags to be characterized as degraded rather than simply vectorized or not vectorized, a classification the remark text alone does not support.

Baseline vectorization outcomes

B. Baseline vectorization outcomes

Table II summarizes vectorization outcomes for all kernels under the baseline target -march=rv64gcv.

KernelVectorized?WidthNotes
dot_int8Yes (clean)vscale x 4No degradation remarks
fma_int8Yes (clean)vscale x 4No degradation remarks
fma_f32Yes (clean)vscale x 4No degradation remarks
fma_bf16Yes (degraded)4 (fixed)Manual vector reconstruction
fma_fp16Yes (degraded)4 (fixed)Scalar __extendhfsf2 calls

Table II. Vectorization outcome under baseline -march=rv64gcv.

Effect of narrow-type extensions

C. Effect of narrow-type extensions

Table III shows the effect of adding the relevant narrow-type extension for the two kernels that degraded under the baseline.

KernelExtension addedWidth (before → after)Degradation removed?
fma_bf16zvfbfmin+zvfbfwma4 → vscale x 4Yes
fma_fp16zvfhmin4 → vscale x 4Yes

Table III. Effect of adding minimal narrow-type vector extensions.

For fma_fp16, the compiler’s diagnostics show this distinction directly. Under the baseline flags, the vectorizer reports:

remark: Scalable vectorization is
not supported for all element types
found in this loop.

alongside:

remark: floating point conversion
changes vector width. Mixed
floating point precision requires
an up/down cast that will
negatively impact performance.

After adding zvfhmin, the first remark disappears entirely and vectorization width changes from a fixed 4 to vscale x 4, confirming that this remark corresponded to a limitation a flag can fix. The second remark, however, appears unchanged in both runs, since it reflects an inherent cost of the fp16-to-fp32 conversion rather than a missing hardware capability. Both remarks are emitted through the same -Rpass-analysis=loop-vectorize channel with similar phrasing, so nothing in the text tells the developer that one can be fixed with a compiler flag and the other cannot.

Instruction-level comparison

To support the “degraded” classification beyond the compiler’s own remark text, the instructions in each kernel’s loop body are counted and classified as either productive (vector loads/stores and the actual arithmetic: vle*.v, vse*.v, vfmacc.vv, vwmacc.vv, vfwmaccbf16.vv, vfwcvt.f.f.v) or overhead (everything required only to work around a missing capability: scalar per-element loads, vslideup.vi sequences used to hand-assemble a vector, register spills, or software conversion calls). Figure 3 places the three resulting code paths side by side.

Per-iteration loop bodies for the three code paths observed

Figure 3. Per-iteration loop bodies for the three code paths observed. White boxes are productive vector work, gray boxes are overhead present only because the extension is absent, and dashed boxes are loop control. Path (a) is shown for fp16; the bf16 extension build is similar, with a single vfwmaccbf16.vv replacing the two conversions and the vfmacc.vv. Measured values are dynamic instructions per element from Table IV.

bf16, baseline. The loop body processes a fixed 4 elements per iteration using 39 instructions. Of these, only 3 are productive (vle32.v, vfmacc.vv, vse32.v); the rest are 8 scalar half-word loads (lhu), 8 scalar-to-vector element inserts (vmv.s.x), 8 shift instructions preparing each half-word, and 6 vslideup.vi instructions manually assembling the four scalar-loaded elements into vector registers. The compiler must repeat this overhead on every iteration.

bf16, with zvfbfmin+zvfbfwma. The loop body shrinks to 14 instructions, of which 5 are productive (vle32.v, two vle16.v, vfwmaccbf16.vv, vse32.v); the remaining instructions are address arithmetic and loop control, not per-element reconstruction. The loop also moves from a fixed width of 4 to vscale x 4, so each iteration also covers a number of elements set at runtime (usually larger). Both the per-iteration instruction count and the number of iterations improve at once.

fp16, baseline. This case is more severe. Before the loop begins, the function saves 13 integer registers and one floating-point register to the stack, a sign of high register pressure. The loop body is 114 instructions long and performs 8 explicit calls to the software routine __extendhfsf2 (one per scalar element requiring conversion; 4 elements × 2 operands), each surrounded by register moves and vector-register spill/reload pairs (vslr.v/vllr.v) to preserve state across the call. Function-call overhead of this kind (branch-and-link, callee save/restore, return) is much more expensive per element than the register-only shuffling seen in the bf16 baseline.

fp16, with zvfhmin. The loop body drops to 17 instructions, replacing all 8 software conversion calls with 2 hardware widening-convert instructions (vf即cvt.f.f.v), one per input array, with no calls, no spills, and no manual element assembly.

int8, baseline (dot_int8). By comparison, this loop body uses 12 instructions to process an already-scalable (vscale x 4) chunk per iteration, of which 5 are productive (two vle8.v loads, two vsext.vf2 widening sign-extends, and vwmacc.vv), with a single vredsum.v reduction after the loop and no scalar reconstruction of any kind, consistent with its clean vectorization remark and requiring no additional extension flags.

int8, baseline (fma_int8, elementwise). This element-wise counterpart to dot_int8 uses 15 instructions to process a vscale x 4 chunk per iteration, of which 7 are productive: two vle8.v loads, two vsext.vf2 widening sign-extends, one vle32.v load of the accumulator, one vwmacc.vv widening multiply-accumulate, and one vse32.v store. The rest are address arithmetic and loop control, with no scalar reconstruction. This matches the clean outcome observed for dot_int8 despite the different loop structure (elementwise update versus reduction), confirming that int8’s clean vectorization is not a side effect of the reduction structure but holds for the same elementwise shape used by fma_f32, fma_bf16, and fma_fp16.

fp32, baseline (reference). This loop body uses 13 instructions to process a vscale x 4 chunk per iteration, of which 5 are productive: three vle32.v loads (both inputs and the accumulator), one vfmacc.vv, and one vse32.v store. The rest are address arithmetic and loop control, with no scalar reconstruction of any kind. This serves as the reference case against which the narrow-type results are compared: fma_bf16 and fma_fp16, under baseline flags, do not reach this level of code quality despite processing narrower data, while dot_int8, fma_int8, and the extension-enabled fma_bf16/fma_fp16 do.

These counts cover each vector loop body, from its label to its closing branch, in the static assembly. They exclude the scalar loop that handles leftover elements in the fixed-width cases. Their purpose is to make the severity ordering concrete and checkable against the assembly: fp32 ≈ int8 (both dot_int8 and fma_int8) < bf16 (baseline) < fp16 (baseline), with both floating-point narrow cases resolved by the corresponding extension. The next subsection measures the same ordering dynamically.

Measured execution cost

Table IV and Figure 4 report the measured results. Run-to-run variation was below 0.001% in instruction count and below 9%

Dynamic instructions executed per array element

Figure 4. Dynamic instructions executed per array element (log scale), from Table IV.

BinaryInsns (M)Insns/elemvs. fp32Wall (s)
fma_f328.600.581.00×0.62
fma_int810.240.691.19×0.60
dot_int88.400.570.98×0.28
fma_bf16 base143.949.7616.74×1.44
fma_bf16 ext8.470.570.98×1.01
fma_fp16 base921.8762.52107.19×15.37
fma_fp16 ext12.150.821.41×0.95

Table 4. TABLE IV. MEASURED EXECUTION UNDER QEMU (MEDIAN OF 5 RUNS, 14.7M ELEMENTS PER RUN)

in wall-clock time. The baseline fma_bf16 executes 17.0× as many instructions as its extension-enabled build, and the baseline fma_fp16 executes 75.9× as many as its extension-enabled build. With the extensions, both narrow floating-point kernels execute within 1.41× of the fp32 kernel’s instruction count, and the bf16 build executes slightly fewer instructions than fp32, while reading half as many input bytes. Without them, bf16 executes 16.7×16.7 \times and fp16 107×107 \times the instructions of fp32 for the same elements.

The measured counts are consistent with the static counts. The baseline bf16 loop body of 39 instructions per 4 elements corresponds to 9.75 instructions per element, against a measured 9.76. The baseline fp16 loop body of 114 instructions per 4 elements accounts for 28.5 of the measured 62.5 instructions per element; the rest comes from the two calls to __extendhfsf2 per element, about 17 instructions each. The clean kernels execute between 0.57 and 0.82 instructions per element, because each scalable-width iteration covers many elements at VLEN = 256.

Host wall-clock time follows the same ordering for the baseline builds: 1.43×1.43 \times for bf16 and 16.2×16.2 \times for fp16, baseline over extension. It is not a reliable measure of relative cost, however. The bf16 extension build executes 1.6% fewer instructions than fp32 but takes 1.63×1.63 \times as long on the host, meaning QEMU’s implementation of vfwmaccbf16.vv costs more host time per instruction than its fp32 counterpart. This reflects the emulator’s implementation, not RISC-V hardware.

Discussion

The results show that narrow AI-relevant types are not vectorized equally well under a single -march=rv64gcv target. Integer narrow types (int8 widening to int32) vectorize as cleanly as standard-width fp32, with no additional flags, because integer widening is covered by the base vector extension. Floating-point narrow types (bf16, fp16) instead require separate, easily overlooked opt-in extensions (zvfbfmin+zvfbfwma, zvfhmin) to reach the same quality of vectorization; without them, the compiler still emits a vector loop, but pads it with scalar work to reconstruct or convert operands one element at a time. In the fp16 case, it falls back to a software conversion routine per element, a much more expensive failure than bf16’s in-register shuffling.

For developers deploying AI models, the practical consequence is that storing weights in bf16 or fp16 to save memory can make the hot loop slower than leaving them in fp32. Under the baseline target, the bf16 kernel executes 16.7×16.7 \times the instructions of the fp32 kernel while reading half the input bytes, and nothing in the build output, the program’s results, or its default diagnostics shows this. With the extensions enabled, the narrow types deliver the expected benefit: fp32-level instruction counts with half the memory footprint.

This difference is not made clear to a developer relying on compiler diagnostics. The vectorizer correctly detects and reports degraded vectorization for both narrow floating-point types, but does so using remark text that does not distinguish a fixable limitation (missing extension support, resolved entirely by adding a flag) from an inherent one (the unavoidable cost of a precision-changing cast). Both are reported through the same -Rpass-analysis=loop-vectorize channel with similar phrasing. A developer encountering the “not supported for all element types” remark has no textual indication that a two-flag change would resolve it, and could reasonably conclude that degraded performance is an inherent property of the workload rather than a missing target-feature flag. As Figure 1 shows, the remark originates from a single legality check in RISCVTargetTransformInfo that depends only on the enabled extensions, so the compiler has the information needed to name the missing extensions in the remark. Clang already attaches fix-it hints to front-end errors when the correction is clear; a remark naming the missing extension would apply the same idea to optimization remarks, which matters because messages that are harder to read lead developers to worse results [13]. This gap has been reported to the LLVM project, along with a proposal to name the rejected type in the remark [16].

The overhead observed in the unpatched bf16 and fp16 cases consists partly of independent scalar operations, which a wide enough out-of-order core could in principle run in parallel, partly offsetting the instruction-count penalty. However, this offset is incomplete for three reasons: instruction fetch/decode bandwidth is used up regardless of execution-level parallelism; the overhead does not scale with vector width the way productive vector instructions do, so the relative gap widens on hardware with wider vector units; and part of the overhead (the sequential vslideup.vi chain in the bf16 case, and the call overhead in the fp16 case) is inherently serial and cannot be parallelized regardless of core width. This offset is also weaker in practice for the edge-inference setting that motivates this study, where RISC-V implementations are often simpler, narrower-issue cores with less instruction-level parallelism than desktop-class hardware.

Developer Tips

The following checks apply to developers building AI inference code for RVV hardware with Clang.

Compiler version. Scalable vectorization of half and bfloat under Zvfhmin and Zvfbfmin alone was added to LLVM in November 2024 [7]. With older releases, adding these extensions to -march may not produce the scalable loops measured here, so a compiler release that includes this change is needed.

Matching -march to the hardware. On Linux, the isa line of /proc/cpuinfo lists the extensions the processor implements. If it includes zvfhmin, zvfbfmin, or zvfbfwma, these should be added to the -march string. They should not be added otherwise, since a binary that uses these instructions stops with an illegal-instruction fault on hardware that lacks them. For binaries distributed to hardware that is not known in advance, Clang supports function multi-versioning on RISC-V through the target_clones attribute, which builds several versions of a function and picks one at run time.

Profile coverage of bf16. Targeting the RVA23U64 profile enables Zvfhmin, which resolves the fp16 case, but Zvfbfmin and Zvfbfwma are optional in that profile [9] and must be requested separately. Hardware support is also limited: llama.cpp’s Zvfbfwma kernels have so far been tested only under QEMU [4]. On hardware without these extensions, bf16 loops take the baseline path measured in this paper.

Remarks for performance-critical kernels. Building performance-critical files with -Rpass=loop-vectorize -Rpass-analysis=loop-vectorize shows the degradation without reading assembly. Two outputs indicate a missing extension: the remark “Scalable vectorization is not supported for all element types found in this loop,” and a fixed vectorization width such as 4 where vscale x N is expected. The remark “floating point conversion changes vector width” appears with or without the extension and does not by itself indicate a missing flag.

Conversion calls in the assembly. Compiling a kernel with -S and searching for call __extendhfsf2 (or __truncsfhf2 for the reverse conversion) identifies fp16 loops that convert in software. For bf16, the signature is a loop with per-element lhu loads and vslideup.vi sequences and no vfwmaccbf16.vv or vfwcvtbf16.f.f.v.

Storage type and target. If the target lacks the extensions, the measured options for this loop are: fp32 at 0.58 instructions per element with twice the memory of bf16; int8 at 0.57 to 0.69 instructions per element, since integer widening needs only the base vector extension; bf16 at 9.76; and fp16 at 62.5. Converting bf16 or fp16 weights to fp32 once at load time moves the conversion out of the inner loop at the cost of memory.

Measurement under QEMU. Wall-clock time under QEMU reflects the emulator’s implementation of each instruction, and in this study it ranked the bf16 extension build behind fp32 despite nearly equal instruction counts. QEMU’s instruction-count plugin gives a more stable comparison of code paths.

Limitations

This study has several limitations. First, it evaluates a single compiler, Clang. The static results were checked with three LLVM versions, but the measured binaries were built only with Clang 22.1.8. Second, execution was measured under emulation, not on RISC-V hardware. Dynamic instruction counts weight all instructions equally and do not model pipelines, caches, memory bandwidth, or the per-element cost of vector instructions on a particular implementation, and results were collected at a single vector length (VLEN = 256). Third, the absolute count for the baseline fp16 binary depends on the implementation of __extendhfsf2 linked into it; a different runtime library would change the count, though not the number of calls per element. Fourth, the kernel set is narrow (one or two FMA-shaped kernels per type) and does not establish how broadly this pattern holds across other loop shapes (reductions beyond dot_int8, strided access, larger AI kernels such as full GEMM tiles). The question of whether the same asymmetry holds on GCC’s RISC-V vectorizer is also not addressed, so it is not known whether this is an LLVM-specific gap or a broader one across compilers.

Conclusion

This study finds that LLVM’s RISC-V auto-vectorizer does not handle AI-relevant narrow numeric types equally well under a single -march=rv64gcv target: int8 kernels vectorize cleanly by default, while bf16 and fp16 fused multiply-accumulate kernels degrade to increasingly scalar-heavy fallback code unless additional target-feature extensions are enabled. Under emulation, the degraded bf16 and fp16 kernels execute 17.0× and 75.9× the instructions of their extension-enabled builds, and the extension-enabled builds come within 1.41× of fp32. The fallback is triggered by the element-type legality check the vectorizer performs through RISCVTargetTransformInfo. Compiler diagnostics accurately report degradation but do not distinguish between a limitation that a compiler flag fixes and one that reflects an inherent cost of the operation, giving developers no direct signal toward the fix.

Future work could extend this analysis by measuring the same kernels on physical RISC-V hardware that implements Zvfhmin and Zvfbfwma, and at more than one vector length; evaluating additional loop shapes beyond fused multiply-accumulate, such as reductions or strided access patterns common in convolution and attention kernels; and testing whether the same diagnostic ambiguity is present in GCC’s RISC-V vectorizer, to determine whether this is an LLVM-specific gap or a broader pattern across RISC-V compiler toolchains.

References

  1. [1]D. Kalamkar et al., “A Study of BFLOAT16 for Deep Learning Training,” arXiv preprint arXiv:1905.12322, 2019.arxiv.org/abs/1905.12322
  2. [2]B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2704–2713.arxiv.org/abs/1712.05877
  3. [3]RISC-V International, RISC-V “V” Vector Extension, Version 1.0, 2021. Available: https://github.com/riscv/risc-v-spec
  4. [4]taimur-10x, “llamafile: add rvv support for sgemm kernels,” llama.cpp pull request #18199, merged Dec. 2025. Available: https://github.com/ggml-org/llama.cpp/pull/18199
  5. [5]J. J. Poveda Rodrigo, M. A. Ahmadi, A. Burrello, D. Jahier Pagliari, and L. Benini, “V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V Platforms,” in Proc. 22nd ACM Int. Conf. on Computing Frontiers (CF), 2025, 2025. arXiv:2503.17422.arxiv.org/abs/2503.17422
  6. [6]RISC-V International, RISC-V BF16 Extensions (Zfbfmin, Zvfbfmin, Zvfbfwma), Version 1.0, ratified 2024. Available: https://github.com/riscv/riscv-bfloat16
  7. [7]L. Lau, “[RISC[V] Enable scalable loop vectorization for zvfhmin/zvfbfmin,” LLVM pull request #115272, merged Nov. 2024. Available: https://github.com/llvm/llvm-project/pull/115272
  8. [8]L. Lau and A. Bradbury, “Improvements to RISC-V Vector Code Generation in LLVM,” RISC-V Summit Europe, 2025.
  9. [9]RISC-V International, “RVA23 Profiles,” ratified specification, 2024. Available: https://docs.riscv.org/reference/rv a23/rva23-profiles.html
  10. [10]H.-M. Lai, P.-H. Lin, M. B. Gokhale, I. Peng, H. Patel, and J.-K. Lee, “RISC-V Vectorization Coverage for HPC: A TSVC-Based Analysis,” in Proc. SC’25 Workshops, 2025, pp. 1676–1683.DOI
  11. [11]L. Carpentieri, M. VazirPanah, and B. Cosenza, “A Performance Analysis of Autovectorization on RVV RISC-V Boards,” in 2025 33rd Euromicro Int. Conf. on Parallel, Distributed, and Network-Based Processing (PDP), 2025, pp. 129–136.DOI
  12. [12]R. Shi et al., “Closer in the Gap: Towards Portable Performance on RISC-V Vector Processors,” arXiv preprint arXiv:2605.10860, 2026.arxiv.org/abs/2605.10860
  13. [13]T. Barik, J. Smith, K. Lubick, E. Holmes, J. Feng, E. Murphy-Hill, and C. Parnin, “Do Developers Read Compiler Error Messages?” in Proc. 39th Int. Conf. on Software Engineering (ICSE), 2017.DOI
  14. [14]F. Bellard, “QEMU, a Fast and Portable Dynamic Translator,” in Proc. USENIX Annual Technical Conf., FREENIX Track, 2005, pp. 41–46.
  15. [15]M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4510–4520.arxiv.org/abs/1801.04381
  16. [16]V. Kourou, “[LV][RISCV] ‘Scalable vectorization is not supported for all element types’ remark does not name the unsupported type or the extension that would enable it,” LLVM issue #232987, Oct. 2026. Available: https://github.com/llvm/llvm-project/issues/232987

Paper details

vlnc:W_djswxek9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents