SIMD CPU dispatch
One binary runs on a 2010 Westmere CPU and on a 2023 Zen 4. Axle programs that use SIMD pick the best implementation of each dispatched kernel when the process starts, and the calls themselves carry no per-call decision.
This page explains how. The user-facing API is in SIMD and auto-vectorisation. Here we look at the dispatch model underneath it.
What this page covers: why one binary must adapt to the CPU under it · the microarchitecture levels it chooses between · the slot, trampoline and variant model · the pointer-based kernel ABI · when the probe runs · what a dispatched call looks like in the emitted IR · which shapes are dispatched and which are not.
The problem
Newer CPUs add wider SIMD instructions over time: SSE to AVX2 to AVX-512 on x86-64, and NEON on ARM. A binary that uses an AVX2 instruction runs faster on a chip that has it. On a chip without it, the same binary stops with an illegal-instruction fault. Runtime feature detection removes the dilemma. The program asks the CPU what it supports, and at startup it connects each dispatched kernel to the best variant the host can run. One binary, the best kernel everywhere.
The usual alternative is to ship one binary per level. That multiplies the deployment artefacts, and tools such as the Steam Runtime and multi-architecture Linux packages exist largely to manage that multiplication.
Microarchitecture levels
x86-64 CPUs fall into four levels, following the convention Red Hat and glibc use:
| Level | Features added on top of the previous one |
|---|---|
| Baseline (v1) | SSE2, which x86-64 itself requires |
| v2 | SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT |
| v3 | AVX, AVX2, FMA, BMI1, BMI2 |
| v4 | AVX-512 F, BW, CD, DQ, VL |
Axle’s probe chooses among four answers that map onto these levels. Each answer needs more feature bits and operating-system state than the one before:
Level Needs CPU feature bits Needs OS state (XCR0)
───────── ────────────────────────────────────── ─────────────────────────
Baseline (not all of the v2 bits) —
V2 SSSE3, SSE4.1, SSE4.2, POPCNT —
V3 V2 + AVX, FMA, AVX2, BMI1, BMI2 SSE and YMM state
V4 V3 + AVX-512 F, DQ, CD, BW, VL … and opmask and ZMM state Why the OS state is part of the answer
A CPU feature bit (read with cpuid) says the processor has an instruction.
It does not say the operating system saves that instruction’s register state across a context switch. On x86-64 that second answer is in XCR0, read with xgetbv. An AVX instruction on a host whose OS has the YMM state disabled faults
with #UD, even though cpuid reports the bit. This can happen under a
hypervisor that exposes the bits, or on a Windows host with xsavedisable.
So the probe reads the leaf that reports OSXSAVE. When that bit is set, it
reads XCR0 and requires the register-state bits alongside the feature bits.
A CPU that answers no cpuid leaf 7 is a V2 machine, and a host whose OS saves
no YMM state stays at V2, however many AVX2 bits cpuid advertises. The
decision is a pure function of the register values, so it is unit-tested against
injected registers rather than against the machine the test runs on.
Four places a CPU-feature answer can come from
The probe is one of four sources. They are not interchangeable:
| Who answers | When | What it reads |
|---|---|---|
| the target triple | always, as the default | the target’s baseline (x86-64-v2 on x86-64, generic on aarch64) the module is built for, never the build machine |
--target-cpu and --target-features | when you pass them | exactly what you asked for, overriding everything below |
--target-cpu native | only when you ask for it | the host’s own feature answer, from LLVM |
| this probe | every run of a binary that uses SIMD | cpuid and XCR0, choosing among the kernels the runtime ships |
Only the last one is Axle’s own code, and it is the one that had the OS-state problem. That is why the other three stay where they are.
Note The probe’s ladder is the one the SIMD dispatch uses. The levels are wider than vector code: V2 carries POPCNT, and V3 carries BMI1 and BMI2, which are scalar bit operations. A scalar path could use the same answer, but nothing reads those bits today. A feature that is orthogonal to the ladder, such as AES-NI, would need its own probe bit.
The model: slots, trampolines and variants
Each dispatched kernel has three parts:
generated code
│ call __axle_simd_fma_f32x8(a*, b*, c*, out*)
▼
┌──────────────────────────────┐
│ trampoline │ one stable symbol per (operation, shape)
│ p = SLOT.load(Acquire) │
│ p(a, b, c, out) │ one indirect call
└──────────────┬───────────────┘
│
┌───────────┴───────────┐
▼ ▼
baseline kernel v3 kernel (AVX2 + FMA)
(any x86-64, scalar (selected at startup when the
or SSE2 code) probe reports V3 or V4) - Variants. Each kernel has a portable baseline and, where it helps, a v3 variant compiled with AVX2 and FMA enabled. A v4 host runs the v3 variant, because the table holds no AVX-512 kernel.
- Trampoline. One stable symbol,
__axle_simd_<op>_<shape>, is what generated code calls. - Slot. One atomic pointer per (operation, shape) pair holds the function the trampoline calls. Before startup has filled it, the trampoline falls back to the baseline kernel, so a call before the install is still correct.
At process start, the probe picks the level and an install routine stores the
matching variant into every slot, with a Release store. The trampolines read
the slot with an Acquire load and then make one indirect call. The branch
predictor learns the target after the first call.
Installation is idempotent. Running it again probes the same host, picks the
same level and rewrites the same pointers. Every variant gives the same
result except a floating-point sum: its v3 variant adds adjacent lanes first,
so the last bits of the total can differ from the baseline’s in-order sum on a
host of another CPU level.
The kernel ABI: pointers, not vectors
Trampolines take pointer arguments, not vector-typed register arguments:
__axle_simd_fma_f64x4(a*, b*, c*, out*) 4 pointers
__axle_simd_reduce_fadd_f64x4(in*, out*) 2 pointers The reason is the calling convention. On x86-64 System V, a __m256 argument
travels in a YMM register. On Windows x64 it travels as a hidden pointer. Keeping
the kernels pointer-based means one ABI on every target. Each operand already
lives in a stack slot, because the source-level binding let v : f64x4 = …; lowers to one, so the spill is paid for by the binding rather than by the call.
When the probe runs
__axle_simd_init is the install entry point. The compiler emits a call to it
as the first statement of the generated main, and only when the program
uses a vector builtin. Binaries that never touch SIMD never reference the
symbol. They do not pay for the CPU probe or the slot stores.
What a dispatched call looks like in the IR
When the compiler lowers a.fma(b, c) on an f32x8, the operands are spilled to
the entry block, the trampoline is called, and the result is loaded back:
%a_ptr = alloca <8 x float>
%b_ptr = alloca <8 x float>
%c_ptr = alloca <8 x float>
%out_ptr = alloca <8 x float>
store <8 x float> %a, ptr %a_ptr
store <8 x float> %b, ptr %b_ptr
store <8 x float> %c, ptr %c_ptr
call void @__axle_simd_fma_f32x8(ptr %a_ptr, ptr %b_ptr,
ptr %c_ptr, ptr %out_ptr)
%result = load <8 x float>, ptr %out_ptr LLVM’s mem2reg and SROA passes can often remove the input slots when the
operands come from SSA values, so the pointer ABI costs little on the caller
side. Reductions follow the same shape, with one input slot and a scalar-sized
output slot.
Which shapes are dispatched
Only two families are dispatched. Everything else is a plain LLVM vector operation, legalised at compile time for the target features in force.
| Operation | Dispatched shapes | Variants |
|---|---|---|
FMA (fma) | f32x4, f32x8, f64x2, f64x4 | baseline and v3 |
Floating reductions (sum, max, min) | f32x4, f32x8, f64x2, f64x4 | sum has baseline and v3; max and min have baseline only |
That is one slot per (operation, shape) pair: four FMA slots and twelve reduction slots, sixteen in all. Each slot is filled once at startup.
A build whose target already has FMA (--target-cpu native, x86-64-v3 or
newer, any AArch64 target) does not use the FMA slots at all: fma compiles to
the fused instruction in line, with no call. The slots serve the default
x86-64 target, which cannot assume the instruction and so asks the CPU at
startup. Both give the same bits.
Not dispatched:
splat, the literal-list constructor,load,store,select,shuffle,eqandmovemaskare plain vector IR. They have no CPU-specific intrinsic that a runtime choice would improve.- FMA and reductions on shapes outside the table, such as
f32x16orf64x8and integer reductions, become LLVM’s inline vector intrinsic. They follow the compile-timetarget-featuresbaseline.
Limits / Uncabled
| Feature | State | Detail |
|---|---|---|
| Two variants per shape, not one per level | Design | the table holds a portable baseline and an AVX2-plus-FMA variant. A v4 host runs the v3 variant. |
| AVX-512 kernels | Not provided | a v4 host runs the v3 kernels; wider shapes take the inline intrinsic |
max and min reductions with an AVX2 variant | Not provided | the maximum and minimum run the baseline loop on every host |
| Dispatch for NEON or other non-x86 hosts | Not provided | outside x86-64 the probe answers Baseline, and the portable kernel is installed |
| A program that only annotates loops starts no dispatch | By design | the install call is emitted only when a vector builtin appears. A @vectorize loop is compiled for the build’s target-features. |
Dispatched shapes are f32x4, f32x8, f64x2, f64x4 | Design | a wider or narrower shape takes the inline LLVM intrinsic on the compile-time baseline |
See also
- SIMD and auto-vectorisation: the user-facing API
- Inlining and tail calls: the same idea of a compile-time decision with no per-call dispatch, applied to call sites
- Compiler internals overview: index of the internals pages
- Concept index: every SIMD concept on one page