Axle v0.22.0

SIMD CPU dispatch

One binary runs on a 2010 Westmere CPU and on a 2023 Zen 4. Axle programs that use SIMD pick the best implementation of each dispatched kernel when the process starts, and the calls themselves carry no per-call decision.

This page explains how. The user-facing API is in SIMD and auto-vectorisation. Here we look at the dispatch model underneath it.

What this page covers: why one binary must adapt to the CPU under it · the microarchitecture levels it chooses between · the slot, trampoline and variant model · the pointer-based kernel ABI · when the probe runs · what a dispatched call looks like in the emitted IR · which shapes are dispatched and which are not.

The problem

Newer CPUs add wider SIMD instructions over time: SSE to AVX2 to AVX-512 on x86-64, and NEON on ARM. A binary that uses an AVX2 instruction runs faster on a chip that has it. On a chip without it, the same binary stops with an illegal-instruction fault. Runtime feature detection removes the dilemma. The program asks the CPU what it supports, and at startup it connects each dispatched kernel to the best variant the host can run. One binary, the best kernel everywhere.

The usual alternative is to ship one binary per level. That multiplies the deployment artefacts, and tools such as the Steam Runtime and multi-architecture Linux packages exist largely to manage that multiplication.

Microarchitecture levels

x86-64 CPUs fall into four levels, following the convention Red Hat and glibc use:

LevelFeatures added on top of the previous one
Baseline (v1)SSE2, which x86-64 itself requires
v2SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT
v3AVX, AVX2, FMA, BMI1, BMI2
v4AVX-512 F, BW, CD, DQ, VL

Axle’s probe chooses among four answers that map onto these levels. Each answer needs more feature bits and operating-system state than the one before:

  Level      Needs CPU feature bits                  Needs OS state (XCR0)
  ─────────  ──────────────────────────────────────  ─────────────────────────
  Baseline   (not all of the v2 bits)                —
  V2         SSSE3, SSE4.1, SSE4.2, POPCNT           —
  V3         V2 + AVX, FMA, AVX2, BMI1, BMI2         SSE and YMM state
  V4         V3 + AVX-512 F, DQ, CD, BW, VL          … and opmask and ZMM state

Why the OS state is part of the answer

A CPU feature bit (read with cpuid) says the processor has an instruction. It does not say the operating system saves that instruction’s register state across a context switch. On x86-64 that second answer is in XCR0, read with xgetbv. An AVX instruction on a host whose OS has the YMM state disabled faults with #UD, even though cpuid reports the bit. This can happen under a hypervisor that exposes the bits, or on a Windows host with xsavedisable.

So the probe reads the leaf that reports OSXSAVE. When that bit is set, it reads XCR0 and requires the register-state bits alongside the feature bits. A CPU that answers no cpuid leaf 7 is a V2 machine, and a host whose OS saves no YMM state stays at V2, however many AVX2 bits cpuid advertises. The decision is a pure function of the register values, so it is unit-tested against injected registers rather than against the machine the test runs on.

Four places a CPU-feature answer can come from

The probe is one of four sources. They are not interchangeable:

Who answersWhenWhat it reads
the target triplealways, as the defaultthe target’s baseline (x86-64-v2 on x86-64, generic on aarch64) the module is built for, never the build machine
--target-cpu and --target-featureswhen you pass themexactly what you asked for, overriding everything below
--target-cpu nativeonly when you ask for itthe host’s own feature answer, from LLVM
this probeevery run of a binary that uses SIMDcpuid and XCR0, choosing among the kernels the runtime ships

Only the last one is Axle’s own code, and it is the one that had the OS-state problem. That is why the other three stay where they are.

Note The probe’s ladder is the one the SIMD dispatch uses. The levels are wider than vector code: V2 carries POPCNT, and V3 carries BMI1 and BMI2, which are scalar bit operations. A scalar path could use the same answer, but nothing reads those bits today. A feature that is orthogonal to the ladder, such as AES-NI, would need its own probe bit.

The model: slots, trampolines and variants

Each dispatched kernel has three parts:

  generated code
       │  call __axle_simd_fma_f32x8(a*, b*, c*, out*)
       ▼
  ┌──────────────────────────────┐
  │ trampoline                   │  one stable symbol per (operation, shape)
  │  p = SLOT.load(Acquire)      │
  │  p(a, b, c, out)             │  one indirect call
  └──────────────┬───────────────┘
                 │
     ┌───────────┴───────────┐
     ▼                       ▼
  baseline kernel        v3 kernel (AVX2 + FMA)
  (any x86-64, scalar    (selected at startup when the
   or SSE2 code)          probe reports V3 or V4)
  • Variants. Each kernel has a portable baseline and, where it helps, a v3 variant compiled with AVX2 and FMA enabled. A v4 host runs the v3 variant, because the table holds no AVX-512 kernel.
  • Trampoline. One stable symbol, __axle_simd_<op>_<shape>, is what generated code calls.
  • Slot. One atomic pointer per (operation, shape) pair holds the function the trampoline calls. Before startup has filled it, the trampoline falls back to the baseline kernel, so a call before the install is still correct.

At process start, the probe picks the level and an install routine stores the matching variant into every slot, with a Release store. The trampolines read the slot with an Acquire load and then make one indirect call. The branch predictor learns the target after the first call.

Installation is idempotent. Running it again probes the same host, picks the same level and rewrites the same pointers. Every variant gives the same result except a floating-point sum: its v3 variant adds adjacent lanes first, so the last bits of the total can differ from the baseline’s in-order sum on a host of another CPU level.

The kernel ABI: pointers, not vectors

Trampolines take pointer arguments, not vector-typed register arguments:

__axle_simd_fma_f64x4(a*, b*, c*, out*)        4 pointers
__axle_simd_reduce_fadd_f64x4(in*, out*)       2 pointers

The reason is the calling convention. On x86-64 System V, a __m256 argument travels in a YMM register. On Windows x64 it travels as a hidden pointer. Keeping the kernels pointer-based means one ABI on every target. Each operand already lives in a stack slot, because the source-level binding let v : f64x4 = …; lowers to one, so the spill is paid for by the binding rather than by the call.

When the probe runs

__axle_simd_init is the install entry point. The compiler emits a call to it as the first statement of the generated main, and only when the program uses a vector builtin. Binaries that never touch SIMD never reference the symbol. They do not pay for the CPU probe or the slot stores.

What a dispatched call looks like in the IR

When the compiler lowers a.fma(b, c) on an f32x8, the operands are spilled to the entry block, the trampoline is called, and the result is loaded back:

%a_ptr   = alloca <8 x float>
%b_ptr   = alloca <8 x float>
%c_ptr   = alloca <8 x float>
%out_ptr = alloca <8 x float>
store <8 x float> %a, ptr %a_ptr
store <8 x float> %b, ptr %b_ptr
store <8 x float> %c, ptr %c_ptr
call void @__axle_simd_fma_f32x8(ptr %a_ptr, ptr %b_ptr,
                                 ptr %c_ptr, ptr %out_ptr)
%result = load <8 x float>, ptr %out_ptr

LLVM’s mem2reg and SROA passes can often remove the input slots when the operands come from SSA values, so the pointer ABI costs little on the caller side. Reductions follow the same shape, with one input slot and a scalar-sized output slot.

Which shapes are dispatched

Only two families are dispatched. Everything else is a plain LLVM vector operation, legalised at compile time for the target features in force.

OperationDispatched shapesVariants
FMA (fma)f32x4, f32x8, f64x2, f64x4baseline and v3
Floating reductions (sum, max, min)f32x4, f32x8, f64x2, f64x4sum has baseline and v3; max and min have baseline only

That is one slot per (operation, shape) pair: four FMA slots and twelve reduction slots, sixteen in all. Each slot is filled once at startup.

A build whose target already has FMA (--target-cpu native, x86-64-v3 or newer, any AArch64 target) does not use the FMA slots at all: fma compiles to the fused instruction in line, with no call. The slots serve the default x86-64 target, which cannot assume the instruction and so asks the CPU at startup. Both give the same bits.

Not dispatched:

  • splat, the literal-list constructor, load, store, select, shuffle, eq and movemask are plain vector IR. They have no CPU-specific intrinsic that a runtime choice would improve.
  • FMA and reductions on shapes outside the table, such as f32x16 or f64x8 and integer reductions, become LLVM’s inline vector intrinsic. They follow the compile-time target-features baseline.

Limits / Uncabled

FeatureStateDetail
Two variants per shape, not one per levelDesignthe table holds a portable baseline and an AVX2-plus-FMA variant. A v4 host runs the v3 variant.
AVX-512 kernelsNot provideda v4 host runs the v3 kernels; wider shapes take the inline intrinsic
max and min reductions with an AVX2 variantNot providedthe maximum and minimum run the baseline loop on every host
Dispatch for NEON or other non-x86 hostsNot providedoutside x86-64 the probe answers Baseline, and the portable kernel is installed
A program that only annotates loops starts no dispatchBy designthe install call is emitted only when a vector builtin appears. A @vectorize loop is compiled for the build’s target-features.
Dispatched shapes are f32x4, f32x8, f64x2, f64x4Designa wider or narrower shape takes the inline LLVM intrinsic on the compile-time baseline

See also

simdcpu-dispatchperformance