Skip to content
All work

01 · LLM inference runtime

Kestrel

Llama-family inference in C++20 with CPU and Metal backends. No PyTorch, no llama.cpp. I wrote the forward pass myself and checked every layer against Hugging Face.

GitHub
CAUSAL MASK · KV CACHED
StackC++20MetalCMakeSafetensors
inference frameworks used
0
inference frameworks used
no PyTorch, llama.cpp, MLX or ONNX
worst logit error vs. HF FP32
4.1e-5
worst logit error vs. HF FP32
CPU decode
53 tok/s
CPU decode
Apple M2 Pro, batch 1
time to first token
311 ms
time to first token
16-token prompt, CPU

Why

I wanted to see every step between a prompt and a token.

Libraries make inference look like one function call. I wanted to know what the call does, so Kestrel implements the whole transformer forward pass in C++20. It loads Safetensors weights with memory mapping, runs the model on a CPU or Metal backend, and samples the next token. PyTorch, llama.cpp, MLX and ONNX Runtime are not involved.

Tokenization runs in a small separate helper. Everything after token IDs is native: the model, the KV cache, the kernels and the sampler.

Architecture

Session::prefill runs the prompt through the model. Session::decode then reuses the cached keys and values, so each new token only computes its own row of attention.

One decode step, end to end.

  1. 01

    Tokenizer helper

    text → token IDs

  2. 02

    Session

    prefill, then decode

  3. 03

    Transformer

    native forward pass

  4. 04

    CPU / Metal

    backend operations

  5. 05

    Logits

    vocab-sized vector

  6. 06

    Sampler

    greedy · temp · top-k/p

Each sampled token goes back into the session. Mapped Safetensors weights and the KV cache feed the transformer.

Inside one transformer layer

× n_layers, then final RMSNorm → LM head

  1. RMSNormpre-attention
  2. Attentionsplit-half RoPE · grouped-query · causal↔ KV CACHE
  3. Residual add
  4. RMSNormpre-MLP
  5. SwiGLU MLPgate · up · down
  6. Residual add

Weights are stored as F32, F16 or BF16 and loaded with shape checks and shard support. Activations stay FP32. The KV cache is contiguous host memory.

Correctness

Pinned TinyStories checkpoint (15.19M params) and six token IDs, compared with Hugging Face eager FP32 inference.

Checked against Hugging Face at every layer.

Fast and wrong is still wrong, so correctness came first. A reference harness compares Kestrel against Hugging Face at the embeddings, every transformer layer, the final hidden state and the logits, for all six positions. Each stage has its own tolerance, so a bug shows up at the layer where it happens, not three layers later.

BackendMax logit errorMean logit errorCosine similarity
CPU3.70e-54.12e-60.999999999999775
Metal4.12e-54.93e-60.999999999999685
The suite also includes native, malformed-input, CLI and tokenizer tests, run under UBSan.

Performance

Release build, batch 1, 16 prompt tokens, 32 generated tokens. Two warm-up runs, five measured runs. Loading and tokenization excluded.

Measured, including where it's slow.

CPUMetal

Decode throughputhigher is better

53.05tok/s
38.61tok/s

Prefill throughputhigher is better

51.47tok/s
38.49tok/s

Time to first tokenlower is better

310.91ms
415.8ms

Apple M2 Pro · pinned 15.19M-parameter FP32 TinyStories model. Local desktop measurements, not serving benchmarks.

Metal is slower than the CPU here, and I left that in. With a 15M-parameter model at batch 1, each GPU kernel has very little work. The Metal backend also uses a synchronous FP32 host interface with per-operation buffers. My working hypothesis is that allocation and synchronization cost more than the math itself. Profiling that, fusing kernels and keeping buffers on the device are next.

Run it

One command to generate text.

kestrel — zsh
# build with Metal + reference tests
cmake -S . -B build-audit -G Ninja -DCMAKE_BUILD_TYPE=Release -DKESTREL_METAL=ON
cmake --build build-audit -j 4

# greedy decode on CPU (or --backend metal)
./build-audit/kestrel --model models/tinystories-15m --backend cpu \
  --prompt "Once upon a time" --max-tokens 32 --temperature 0

Limits

Each limit is written down, so nothing in the results above is overstated.

What it doesn't do (yet).

  • Prefill runs one token at a time: no fused prompt GEMMs and no FlashAttention.
  • The KV cache is contiguous host memory: no paged allocation and no continuous batching.
  • F16/BF16 are storage formats only. Activations and GPU buffers are FP32.
  • Only selected bias-free Llama configs. Scaled RoPE and quantization are rejected.
  • The CUDA source exists but hasn't been checked on NVIDIA hardware.
  • Single-user only: no serving, no thread safety, no production-scale claims.

Next project

RepoGraph

Static-analysis engine

IMPACT ← CHANGED SYMBOL