First baseline · deterministic probe + agentic suite · graded n=3

Splash Engine Baseline

Splash 1.6.0 · model Qwen3.8-Flash-Next-GSQ-RCO IQ3_S · 262,144-token context · int8 KV · Apple M5 Ultra (64-core GPU, 96 GiB unified) · engine args 56 GiB working set / 128 GiB persistent SSD cache · warm prefix cache (hit rate 0.992).

201 ms
TTFT warm (341-token prompt)
380 ms
TTFT cold — prefix cache halves it
375 tok/s
aggregate plateau across widths 4→32 (2.5× single stream)
0
suspensions at 4.4× the KV budget (v4 probe)
11/12
agent tasks grader-verified
12/12
trials honest vs ground truth

Concurrency scaling — past the dip, onto the plateau

The benchmark was upgraded twice mid-session. Final rule: escalate 1→2→3→4→6→8→12→16→24→32 and stop only when aggregate falls within 10% of the single-stream baseline. It never fired — there is no collapse by width 32. Same prompt, 512 output tokens, two rounds per width, /status sampled during every round.

0100 200300 400 tok/s 1147.6 2259.1 3306.1 4377.7 6330.4 dip 8373.9 12375.7 16371.9 24382.9 ← max 32371.7 thin bars = per-stream tok/s: 147.6 → 129.5 → 102.1 → 94.5 → 55.1 → 46.8 → 31.3 → 23.3 → 16.0 → 11.7 The width-6 dip is a reproducible scheduling anomaly (327 in the v2 run, 330 here) that fully recovers by width 8. Admission pacing is healthy under load: peak waiting 2→4→8→12 with width, zero suspensions, no aggregate loss. Verdict: the ceiling is a wall, not a cliff — compute saturation at ~2.5× single-stream; pick width by latency.
Real traffic runs at width ≤2 (lifetime counters: b1 74,602 · b2 23,546 · b3 959 · b4 0), so everything from width 3 to 32 is measured, usable headroom. Also: single-stream median 132.7 tok/s on a lighter 300-token pass; decode gap p50 65.5 ms.

KV pressure — chasing the first suspension

Width alone can never suspend anything (the transport caps at 32 requests). So v4 pushes the budget that matters instead: unique long-context prefills, ~46.7k tokens each, 8 output tokens — total live token demand climbing to 611,244, four times the resident KV ring of 139,264 tokens. A suspension would show on the gauge or the lifetime counters, which catch even instant-longer-than-a-sample parkings. Neither moved.

0100k 200k300k 400k500k 600k tok resident KV budget 139,264 tok w293,394 w4186,788 w6280,182ring full: 142,272 w8373,576 w10509,370 w12611,244 · 4.4× — susp 0 thin bars = peak simultaneous live KV (tok): 52,480 · 52,448 · 142,272 · 52,512 · 60,928 · 59,104 — never starved (free pages ≥ 460). Bar totals pass the budget from w4 on; the scheduler just serializes prefills (~3.6k tok/s, linear walls) and demoted 1,050 blocks to SSD instead of spilling. One 265 ms memory-paced hold at w12 — pacing, not suspension.
Suspension needs held concurrent contexts — agent sessions sitting on big contexts for minutes — not fire-and-forget prefill volume. The engine has already done exactly that 90 times today under real use: 90 suspensions, 90 resumptions, 263k replay tokens, zero user-visible failures. It's routine park-and-replay housekeeping, not a cliff to watch for.

Agentic suite — headless Claude Code served by Qwen

Fresh session per trial, all model aliases pinned to the resident Qwen, minimum tool allowlists (curl-only for S1/S2/S3, Read-only for S4), no MCP inheritance, sequential trials. Graded by independent read-backs: constants exact-checked, arithmetic recomputed from the model's own echoed numbers, written files structurally validated, diagnosis on a rubric.

taskr1r2r3verdict
S1 status-read (six named fields)2t 13.6s2t 5.8s2t 6.1s3/3 ✓ live counters tie to grader snapshots
S2 derived arithmetic (2-dp %)2t 7.5s2t 4.6s2t 6.3s2/3 r2 ✗ reported 20.13; own inputs compute 20.11
S3 structured file from live fetch3t 7.7s3t 6.6s3t 7.6s3/3 ✓ exact shape, derived == queued+decoding
S4 diagnosis over planted fixture2t 9.0s2t 7.1s2t 6.9s3/3 ✓ all cited admission.waiting_memory unprompted

Median 2 turns / ~7 s per trial. Prefix cache did real work across fresh sessions: each read 20–45 k cached tokens against ~10 k fresh input. Honesty: every trial's claims matched grader read-backs.

Method honesty

The first suite pass was discarded — orchestrator error, not model behavior. An allowlist override blocked the intended curl route; headless sessions inherited the operator's MCP servers (two agents fetched /status through the Resolve MCP's Python sandbox); one shared work dir let blocked trials discover the diagnosis fixture. Run 2 fixed all three (per-task allowlists, --strict-mcp-config, isolated task dirs).

The discarded run still yielded a signal worth keeping: 4 of 12 blocked sessions refused to invent numbers, one volunteered that its values came from the fixture rather than a live fetch. The refusal-to-fabricate check is now part of the suite.

Next upgrade — diff procedure

Scripts + raw JSONL committed at bench/2026-11-06/. Start the engine as usual, warm it with one unscored session, then run the engine probe and width ladder, then the 4×3 driver. Compare against the header line above — same tasks, same conditions, no engine restarts mid-suite. Full spec: docs/splash-engine-benchmark-results.md.