# boar-bench: the hidden-thinking scorer reads 0 on every llama-server trace with recorded reasoning Submission for poidh bounty 1520 (Base), "Audit a Month of Offline AI Research". ## Summary `hidden_think_ms`, the bench's measure of invisible thinking before the first visible token, is exactly 0.0 on all 2,086 public traces from the llama-server backend in which the adapter itself recorded more than one second of hidden reasoning. In 2,002 of them the adapter recorded more than 10 seconds, the threshold of the gate rule written to catch that failure. The cause is in the code, not in the data: on the llama-server backend, reasoning never enters the token event stream that the scorer reads, and the fields the adapter writes instead (`timing.reasoningMs`, `timing.firstVisibleMs`, `timing.reasoningTokens`) are not read by any Python file of the harness. The practical consequence: on the backend that produced 28,737 of the 45,062 public traces, part (b) of the gate rule `empty-or-silent` cannot detect silent thinking, the report prints a hidden-thinking row of 0 with no warning, and the diff compares zeros. The fix is a few lines. Pinned sources: - Code: https://github.com/r4topunk/boar-bench/tree/b41a25acfabf89f1d015103ca1329554f985a257 - Traces: https://huggingface.co/datasets/r4topunk/boar-traces/tree/f96e277cf8584b38c35c01106c1fe77457dfaa81 ## Measurement All 1,062 public trace files were read (350,580,885 bytes, each checked against its SHA-256 in TRACES.tsv). The scorer logic was transcribed from `scorers/timing.py`; no repository code was executed. | Measurement | Result | | --- | --- | | Traces with events | 45,062 in 1,062 files | | Traces from the llama-server backend (`runtime.host` contains "llama-server") | 28,737 | | Traces with `timing.reasoningMs` above 1 s | 2,186 | | ... on the llama-server backend (all Qwen, 26 run folders) | 2,086 | | ... of those, scorer `hidden_think_ms` equal to 0.0 | 2,086 of 2,086 | | ... of those, with a blank token event before the first visible token | 0 of 2,086 | | ... of those, `reasoningMs` above the gate threshold of 10 s | 2,002 | | Control: node backend, gemma-4-e4b-it-q4_0, `reasoningMs` above 1 s | 100 | | ... of those, detected by the scorer (above 1 s) | 100 of 100; scorer p50 11.4 s vs adapter p50 11.3 s | By model, llama-server traces with `timing.reasoningMs` above 1 s: | `runtime.model` | Traces | Adapter `reasoningMs` p50 | Above 10 s | Scorer `hidden_think_ms` p50 | | --- | --- | --- | --- | --- | | qwen3-4b-instruct-2507-q4km | 1,853 | 52.9 s | 1,853 | 0.0 s | | qwen3-4b-thinking-2507-q4km | 192 | 12.6 s | 148 | 0.0 s | | qwen3-1.7b-q4km | 41 | 3.2 s | 1 | 0.0 s | Three single traces (the first one with reasoning in each file), to show what the scorer sees. Times in ms. | File, item | Backend | `reasoningMs` | `reasoningTokens` | `firstVisibleMs` | First token event | First visible token event | Scorer `hidden_think_ms` | | --- | --- | --- | --- | --- | --- | --- | --- | | runs-think4b/t4-ZV-v02-s42, v02-explanation-A01 | llama-server | 10,946.0 | 272 | 15,251.5 | 15,251.8 | 15,251.8 | 0.0 | | runs-deepthinkv02/dtv-TV-v02-s42, v02-explanation-A01 | llama-server | 91,076.7 | 1,034 | 121,913.2 | 121,914.2 | 121,914.2 | 0.0 | | g4eng/runs/g4eng-v01-s44, vit-hum-001 | node | 11,360.6 | 256 | 15,235.2 | 3,739.5 | 15,235.2 | 11,495.7 | On the llama-server traces the first token event is already visible text: the 11 s and 91 s of reasoning before it left no event. On the node trace, blank token events start at 3.7 s and the scorer measures the gap correctly. ## Cause 1. The scorer measures hidden thinking from token events only. `scorers/timing.py` lines 24-42, `visible_timing`: `hidden = max(0.0, vis_tok - first_tok)`, where `first_tok` is the first `token` event and `vis_tok` the first `token` event with non-blank text. It can only be above zero if blank token events precede the visible text. 2. On the llama-server backend, reasoning is never emitted as an event. `adapters/desktop-node/runner/server.ts` lines 107-108: "Chat chunks: choices[0].delta.{reasoning_content,content}; ... Visible text goes to onText; reasoning never does." Lines 123-126: a `reasoning_content` chunk only sets `firstReasoningMs` and increments `reasoningTokens`. 3. The adapter records the real value elsewhere. `adapters/desktop-node/runner/adapter.ts` lines 986-997, `hiddenTiming`, writes `timing.firstVisibleMs`, `timing.reasoningMs` and `timing.reasoningTokens`; `packages/protocol/src/index.ts` lines 127-131 declares them. Lines 338-340 of adapter.ts: "Hidden reasoning: always on the server backend (reasoning_content chunks + peak RSS); on the node backend only when a thought budget was active". 4. Nothing in the harness reads those fields. No `.py` file in the repository mentions `reasoningMs`, `firstVisibleMs` or `reasoningTokens`. 5. The author intended them to be read. boar-research, `rnd/research/anvil/specs/adapter-llama-server-backend.md` line 63: "`timing.firstVisibleMs` (= `firstTextMs`) and `timing.reasoningMs` (= `firstTextMs − firstReasoningMs` when reasoning happened). Together these are the harness TTFT fix the coordinator assigned on 2026-10-02." The adapter half of that fix shipped; the harness half did not. In the node-backend control traces the thinking shows up as blank token events before the visible text (100 of 100), which is why the scorer detects it there and why the existing gate test passes. ## What it breaks These are the outputs the code produces on such traces. It was not checked whether the author ran these commands on the affected folders. - **Gate.** `harness/gate.py` lines 16-20 describe rule `empty-or-silent` (b): "hidden thinking (scorers/timing.py: first visible token - first token) > 10 s on an item where every base run had none (< 1 s or no tokens)", and name its purpose: "Anvil's thinking-model failure: empty answers after ~19 s of invisible tokens." Lines 45-46 set `SILENT_MS = 10_000` and `NONE_MS = 1_000`; lines 59-67 apply them to `visible_timing(...)["hidden_think_ms"]`. With the value at 0 on llama-server traces, a candidate that thinks silently for 19 s, or for 90 s, passes (b). The test of the rule, `harness/tests/test_gate.py` lines 163-173, builds its case from a blank token event (`{"tMs": 1000, "type": "token", "text": ""}`), a shape the llama-server backend does not produce for reasoning, so the test passes while the rule is blind on that backend. Part (a) of the rule (empty answers) is not affected. - **Report.** `harness/report.py` lines 39-55 build the row "hidden thinking before first visible token (hidden_think_ms)", `hidden_think_p50` and the flag `hidden_thinking` (p50 above 1000 ms); lines 155 and 232-233 print the warning only with that flag. On the qwen3-4b-thinking runs the p50 is 0 and the flag is off, while the adapter recorded a p50 of 12.6 s. - **Diff.** `harness/diff.py` lines 37 and 125 compare the same zeros. ## Reproduction Three files (two llama-server, one node control), about 2 MB in total, with repro.py (https://files.catbox.moe/f8thtn.py): ```sh python3 -I repro.py \ rnd/heldout/bench-v02/runs-think4b/t4-ZV-v02-s42.traces.jsonl \ rnd/heldout/vitalik-para1/runs-think4b/t4-Z-vpara-s42.traces.jsonl \ rnd/research/coordinator/arms/g4eng/runs/g4eng-v01-s44.traces.jsonl ``` Expected (full output: https://files.catbox.moe/p72ec9.txt): the scorer reads 0 on 89 of 89 and on 51 of 51 traces with reasoning above 1 s in the two Qwen files (adapter p50 12,140.5 ms and 15,596 ms); in the Gemma control it reads 0 on 0 of 50 (scorer p50 11,416 ms vs adapter p50 11,278.1 ms). Whole dataset, with count_all.py (https://files.catbox.moe/vr0fcd.py) on a local copy of the dataset (about 350 MB): ```sh python3 -I count_all.py /path/to/boar-traces ``` Output of our run (also at https://files.catbox.moe/f6rbr6.txt): ``` files: 1062; traces: 45062 backend | model | traces | reasoning>1s | reasoning>10s | blank_tokens_first | scorer==0 | scorer>1s | scorer>10s llama-server | qwen3-1.7b-q4km | 17006 | 41 | 1 | 0 | 41 | 0 | 0 llama-server | qwen3-4b-instruct-2507-q4km | 11062 | 1853 | 1853 | 0 | 1853 | 0 | 0 llama-server | qwen3-4b-thinking-2507-q4km | 228 | 192 | 148 | 0 | 192 | 0 | 0 node | gemma-4-e4b-it-q4_0 | 120 | 100 | 89 | 100 | 0 | 100 | 89 llama-server | ALL MODELS | 28737 | 2086 | 2002 | 0 | 2086 | 0 | 0 node | ALL MODELS | 16325 | 100 | 89 | 100 | 0 | 100 | 89 ``` Both scripts only read JSONL files and transcribe `visible_timing`; they import nothing from the repository. ## Proposed fix In `visible_timing`, when `timing.reasoningMs` is a number, use it as `hidden_think_ms`; keep the event-based computation as the fallback for traces without the field. Add a gate test whose candidate trace has `timing.reasoningMs` above 10 s and no blank token events. Optionally show `timing.reasoningTokens` in the report row. ## Limits - This concerns the latency instruments (report row and warning, gate rule (b), diff). Quality scores and the ratios in the article are not affected. - The author reported the thinking cost of the think4b arm by hand in THINK4B.md ("ttft 19.9 s", reasoning tokens). Those written conclusions stand; the finding is that the automated instruments would not have shown it. - `ttft_visible_ms` is correct on these traces: its median difference from `timing.firstVisibleMs` is 0.1 ms over 23,114 traces; in 742 it is earlier by design, because an `instant` snippet counts as visible. - Whether any gate decision in the record would have changed was not established. That needs the base and candidate pairs the author actually gated. - Novelty was checked against the 31 claims listed on the bounty on 13 November 2026 at 08:38 UTC; none concerns the timing scorer or these fields. This finding does not derive from an earlier claim. ## Secondary note A separate defect in the same function (the truncation guard of the `done` fallback can be bypassed) was also examined: it occurs in 0 of 8,132 public traces with elided tokens, so it has no effect on the published runs and is not part of this claim.