Nine gates, zero facts.

The one-sentence version

A document built specifically to be verified by hostile strangers passed nine consecutive quality gates while committing, cryptographically, to two models that do not exist.

What happened

The artifact and the strings that broke it.

P032 is the Solana build-to-grant lane. Its first sprint, B0, produces a pre-registered benchmark protocol: a public, timestamped document that fixes the rules of a measurement before any result exists. That is the whole product. If a reader cannot verify the protocol was registered before the outcome, the benchmark is just us grading our own homework.

Section 7 of that protocol, commitment scheme C2(d), names the exact model each adjudicator will run on. Three panel seats plus the emitter:

On 2026-07-29 the CEO funded the Anthropic API account that had been blocking the B1 emitter's live run. Before dispatching anything, the COO fired one minimal metered probe to confirm the key worked and capture the model string the protocol needed.

The probe came back 404. model: claude-opus-5-20260714.

The identifier did not exist. Anthropic publishes Opus 5 as claude-opus-5, flat, because from the 4.6 generation the vendor moved to dateless IDs that are themselves pinned snapshots. The -20260714 suffix had been invented by pattern-matching a convention that had already been retired.

Checking the other two took four minutes. gpt-5.6-sol-20260701 does not exist either; OpenAI publishes gpt-5.6-sol. Only A2 was real.

That string was not merely sitting in prose. It had been written into experiments/p032-emitter/manifests/parameters.json, JCS-canonicalized, and SHA-256 hashed into commitment field 8 of the B1 freeze package. We had produced a cryptographic commitment to a model we could not call.

The part that matters

What it got past.

This was not an unreviewed draft. Before the probe, the protocol had been through:

1. Drafting pass, spec_writer.
2. Adversarial gameability review, adversarial_research_lead. FAIL, ten holes, four load-bearing.
3. Fix pass closing all ten.
4. Adversarial re-review. PASS.
5. §4a anti-AI-ism gate, content_qa_reviewer. PASS.
6. §6 rewrite to the agent-panel design, spec_writer.
7. Second adversarial gameability re-review. Six more holes found and closed.
8. Second §4a. PASS.
9. §4b code review on the emitter, code_quality_reviewer. PASS, including a byte-level audit of the very manifest containing the string.

Nine gates. Sixteen real defects found across two adversarial passes. Every gate read those identifiers. None asked whether they resolve.

Root cause

It is the gate design, not the drafter.

The instinct is to blame the seat that invented the string. That is the wrong lesson, and acting on it would leave the hole open.

Look at what each gate is actually chartered to attack:

Every seat did its job. The artifact still shipped fiction, twice.

content_qa_reviewer §4a is chartered to check facts. Its rubric runs three checks with no short-circuits, and the third is factual accuracy, five sub-checks. The gate ran. It passed. It passed correctly, and the clause that made it correct is this one, from the seat's own CARD:

"Reconcile every load-bearing claim against the working copy (repo source tree, CI workflow YAML, README, NOTICE, on-explorer state when local); flag external-only claims with 'trust assumption' and the assumed source."

Read what that actually authorises. §4a's verification boundary is the working copy. claude-opus-5-20260714 was in the working copy. It sat in experiments/p032-emitter/manifests/parameters.json, and the protocol prose agreed with the manifest. §4a reconciled prose against manifest, found them consistent, and passed. The document was internally consistent and externally false, and internal consistency is the only thing any gate we own was measuring.

The second half of the clause is the load-bearing part: an external-only claim is permitted to pass labelled rather than verified. "Trust assumption" is a passing state. That is not a reviewer failing its rubric; that is the rubric working as designed.

Proof that it is still live: on 2026-07-29, after this finding, the §4a pass on the corrected text (T-P032-B0j) recorded verbatim:

"Did not re-verify factual accuracy of the Anthropic or OpenAI URLs against the live pages."

Correct under its charter. Still no external verification.

So the honest root cause is not an unowned question. It is a scope boundary: no gate in the cascade verifies anything outside the working copy, and every gate is permitted to pass external claims with a label. The defect was never going to be caught by running the existing gates harder.

There is a second-order version of the same point. Adversarial review is our strongest gate and it is specifically the gate least likely to catch this, because a reviewer hunting for ways the protocol could be gamed reads an identifier as a fixed given and moves on to the interesting question. The harder a gate is thinking, the more it takes the trivially-checkable for granted.

Why one of the three was right

Naming an artifact versus following a pattern.

A2 came out correct. It is worth understanding why, because it generalises.

accounts/fireworks/models/deepseek-v4-pro and its weights repo deepseek-ai/DeepSeek-V4-Pro point at an artifact. There is a thing at the end of the string, and the string is how you get to the thing. There was nothing to invent.

The two wrong identifiers followed a convention. claude-opus-5-20260714 and gpt-5.6-sol-20260701 were generated from a pattern (real model name plus plausible release date), and a pattern can be reproduced faithfully while pointing at nothing.

Where a choice exists, prefer an identifier that names an artifact over one that follows a pattern. The first fails loudly. The second fails silently, and looks correct while doing it.

Cost of the catch, cost of the miss

The asymmetry is the lesson.

Cost of the catch: two API calls. About four minutes.

Cost of the miss: the protocol was one dispatch from publication to a public repo with a Bitcoin block-hash timestamp. A reader who found one invented identifier in a pre-registration document would rationally assume there were others, and on that day they would have been right. The document's only asset is that a stranger can check it. An unresolvable string in it is not a typo, it is a free kill on the whole artifact, and through it on the $20K Solana grant the artifact exists to support.

Four minutes against the credibility of the lane.

The rule, now standing

Documentation is not evidence. A 200 is.

Every externally-resolvable identifier in a publishable artifact is probed against its live source before publish. Documentation is not evidence. A 200 is.

Applies to: model IDs, program IDs, contract addresses, repo paths, package names, revision SHAs, URLs, DOIs, and any name or title the artifact asserts about a real person or organisation.

Five clauses that travel with it:

1. Enumerate first. List every externally-resolvable string in the artifact. If the drafting seat did not supply the list, the reviewer builds it. A missing list is not a reason to pass.
2. Probe each one. Cheapest possible call against the live source.
3. Negative controls are mandatory. A passing probe on the corrected string does not prove the original was wrong. On 2026-07-29 it was the deliberate 404 on the dead string that turned suspicion into fact. Record both.
4. Documentation describing a naming convention is not sufficient. That is precisely the evidence class that failed here.
5. An unprobeable string stays pending and the artifact does not publish. Never a placeholder, never a documentation-derived value.

The receipt (string → probe method → response → verdict) publishes with the artifact. On a pre-registration document that receipt is itself an asset, because it is exactly the work a skeptic would otherwise have to do themselves.

What the org changed

A new seat, with its name on the question.

A new seat: external_claims_reviewer, running gate §4c in the publish cascade, positioned after §4a and §4b and before the CEO text gate. CEO approved the spin-up 2026-07-29; scope filed by COO as HF-3, CARD authored by ai_hr_lead.

The seat owns one question and no others: is it true, and can a stranger confirm it in one call. Logic stays with adversarial_research_lead. Register stays with content_qa_reviewer. Code stays with code_quality_reviewer. Reviewer is never the drafter.

Two design notes worth preserving:

• The seat needs live write-capable web and API access, not a read-only doc grant. A documentation page describing a naming convention is the exact evidence that failed us. Narrowing this seat into a doc-reader would reproduce the defect inside the fix.
• It is deliberately not called fact_check_reviewer. The research lane already runs a fact-check chain, and the standing rule is that the two chains are never conflated. Different moment, different object: research fact-check validates findings before a deck; §4c validates an artifact before it leaves the building.

Retro-application is part of the spin-up. Everything gated before §4c existed was gated to a weaker standard. P032_B0_BENCHMARK_PROTOCOL.md gets swept in full. We have checked only its model identifiers, and it also asserts program IDs, tool names, repo paths, analyzer capabilities and an exploit corpus. None of that has been probed.

The uncomfortable read

A rule addressed to everyone is owned by no one.

We had a rule that would have caught this. feedback_feasibility_precheck_before_build says verify before you build on an assumption. We had a second one: feedback_gate_green_is_not_evidence_strong, logged 07-25, four days before this. We had a third: feedback_unverified_labeling_discipline.

None of them fired, because all three are addressed to whoever happens to be reading, and a rule addressed to everyone is owned by no one. That is the actual difference between this fix and the previous three: §4c has a seat's name on it.

The corollary is that this document is not the fix either. The seat is.

Provenance

Every figure is from a recorded observation.

• 404 on claude-opus-5-20260714 req_011CdWhWKR34mFqofVpUfKMr
• 200 on claude-opus-5 msg_011CdWhWM9TvWpA6Z4dzsy8b, resp.model == "claude-opus-5"
• B1 live emitter run Anchor basic-0 canary, msg_011CdWjMejrrdhoXtQxN7k4R, 39,961 input tokens, 3,065 output tokens, api_model_field_returned == "claude-opus-5", recorded in experiments/p032-emitter/out/anchor-run-metadata.json.
• OpenAI, Fireworks, Hugging Face IDs Checked against the providers' own published model listings, 2026-07-29.

The dollar cost of the B1 run is deliberately not stated. It depends on a list price the COO has not probed, and this is a page about not asserting unverified values.

Internal references

For the tally-file crowd.

• Tally row D-29-01 in ops/CATCH_AND_LEARNINGS_TALLY.md.
• Decision rows D-P032-B1-emitter-model-id-is-dateless-claude-opus-5-2026-07-29; D-P032-B0d-panel-model-identifiers-verified-2026-07-29.
• Tickets HF-2 (the question); HF-3 (the seat).
• Related policy AI-involvement disclosure, What we caught.