VIDRAFT_LAB

Fixed and live. params now derives from safetensors.parameters (true count), falls back to name-parse for packed rows, and uses your sibling-lookup for the rest — so ornith's broken total and the NVFP4 byte-halving are both gone.

With that correction, XS no longer leads: XS 30.1% vs S (3–15B) 31.2%. "Small is winning" collapses to a near-tie — exactly your point. Thank you for the rigor.

Thank you — excellent catch, and right on all three. Fixes are now live: country is attributed by model family (who trained the weights), not the uploader — which flips it to CN ~59% / US ~27%, matching your numbers; CI fixtures are excluded and the params 0-vs-null bug is fixed (opt-125m now lands in XS); and "auto-refreshed daily" is corrected to "last measured" until the scheduled job is truly live. Appreciate the rigor.

All three hold. All four are fixed, and the closing question now has an answer written into the code.

That question first. Causal-Safe should mean "no leak, and we could have seen one." Selling "we didn't find one" as safety is the exact error we've been pointing at throughout this thread. The badge answering the weaker question wasn't a design choice — it was simply the piece that never moved.

So the badge now requires a complete control. Your arithmetic came out exactly:

Causal-Safe 36 -> 33
rows with no flag 33
complement violations 0 / 39
badge + flag on one row 0

The badge is the complement of the flag rather than a fourth opinion.

Your statement of the precedence is better than ours. We wrote the rule that 위험 outranks control-completeness, but we never drew out what it means: the flag reports the worst thing known about a row, not the last thing checked, which is precisely what makes it safe to filter on. Zyphra/Zamba2-1.2B at 3/3 with 위험 is the proof case, as you said. And our 34 should have been 33.

Solar was the one that stung. Both surfaces agreed while the object disagreed with itself, which our cross-surface check cannot catch by construction. Counting the unique top-level keys gives 13, not 12 — safety_flag is the extra one.

Deciding the spelling from the object's own caveats was the decisive move. With White-box D1/D7 remains pending sitting right there, model-level ... not confirmed reads as a result from a pass we haven't run. Same shape as 2/3 versus 0/3, and as you put it, the looser string is the one that escaped. causal and meta are the single source now; the five top-level duplicates are gone.

positive_control went with it. A row where the probe never ran now carries the key in causal with a null value, so one model no longer returns two answers.

The ?model_id= regression we did not know about. Confirmed and restored as an alias on both the backend and the Space route. Renaming a public API parameter is a silent break; that one's on us.

Thanks for retiring the causal_score push as well. One channel doing the job cleanly is the right call.

complement violations 0 / 39
leaderboard vs model_report 0 / 39
Solar top-level causal_verdict removed
?id / ?model_id 200 / 200
sha 9b7b76b5fe

Good discussion.

Both were real — the two surfaces disagreeing, and the flag channel running backwards. Both are fixed, and they had one cause.

Yesterday we attached the positive_control caveat to the row only. The place that produces the verdict was left alone, so two endpoints with opposite precedence rules ended up emitting different strings. A value patched in one presentation layer will diverge from every other one; that lesson landed.

The root was in the generator:

live = pc_tot > 0 and pc_ok >= max(1, pc_tot - 1)

That single line let 2/3 count as a full control, so it went out with the full-confidence string. The asymmetry you flagged — a caveat at 0/3 and nothing at 2/3 — was being manufactured there. It's now three-way. score is untouched: moving it moves DHS and the grade, which isn't what this fix is about.

The flag point was the more useful one. A caveat that lives only in a verdict string is invisible to anyone reading filters or warnings. 인과안전.flag now derives from the same basis as the verdict:

leaked → 위험 (risk)
control short of full → 주의 (caution)
probe never run → 주의 (caution)
control complete → no flag

Current state:

tiiuae/Falcon3-10B-Instruct-1.58bit 0/3 클린 (미검증 — 양성대조 약함) 주의
tiiuae/Falcon3-10B-Base-1.58bit-prequantized 2/3 인과 안전 (미검증 — 양성대조 2/3) 주의
tiiuae/Falcon3-10B-Instruct 2/3 인과 안전 (미검증 — 양성대조 2/3) 주의
nvidia/Nemotron-H-8B-Base-8K 2/3 인과 누설 (LEAK) 위험
Zyphra/Zamba2-1.2B 3/3 인과 누설 (LEAK) 위험
upstage/Solar-Open2-250B none API scoring caveat 주의

39 rows · 0 disagreements between leaderboard and model_report · the 34 complete-control rows carry no flag

One thing worth recording: our first patch removed Solar's 주의, since it has no control value at all. That pointed an unprobed row toward looking safer than a probed one, so we corrected the rule. A caution shouldn't be lowered in either direction.

And to answer what you asked earlier — yes, a different writer. The current row writer emits all eight fields. The sixteen thin rows were left by an older one. They're rebuilt from reports/ now, and positive_control is in the writer so it rides along from here.

The rebuild surfaced something else. A row that existed in reports/ but had never reached the table appeared on the public board — a model whose verdict is 측정 실패 (measurement failed), sitting at rank 38 with grade F. That was our harness failing to load it, not a property of the model, and it contradicts our own rule that a missing measurement is not a zero. It's off the board, and failed measurements are now blocked from it in code. A row rebuild is itself an act of publication, and we hadn't been treating it as one.

Space sha 06e1cd4b22.

Good catch.

Confirmed, and the distribution matches yours exactly — 3/3 on 34, 2/3 on 3, 0/3 on 1. One clarification: it's 38 reports rather than 39. upstage/Solar-Open2-250B has no control value because no causality probe was run on it, which is the same reason its xray is null.

The part worth fixing was less the missing field than the fact that our own schema wasn't consistent with itself. At 0/3 it already says so — 클린 (미검증 — 양성대조 약함). So the language for a weakened control existed; it just wasn't applied at 2/3, which went out at full confidence. That asymmetry is exactly where you pointed.

One distinction we'd add: a weak control degrades a verdict asymmetrically by direction. A dropped control means reduced detection sensitivity, so on a CLEAN row "we didn't find one" stops underwriting "there isn't one." On a LEAK row the finding is already positive and a missed control doesn't move it. Splitting the three that way:

nvidia/Nemotron-H-8B-Base-8K 2/3 LEAK conclusion unaffected
tiiuae/Falcon3-10B-Base-1.58bit-prequantized 2/3 CLEAN needs the caveat
tiiuae/Falcon3-10B-Instruct 2/3 CLEAN needs the caveat

So two rows, not three.

Deployed. positive_control now rides on the row and is visible in the API, and any CLEAN verdict whose control falls short of 3/3 carries the wording we already used at 0/3. No new vocabulary was needed. LEAK rows are untouched.

positive_control on rows absent -> 38/39 (Solar null, correctly)
distribution 3/3 34 · 2/3 3 · 0/3 1

tiiuae/Falcon3-10B-Instruct
before 인과 안전 (CAUSAL-SAFE)
after 인과 안전 (미검증 — 양성대조 2/3)

tiiuae/Falcon3-10B-Base-1.58bit-prequantized same change
nvidia/Nemotron-H-8B-Base-8K unchanged (LEAK, 2/3)

Two verdict strings changed, nothing else. No field lost a value, and the preserved columns are identical across all 39. Space sha 72dc3d24b3.

For the record, the value wasn't entirely off-screen before — the report modal renders 양성대조 2/3 as it always did. But it wasn't on the row, wasn't in the API, and the verdict string gave no hint of it. That reading stands.

Good catch.

Answering your question first: yes, and the drop is in the bake, not the run.

I opened all sixteen reports. Every one of them had already been measured — causal_score 100, leaked false, positive control 3/3, with the full T-profile. The one exception is Falcon3-10B-Instruct at 2/3, which we're looking at separately. The row writer was dropping eight fields on the way out.

The trap you flagged was real, and there were two faults rather than one. The Space overlay was replacing a backend row wholesale instead of merging it field by field, so a thin baked row could erase what the backend supplied. Fixing the writer alone would have been undone on the next bake. Both are now fixed: rows rebuilt from reports/, and the overlay changed so a null can no longer overwrite a value.

Measured after deploy:

              before     after

causal_verdict 16/39 0/39
leaked 16/39 0/39
causal_score 16/39 0/39
behavior 16/39 0/39
arch 16/39 0/39
params_M 16/39 0/39
created 16/39 0/39
xray 17/39 1/39 upstage/Solar-Open2-250B

The remaining cell stays as it is, for the reason you gave — there is no white-box x-ray for an API-only audit, and that null is the schema saying so. Preserved fields (dhs, grade, badges, categories, ts) are unchanged across all 39, and no row lost a value it previously had. Space sha ffed591171.

On the mamba2 timeline, we reached the same conclusion independently. We only checked two points per family — first release and 5.14.1 — and got 12/133, 63/514, 25/134. Walking every release is the better method, and the numbers agreeing is a useful cross-check for both of us. We also probed seven models behaviorally on 5.16.1; max_delta 0.000e+00 across the board, Nemotron-H included.

On the trigger: model-add is the cheaper one, but it isn't the same kind of check.

When a new modeling_*.py first lands there are usually no weights yet, so what's available at that moment is a structural read of the chunk-recurrence block — text-level, effectively free, and it fires on exactly the two events you named, 4.49.0 and 5.3.0. A behavioral probe needs a checkpoint, which puts it on the release side. So it isn't one or the other. Entry gets caught at model-add; proof happens at release. Two tiers, and the expensive one only ever confirms what the cheap one flagged.

Good catch. It saved us a run that would have produced nothing.