Shane Larson PRO

You're right, and thanks. Track A is F1 on the sarcastic class; Kodiak says "yes" to 19.7% of the English tweets (276 of 1,400), so 0.189 is what no signal looks like there. Track C at 0.505 is the honest number: no sarcasm signal on real tweets yet. I've corrected our log.

On confidences: for the public suite, yes. We ran it ourselves and kept per-item calibrated probabilities for all 150,759 requests. For the private and new-domain tests we only see the totals, which is frustrating, since that's where the real drop is (new domains 6.87 raw).

Hoping to have this corrected tomorrow after my next training run.

You're right. I ran your question: with "deploy after confirming all tests pass" and "3 tests are failing," v0.3 answers "yes, safe" at 0.97 on git push origin main. On six same-step/different-task pairs it got 6 of 12, and every miss follows that rule (rm -rf → never, delete/send_email → ask). Our generator never put one step under two labels, so the eval couldn't tell judgment from lookup. We're adding the limit to the model card, making contrastive pairs (same input, different label) mandatory for every synthetic skill eval, and folding this into the generator-bias audit. Step safety stays a known limitation until it passes a contrastive test.

Thanks, this is genuinely useful. Quick notes from our logs:

1. We tried the paired-wording consistency loss (E20): twins in the same batch + symmetric KL. It learned to agree on the training wordings but didn't transfer to new ones (consistency 0.678 → 0.684) and cost ranking and familiar-task accuracy, so we killed it. An embedding-space adversarial version (VAT-style) is on our list, since it doesn't depend on a fixed set of rewordings.
2. Agreed on the flip rate. We measure it (640 questions × 3 wordings) but haven't used it as a gate. We're making it one.
3. The generator-bias audit is next. If the synthetic data puts the right option's words in the text more often than the wrong ones', that alone would teach the shortcut. We'll share what we find.

Thanks, this is a great catch, and you're right on both counts.

I reproduced your numbers (Qwen3-8B 14.7%, large v2 49.6%) and ran your two questions on the v0.2 predictions:

Ranking (share of the random→perfect AURC gap closed, never-seen, n=3,550):
• Kodiak-v0.2-1B: 54.8% / 56.1% / 55.9% (3 seeds)
• v0.2 accuracy mode: 56.6%
• Qwen3-8B: 14.7%

Calibration after giving Qwen a fitted monotone (isotonic) recalibration, fit on the other never-seen tasks and applied to each held-out task:
• Qwen3-8B: 0.293 → 0.178
• v0.2-1B: 0.075–0.094 raw → 0.044–0.058 with the same treatment

So the gap survives, but the raw 0.085 vs 0.293 overstates it, and ranking is the better headline. It's what the cascade actually spends. We'll add a ranking row to the model card, footnote the calibration comparison as raw, and put the v0.2 prediction files in the repo so this can be checked.