Thank you—your criticism was correct, and we tested it directly.
We ran a parallel ablation across Mistral-Nemo and Qwen2.5-7B, comparing each F16 baseline against Q4_K_M, Q5_K_M, and Q8_0. Each of 48 identical boundary scenarios was tested with the original Do not add keys wording, with that sentence removed, and with an explicit retention contract.
The original wording reproduced the asymmetry you identified. Mistral-Nemo Q4 passed 14/48 control cases while F16 passed 48/48. Qwen2.5-7B moved in the opposite direction: F16 passed 15/48 and Q4 passed 32/48. But every tested model and precision passed 48/48 when the phrase was removed, and 48/48 under explicit existing-key retention.
So the old result was not a general Q4 cliff. It was an instruction interaction that affected models and precisions differently.
We did not simply remove the constraint. We replaced that construction with a four-tier stateful family covering foundation, operational, stress, and boundary transformations. Retention and key admission are now stated separately, with an explicit allowed-key set. Calibration and locked evaluation splits are disjoint.
In the new 200-case production evaluation:
- Mistral-Nemo F16 vs Q4_K_M: 150/200 vs 80/200, with 70 F16-only passes and no Q4-only passes.
- Mistral-Nemo F16 vs Q5_K_M: 150/200 vs 150/200.
- Mistral-Nemo F16 vs Q8_0: 150/200 vs 150/200.
- Qwen2.5-7B F16 vs Q8_0: 145/200 vs 147/200, with 9 F16-only and 11 Q8-only passes; statistically inconclusive.
One limitation remains visible: Mistral scored 0/50 in the new boundary tier in both lanes, so that tier is currently non-discriminating for this model. The Mistral Q4 degradation now comes from the other constructions rather than the original one-instruction boundary artifact.
Your comment materially improved the evaluation. Thank you for reproducing the deposited data and identifying the confound.
Your observation directly motivated this ablation and the subsequent revision of the stateful_followup evaluation. If I publish the updated methodology or results, may I acknowledge you by name and reference your Hugging Face comment? If so, please let me know your preferred name, affiliation, and profile or citation link.
If you would like to receive the results and available publication/ bundles, please contact me through pbhappliedsystems.com/contact and reference these run names:
- Mistral_Nemo_Instruct_2407_20261009_033750
- Mistral_Nemo_Instruct_2407_20261009_071015
- Mistral_Nemo_Instruct_2407_20261009_091019
- Qwen2.5_7B_Instruct_20261009_191201
- Qwen2.5_7B_Instruct_20261009_215528
- Qwen2.5_7B_Instruct_20261009_221516
