LoadBrief v8

A LoRA fine-tune of Llama 3 8B Instruct that converts free-text athlete monitoring narratives into structured load-management briefs in a specified register (athlete, coach, or sports scientist).

Trained on tyhob/loadbrief-50k v8.

Research artifact. Not a medical device, and not for decisions about a real person's training or health. The training corpus contains documented defects that this model faithfully reproduces, and two of the three metrics originally used to evaluate it turned out to be measuring something other than what they named. Both are described below and analysed in full in the accompanying paper.

Output format

Fixed section layout: a register-dependent risk header (YOUR STATUS, RISK LEVEL, or OVERALL RISK CLASSIFICATION), then TRAINING LOAD, PHYSIOLOGICAL READINESS (HRV), SUBJECTIVE WELLNESS, SIGNAL INTEGRATION, CLINICAL CLASSIFICATION, RECOMMENDATIONS, MONITORING PRIORITIES, ESCALATION TRIGGERS. Mean output length 173 words.

Evaluation

500 completions generated greedily from the v8 held-out test split; 438 rated by an LLM judge (Gemini 2.5 Flash) on three 1–7 dimensions.

Metric	Zero-shot base	v8	Ground-truth ceiling
Composite reward	0.422 ± 0.116	0.668 ± 0.099	0.701
Risk accuracy, exact	0.000	0.960	—
Risk accuracy, within one class	0.000	0.994	—
Severe errors (≥2 classes off)	0 †	3	—
Unparseable headers	500	0	—
Overreaching accuracy	0.006	0.504	0.551 ‡
Mean words	250.0	173	176

† The base model's zero severe errors reflect 500 unparseable completions, not accuracy. ‡ Extraction ceiling — see "Overreaching is not measurable" below.

Per register, both metrics are uniform: risk at 0.958 / 0.964 / 0.958 and overreaching at 0.521 / 0.503 / 0.488 for athlete / coach / sports scientist.

Read the risk accuracy carefully

Three facts bound what 0.960 establishes.

The base model scores 0.000 because it emits no parseable header at all, not because it classifies badly. The gap from 0.000 to 0.960 is overwhelmingly the distance between unstructured prose and the corpus's brief format.

A TF-IDF bag-of-words classifier reaches 0.950 exact and 0.991 within-one-class on the same labels, from the narrative alone. A model with no capacity for inference is within one point of this fine-tune on the reported quantity.

Both corpus labels are per-scenario constants, so recovering the risk label is equivalent to recovering which of 19 scenarios generated the narrative — which a linear model does at 0.905. The metric cannot distinguish a system that reasons clinically from one that pattern-matches.

Normalized against the zero-shot floor and the ground-truth ceiling rather than against 1.0, the reward of 0.668 represents 88% of the achievable range.

Judge scores by scenario

Scenario	Clinical	Actionability	Clarity	n
high_acwr_stable_physiology	2.00	3.78	5.67	9 †
preseason_intensification	2.67	4.76	6.05	21
undertraining	2.90	3.59	5.77	39
monotony_problem	3.25	4.88	6.04	24
travel_jet_lag	3.27	3.47	5.93	15 †
double_session_accumulation	3.33	4.20	5.87	15 †
acwr_spike	3.47	5.83	6.72	36
normal_progressive	3.71	3.38	5.71	21
heat_acclimatization	3.75	5.33	6.33	24
post_competition	3.78	4.00	6.06	18
fixture_congestion	3.79	5.12	6.30	33
taper	4.26	4.30	6.22	27
recreational_minimal_data	4.67	4.33	6.33	3 †
illness_return	4.71	5.21	6.46	24
youth_growth_spurt	5.24	5.52	6.71	21
early_overreaching	5.29	5.24	6.57	21
altitude_camp	5.41	5.19	6.67	27
overtraining_syndrome	5.60	6.36	6.79	42
wellness_crash_normal_load	5.72	5.72	6.78	18
Overall	4.10	4.90	6.31	438

† n < 15, indicative only. Absolute judge values are judge-specific; a second judge family agrees on ordering (ρ = +0.76) but scores 1.19 points lower on average. Treat the ranking as robust and the levels as not.

The weak strata are inherited from the data, not introduced

Scoring the corpus's own ground-truth briefs with the identical judge, across all 19 scenarios and 3 registers (57 cells, 570 ratings):

• The model tracks its training targets at Spearman ρ = +0.83 (95% CI [+0.71, +0.89]), with a mean gap of −0.19 on the 1–7 scale.
• Reference briefs reach 6.80 and multiple cells exceed 6.0, so there is no judge ceiling depressing the weak strata.
• Scenarios carrying contradictory records average 3.34 ground-truth clinical against 5.35 for clean scenarios (ρ = −0.69).

Where the corpus is clinically incoherent this model reproduces the incoherence; where the corpus is sound it scores near it. The stratum-level weakness above is a property of the training data. The −0.19 residual is real, and individual cells are worse (double_session_accumulation at coach register is −2.70), but the pattern is inherited.

Revision history

This is the eighth corpus revision and the eighth corresponding fine-tune. The training configuration was identical across all eight — LoRA rank 32, lr 2e-4, 1 epoch, 20,000 examples, max length 512 — so the only variable in the sequence is the corpus.

These rows are not directly comparable. Each revision regenerated the corpus and therefore its own test split; all eight formatted test splits are distinct files. The varying reference ceiling is the tell. Read each row against its own ceiling, not against the row above it. The exception is sft_only_greedy / sft_v2 / sft_v2_sampled, which were scored in one evaluation run against one split.

Model	Ceiling	Reward	Risk	W-in 1	Severe	Unread.	Overr.	Clin.
zero-shot floor	0.701	0.422	0.000	0.000	0 †	500	0.006	—
sft_only_greedy (v1)	0.691	0.755	0.792	0.974	10	3	0.758	3.00
sft_only (v1, sampled)	0.691	—	—	—	—	—	—	3.24
sft_v2	0.691	0.675	0.794	0.992	4	0	0.458	4.38
sft_v2_sampled	0.691	0.660	0.702	0.948	13	13	0.352	—
sft_v3	0.712	0.735	0.842	0.994	3	0	0.518	4.09
sft_v4	0.716	0.649	0.856	0.956	22	0	0.482	4.23
sft_v5	0.701	0.720	0.934	0.982	8	1	0.552	4.23
sft_v6	0.697	0.701	0.942	0.972	3	11	0.512	4.11
sft_v7	0.704	0.720	0.948	0.986	3	4	0.554	3.89
sft_v8 (released)	0.701	0.668	0.960	0.994	3	0	0.504	4.10

† The base model's zero severe errors reflect 500 unparseable completions, not accuracy.

What actually improved

Three trends are large and monotone enough to survive the caveat.

Risk accuracy rose 0.792 → 0.960. Most of this tracks repairs to declared-target reachability: nine scenario targets produced metric ranges their own declarations did not cover at v1, and eight were repaired by v8. Making a scenario actually produce its declared range makes it easier to recognize from the narrative — which is what this metric measures.

Unparseable outputs fell to zero, from 13 at worst. Severe errors fell to 3, from a v4 excursion of 22.

What did not

Judged clinical accuracy did not move. It peaks at v2 (4.38) and sits at 4.10 in v8, with no trend across the whole repair sequence. Reachability repair makes the sampled signals match the declared range; it does nothing to make the constant label respond to those signals. Repairing eight of nine unreachable targets bought 17 points of risk accuracy and no measurable clinical quality.

Reward did not improve — v1 records the highest in the series. Reward and accuracy move in opposite directions because they measure different things; see the reward decomposition above.

Overreaching accuracy did not decline, despite appearances. The drop from 0.758 to 0.458 between the first two models is a measurement change, not a capability loss. From v2 onward the corpus phrases conflicting-signal classifications as rationale prose that names no class, so the extraction ceiling fell from 1.00 to ~0.55. Normalized against each corpus's own ceiling:

	v2	v3	v4	v5	v6	v7	v8
extraction ceiling	0.550	0.550	0.551	0.552	0.553	0.552	0.551
% of ceiling reached	83%	94%	87%	100%	93%	100%	91%

The underlying trend is flat to improving — opposite in sign to the reported one.

If you want an earlier revision

Only v8 is released. Earlier checkpoints are not published: several were not retained, they were trained on corpora that are not released, and each would need its own contemporaneous test split to evaluate meaningfully. The v1 corpus is retained under the v1 tag on the dataset repository because this card and the paper both reference it.

Limitations

Inherited: self-contradictory outputs

The training corpus contains 6,433 records (16.1%) that contradict themselves, across two disjoint families: adverse HRV prose under a LOW risk header (2,412), and SIGNAL INTEGRATION naming a different overreaching class than CLINICAL CLASSIFICATION (4,021). This model reproduces both patterns. Expect self-contradictory briefs most often in acwr_spike, undertraining, preseason_intensification and post_competition.

Inherited: CRITICAL unreachable at high load

No training record above ACWR 0.8 carries CRITICAL. This model will not assign CRITICAL to a high-load presentation regardless of physiological state; the maximum expressible severity there is HIGH. The output contains no internal contradiction when this happens, so it is not detectable from a single brief.

Overreaching classification is not measurable, not merely poor

The reported 0.504 should not be read as a capability estimate. The metric that produced it parses a class name out of generated text, and the corpus phrases that section as a rationale paragraph for the conflicting-signal classes. Applying the same metric to the corpus's own correct briefs recovers 55% of them and 0% of overtraining_syndrome. Against that 0.551 ceiling, this model sits at 91%.

Meanwhile the label is recoverable from the narrative at 0.971 by a linear classifier, so the information is present. The honest statement is not that this model fails at overreaching classification but that we cannot currently measure whether it succeeds. A label-based evaluation that does not route through free-text extraction would settle it. Until then, do not rely on this field.

Register: athlete briefs carry less clinical content

Ground-truth athlete briefs score 0.88 below coach briefs on judged clinical accuracy, and a reward decomposition independently shows the lay-vocabulary substitution forfeits 86% of the available clinical-terminology credit. The adaptation removes clinical content rather than restating it, and this model reproduces that. Relevant if you intend the athlete register for actual athletes.

General capability is uncharacterized

An earlier release measured substantial degradation after fine-tuning (MedQA 61.7 → 55.8; MMLU subjects down 4–12 points, with the largest drop on the domain-unrelated control). Those benchmarks were not re-run for v8 and the v1 figures do not transfer — different training recipe, corpus seven revisions removed. Treat this as a specialist of unknown general capability and do not use it for clinical or general question answering.

Scope

Risk accuracy is measured against synthetic labels produced by the generator described in the dataset card, including its defects. It is not evidence of clinical validity. No output of this model has been evaluated against injury or illness outcomes or reviewed by a practising sports scientist.

Intended use

Research on register-controlled structured generation, and on how synthetic-data defects and metric mis-specification propagate into fine-tuned model behaviour. Not for athlete health decisions.

Training

Base model: meta-llama/Meta-Llama-3-8B-Instruct Method: LoRA (PEFT), rank 32, α 64, dropout 0.05, applied to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. Learning rate 2e-4, cosine schedule, 3% warmup, weight decay 0.01, AdamW. Batch size 2 × grad accumulation 16 (effective 32), max sequence length 512, gradient checkpointing, 1 epoch over a 20,000-example subset of the register-expanded training split. Best checkpoint by validation loss. float16 on Apple Silicon (M4, 48 GB) via MPS; 28.7 hours.

No reinforcement-learning stage. An earlier release added GRPO and found no measurable benefit over supervised fine-tuning; the accompanying paper supplies a mechanism for that result.

Checkpoint: sft_checkpoint_v8_final.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16,
                                             device_map="auto")
model = PeftModel.from_pretrained(model, "tyhob/loadbrief").eval()

prompt = """### Monitoring Narrative:
Monitoring snapshot: Soccer Midfielder, professional. ACWR 1.71. HRV suppressed
13 ms below baseline for 6 consecutive days. Wellness moderately depressed across
sleep, fatigue, and mood.

### Audience: coach

### Load Management Brief:
"""
out = model.generate(**tokenizer(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=400, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Citation

---

Built with Meta Llama 3. Weights are subject to the Meta Llama 3 license. Training code, the reachability audit, the self-extraction check, and the judge-calibration harness are on GitHub.