vulnerability-attack-technique-classification-roberta-base-llm-expanded

This is a negative-result comparison checkpoint, published for reproducibility. For applications, use CIRCL/vulnerability-attack-technique-classification-roberta-base.

A multi-label classifier that suggests MITRE ATT&CK (Enterprise) techniques from a free-text vulnerability description. It is identical to the released gold-only model — same base model (roberta-base), same 53-technique label vocabulary, same seed, same evaluation protocol — except for one thing: its training set folds 984 additional LLM-labeled CVEs (CIRCL/vulnerability-attack-techniques-llm-scaling, labeled by qwen3.5:122b at ≈0.39 agreement with the expert gold labels) into the 972 expert-labeled training rows.

The paper Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion uses this pair of checkpoints to answer the question "can LLM-assisted labeling extend a small expert gold set?" — and the answer is no, not at this agreement level: no reliable ranking improvement at any expansion size from 100 to 984 CVEs, and measurable degradation of rare-technique coverage at scale.

DOI: 10.57967/hf/9624

What this checkpoint shows

Five seeds, corrected protocol (checkpoint selection on the validation split), identical test split — gold-only vs. this configuration (gold + 984 LLM rows):

Metric	Gold-only	Gold + 984 LLM
Recall@5	0.673 ± 0.019	0.651 ± 0.022
Recall@3	0.536 ± 0.032	0.534 ± 0.012
F1 micro	0.410 ± 0.006	0.427 ± 0.028
F1 macro	0.177 ± 0.014	0.151 ± 0.014

The pattern: the noisy labels concentrate mass on frequent, "obvious" techniques (micro-F1 up a little) while deflating exactly the rare-technique coverage the expert labels paid for (macro-F1 down ≈3 SEM, no recall@5 gain). On CVE-2021-44077, for example, this checkpoint is more confident than the gold model about T1190 (Exploit Public-Facing Application) but drops the analyst-credited T1505 (Server Software Component) below the prediction threshold and pushes tail techniques such as T1136 (Create Account) from rank 18 to 32. Section 6 of the paper gives the full account, including why an earlier apparent gain turned out to be evaluation noise.

How to use

Same interface as the gold-only model:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

description = "..."  # free-text vulnerability description
inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits)[0]

for i in probs.argsort(descending=True)[:5]:
    print(f"{model.config.id2label[int(i)]}  {probs[i]:.4f}")

Or side by side with the released model on a live CVE:

vulntrain-infer-attack-classification --cve CVE-2021-44077 \
    --model CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded

Intended uses & limitations

Intended: reproducing and extending the paper's expansion experiments — e.g. contrasting its per-technique behaviour with the gold-only checkpoint, or as a baseline for better silver-labeling strategies (higher-agreement labelers, agreement-weighted losses, human-in-the-loop curation).

Not intended: production use. It is strictly dominated by the gold-only model on ranking and rare-technique metrics, which is why Vulnerability-Lookup deploys the gold-only checkpoint. All limitations of the gold-only model (53-technique vocabulary, KEV-skewed data, English only, 512-token truncation, uncalibrated scores, unverified suggestions) apply here too.

Training and evaluation data

• Expert rows: the 972-row train split of CIRCL/vulnerability-attack-techniques (MITRE CTID gold mappings).
• LLM rows (train only): 984 CVEs from CIRCL/vulnerability-attack-techniques-llm-scaling, labeled by qwen3.5:122b (Ollama, assertive single-call prompt following the CTID methodology) — the best configuration of the paper's labeler benchmark, at ≈0.39 F1 agreement with held-out expert labels.
• The label vocabulary stays frozen to the gold train split, and the validation (106) and test (118) splits contain only expert-labeled rows; checkpoint selection uses the validation split.

Training procedure

Binary cross-entropy over 53 sigmoid outputs with balanced per-label pos_weight, trained with vulntrain-train-attack-classification (VulnTrain), like the gold-only model — only the training set differs (1,956 rows instead of 972).

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 40
• max_length: 512
• loss: BCEWithLogitsLoss, balanced pos_weight
• checkpoint selection: best macro-F1 on the validation split

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Micro	F1 Macro	Precision Micro	Recall Micro	Recall At 3	Recall At 5
0.7195	1.0	62	0.7652	0.2547	0.0278	0.1821	0.4232	0.2862	0.3870
0.6487	2.0	124	0.7303	0.2628	0.0431	0.1779	0.5021	0.2717	0.3564
0.6442	3.0	186	0.7133	0.2931	0.0528	0.2042	0.5187	0.3741	0.4821
0.6053	4.0	248	0.6903	0.3353	0.0752	0.2597	0.4730	0.3730	0.5167
0.6037	5.0	310	0.6715	0.3392	0.0886	0.2619	0.4813	0.4193	0.5561
0.5560	6.0	372	0.6557	0.3356	0.0970	0.2521	0.5021	0.4002	0.5483
0.5204	7.0	434	0.6496	0.3205	0.0969	0.2319	0.5187	0.3782	0.5023
0.5067	8.0	496	0.6369	0.3470	0.1098	0.2628	0.5104	0.4092	0.5781
0.5060	9.0	558	0.6256	0.3626	0.1195	0.2731	0.5394	0.4548	0.6115
0.4426	10.0	620	0.6207	0.3212	0.1076	0.238	0.4938	0.4304	0.5597
0.4414	11.0	682	0.6174	0.3840	0.1213	0.3049	0.5187	0.4700	0.6059
0.4453	12.0	744	0.6184	0.3163	0.1432	0.2217	0.5519	0.4156	0.5636
0.4328	13.0	806	0.6122	0.3351	0.1403	0.2447	0.5311	0.4441	0.5816
0.4219	14.0	868	0.6133	0.3773	0.1543	0.2866	0.5519	0.4642	0.6327
0.4109	15.0	930	0.6075	0.3722	0.1607	0.2720	0.5892	0.4682	0.6197
0.3905	16.0	992	0.6038	0.3778	0.1665	0.2771	0.5934	0.4642	0.6307
0.3948	17.0	1054	0.6025	0.3781	0.1448	0.2822	0.5726	0.4645	0.5977
0.3785	18.0	1116	0.6034	0.3845	0.1521	0.2939	0.5560	0.4529	0.6422
0.3723	19.0	1178	0.6038	0.3810	0.1467	0.2875	0.5643	0.4914	0.6543
0.3580	20.0	1240	0.6036	0.3790	0.1504	0.2842	0.5685	0.4524	0.6257
0.3343	21.0	1302	0.5993	0.3878	0.1522	0.2978	0.5560	0.5228	0.6688
0.3386	22.0	1364	0.6000	0.3994	0.1501	0.3103	0.5602	0.4819	0.6740
0.3444	23.0	1426	0.5977	0.3977	0.1586	0.3013	0.5851	0.4945	0.6787
0.3303	24.0	1488	0.6003	0.3988	0.1571	0.3072	0.5685	0.4862	0.6594
0.3206	25.0	1550	0.6044	0.4	0.1574	0.3124	0.5560	0.4792	0.6825
0.3180	26.0	1612	0.6080	0.4031	0.1515	0.3188	0.5477	0.5008	0.6744
0.3100	27.0	1674	0.6085	0.4037	0.1500	0.3211	0.5436	0.5197	0.6289
0.3075	28.0	1736	0.6061	0.4071	0.1622	0.3158	0.5726	0.4953	0.6656
0.3042	29.0	1798	0.6135	0.3982	0.1550	0.3155	0.5394	0.4961	0.6722
0.3044	30.0	1860	0.6097	0.4	0.1579	0.3178	0.5394	0.4874	0.6751
0.3032	31.0	1922	0.6028	0.3875	0.1588	0.2950	0.5643	0.4796	0.6509
0.2868	32.0	1984	0.6083	0.3994	0.1522	0.3157	0.5436	0.5063	0.6869
0.2821	33.0	2046	0.6093	0.4018	0.1535	0.3187	0.5436	0.4800	0.6727
0.2915	34.0	2108	0.6044	0.3982	0.1528	0.3115	0.5519	0.4972	0.6609
0.2829	35.0	2170	0.6112	0.3988	0.1545	0.3122	0.5519	0.4952	0.6869
0.2915	36.0	2232	0.6111	0.4062	0.1510	0.3227	0.5477	0.5079	0.6853
0.2781	37.0	2294	0.6153	0.4	0.1507	0.3208	0.5311	0.5020	0.6778
0.2724	38.0	2356	0.6115	0.4031	0.1509	0.3203	0.5436	0.4972	0.6778
0.2794	39.0	2418	0.6142	0.3975	0.1468	0.3176	0.5311	0.4984	0.6801
0.2661	40.0	2480	0.6123	0.4025	0.1516	0.3195	0.5436	0.4972	0.6825

Framework versions

• Transformers 5.13.0
• Pytorch 2.12.1+cu130
• Datasets 4.8.5
• Tokenizers 0.22.2

Related artifacts

Artifact	Location	DOI
Released model (use this one)	CIRCL/vulnerability-attack-technique-classification-roberta-base	10.57967/hf/9623
Gold dataset (1,207 CVEs, CTID-curated labels)	CIRCL/vulnerability-attack-techniques	10.57967/hf/9621
LLM expansion dataset (984 LLM-labeled CVEs)	CIRCL/vulnerability-attack-techniques-llm-scaling	10.57967/hf/9622
Code	vulnerability-lookup/VulnTrain	—
Paper + trainer logs	vulnerability-lookup/cve-attack-mapping-paper	—

Citation

@misc{bonhomme2026cveattack,
  title  = {Mapping CVEs to MITRE ATT\&CK Techniques: A Curated Gold-Set
            Classifier and the Limits of LLM-Assisted Label Expansion},
  author = {Bonhomme, C{\'e}dric},
  year   = {2026},
  note   = {Preprint},
}

Acknowledgements

Developed at CIRCL in the context of the AIPITCH project, co-funded by the European Union.