GIST-small-format

A fine-tuned version of the bert architecture (BertForSequenceClassification) optimized for the text-classification task.

• Model type: bert
• Problem Type: single_label_classification
• Number of Labels: 24
• Vocabulary Size: 30522
• License: MIT

Use

To get started with this model in Python using the Hugging Face Transformers library, run the following code:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "agentlans/GIST-small-format"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "Replace this with your input text."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

predicted_class_id = logits.argmax().item()
predicted_class_name = model.config.id2label[predicted_class_id]

print(f"Predicted Class ID: {predicted_class_id}")
print(f"Predicted Class Name: {predicted_class_name}")

Intended Uses & Limitations

Intended Use

This model is designed for sequence classification tasks. Below are the specific class labels mapped to their corresponding IDs:

Label ID	Label Name
0	Academic Writing
1	Content Listing
2	Creative Writing
3	Customer Support Page
4	Discussion Forum / Comment Section
5	FAQs
6	Incomplete Content
7	Knowledge Article
8	Legal Notices
9	Listicle
10	News Article
11	Nonfiction Writing
12	Organizational About Page
13	Organizational Announcement
14	Personal About Page
15	Personal Blog
16	Product Page
17	Q&A Forum
18	Spam / Ads
19	Structured Data
20	Technical Writing
21	Transcript / Interview
22	Tutorial / How-To Guide
23	User Reviews

Training Details

Hyperparameters

The following hyperparameters were used during fine-tuning:

• Learning Rate: 5e-05
• Train Batch Size: 8
• Eval Batch Size: 8
• Optimizer: OptimizerNames.ADAMW_TORCH_FUSED
• Number of Epochs: 3.0
• Mixed Precision: BF16

Show Advanced Training Configuration

Optimization & Regularization

• Gradient Accumulation Steps: 1
• Learning Rate Scheduler: SchedulerType.LINEAR
• Warmup Steps: 0
• Warmup Ratio: None
• Weight Decay: 0.0
• Max Gradient Norm: 1.0

Hardware & Reproducibility

• Number of GPUs: 1
• Seed: 42

Training Results & Evaluation

During fine-tuning, the model achieved the following results on the evaluation set:

Metric	Value
Train Loss	0.3033
Validation Loss	0.3809
Validation F1 Score	0.8518
Total FLOPs	1.8015e+16

For performance on the test set, click here.

Speed Performance

• Training Runtime: 2259.7352 seconds
• Train Samples per Second: 483.897
• Evaluation Runtime: 21.817 seconds
• Eval Samples per Second: 1856.355

Show Detailed Training Logs

Training Logs History

Step	Epoch	Learning Rate	Training Loss	Validation Loss	Validation F1
500	0.011	4.9817e-05	1.7311	N/A	N/A
1000	0.022	4.9635e-05	1.0143	N/A	N/A
1500	0.033	4.9452e-05	0.8819	N/A	N/A
2000	0.044	4.9269e-05	0.6946	N/A	N/A
2500	0.055	4.9086e-05	0.6733	N/A	N/A
3000	0.066	4.8903e-05	0.6471	N/A	N/A
3500	0.077	4.8720e-05	0.5822	N/A	N/A
4000	0.088	4.8537e-05	0.5588	N/A	N/A
4500	0.099	4.8354e-05	0.5702	N/A	N/A
5000	0.11	4.8171e-05	0.5574	N/A	N/A
5500	0.121	4.7988e-05	0.5319	N/A	N/A
6000	0.132	4.7806e-05	0.5464	N/A	N/A
6500	0.143	4.7623e-05	0.5369	N/A	N/A
7000	0.154	4.7440e-05	0.4969	N/A	N/A
7500	0.165	4.7257e-05	0.5199	N/A	N/A
8000	0.176	4.7074e-05	0.4873	N/A	N/A
8500	0.187	4.6891e-05	0.4851	N/A	N/A
9000	0.198	4.6708e-05	0.4956	N/A	N/A
9500	0.209	4.6525e-05	0.4595	N/A	N/A
10000	0.219	4.6342e-05	0.4893	N/A	N/A
10500	0.23	4.6159e-05	0.4713	N/A	N/A
11000	0.241	4.5977e-05	0.4707	N/A	N/A
11500	0.252	4.5794e-05	0.4377	N/A	N/A
12000	0.263	4.5611e-05	0.4906	N/A	N/A
12500	0.274	4.5428e-05	0.4677	N/A	N/A
13000	0.285	4.5245e-05	0.472	N/A	N/A
13500	0.296	4.5062e-05	0.4726	N/A	N/A
14000	0.307	4.4879e-05	0.4702	N/A	N/A
14500	0.318	4.4696e-05	0.4567	N/A	N/A
15000	0.329	4.4513e-05	0.4336	N/A	N/A
15500	0.34	4.4330e-05	0.4196	N/A	N/A
16000	0.351	4.4148e-05	0.478	N/A	N/A
16500	0.362	4.3965e-05	0.4753	N/A	N/A
17000	0.373	4.3782e-05	0.4525	N/A	N/A
17500	0.384	4.3599e-05	0.428	N/A	N/A
18000	0.395	4.3416e-05	0.4129	N/A	N/A
18500	0.406	4.3233e-05	0.4529	N/A	N/A
19000	0.417	4.3050e-05	0.4624	N/A	N/A
19500	0.428	4.2867e-05	0.4271	N/A	N/A
20000	0.439	4.2684e-05	0.415	N/A	N/A
20500	0.45	4.2501e-05	0.4233	N/A	N/A
21000	0.461	4.2319e-05	0.4353	N/A	N/A
21500	0.472	4.2136e-05	0.4334	N/A	N/A
22000	0.483	4.1953e-05	0.4031	N/A	N/A
22500	0.494	4.1770e-05	0.4011	N/A	N/A
23000	0.505	4.1587e-05	0.4137	N/A	N/A
23500	0.516	4.1404e-05	0.4478	N/A	N/A
24000	0.527	4.1221e-05	0.4157	N/A	N/A
24500	0.538	4.1038e-05	0.4276	N/A	N/A
25000	0.549	4.0855e-05	0.4124	N/A	N/A
25500	0.56	4.0672e-05	0.4065	N/A	N/A
26000	0.571	4.0490e-05	0.4333	N/A	N/A
26500	0.582	4.0307e-05	0.4098	N/A	N/A
27000	0.593	4.0124e-05	0.4091	N/A	N/A
27500	0.604	3.9941e-05	0.4346	N/A	N/A
28000	0.615	3.9758e-05	0.4063	N/A	N/A
28500	0.626	3.9575e-05	0.4149	N/A	N/A
29000	0.636	3.9392e-05	0.4145	N/A	N/A
29500	0.647	3.9209e-05	0.3984	N/A	N/A
30000	0.658	3.9026e-05	0.4113	N/A	N/A
30500	0.669	3.8843e-05	0.4131	N/A	N/A
31000	0.68	3.8661e-05	0.4114	N/A	N/A
31500	0.691	3.8478e-05	0.3915	N/A	N/A
32000	0.702	3.8295e-05	0.4179	N/A	N/A
32500	0.713	3.8112e-05	0.3832	N/A	N/A
33000	0.724	3.7929e-05	0.3826	N/A	N/A
33500	0.735	3.7746e-05	0.3716	N/A	N/A
34000	0.746	3.7563e-05	0.3827	N/A	N/A
34500	0.757	3.7380e-05	0.389	N/A	N/A
35000	0.768	3.7197e-05	0.4051	N/A	N/A
35500	0.779	3.7014e-05	0.401	N/A	N/A
36000	0.79	3.6831e-05	0.3962	N/A	N/A
36500	0.801	3.6649e-05	0.3653	N/A	N/A
37000	0.812	3.6466e-05	0.4085	N/A	N/A
37500	0.823	3.6283e-05	0.3887	N/A	N/A
38000	0.834	3.6100e-05	0.4255	N/A	N/A
38500	0.845	3.5917e-05	0.3765	N/A	N/A
39000	0.856	3.5734e-05	0.3964	N/A	N/A
39500	0.867	3.5551e-05	0.3849	N/A	N/A
40000	0.878	3.5368e-05	0.3928	N/A	N/A
40500	0.889	3.5185e-05	0.3885	N/A	N/A
41000	0.9	3.5002e-05	0.3638	N/A	N/A
41500	0.911	3.4820e-05	0.3788	N/A	N/A
42000	0.922	3.4637e-05	0.3845	N/A	N/A
42500	0.933	3.4454e-05	0.397	N/A	N/A
43000	0.944	3.4271e-05	0.3914	N/A	N/A
43500	0.955	3.4088e-05	0.3809	N/A	N/A
44000	0.966	3.3905e-05	0.3856	N/A	N/A
44500	0.977	3.3722e-05	0.3618	N/A	N/A
45000	0.988	3.3539e-05	0.3732	N/A	N/A
45500	0.999	3.3356e-05	0.3595	N/A	N/A
45562	1.0	N/A	N/A	0.3376	0.7906
46000	1.01	3.3173e-05	0.2933	N/A	N/A
46500	1.021	3.2991e-05	0.2831	N/A	N/A
47000	1.032	3.2808e-05	0.2768	N/A	N/A
47500	1.043	3.2625e-05	0.2947	N/A	N/A
48000	1.054	3.2442e-05	0.3016	N/A	N/A
48500	1.064	3.2259e-05	0.2888	N/A	N/A
49000	1.075	3.2076e-05	0.2698	N/A	N/A
49500	1.086	3.1893e-05	0.3016	N/A	N/A
50000	1.097	3.1710e-05	0.2799	N/A	N/A
50500	1.108	3.1527e-05	0.3036	N/A	N/A
51000	1.119	3.1344e-05	0.2973	N/A	N/A
51500	1.13	3.1162e-05	0.2932	N/A	N/A
52000	1.141	3.0979e-05	0.2789	N/A	N/A
52500	1.152	3.0796e-05	0.318	N/A	N/A
53000	1.163	3.0613e-05	0.2868	N/A	N/A
53500	1.174	3.0430e-05	0.3062	N/A	N/A
54000	1.185	3.0247e-05	0.3081	N/A	N/A
54500	1.196	3.0064e-05	0.2754	N/A	N/A
55000	1.207	2.9881e-05	0.3054	N/A	N/A
55500	1.218	2.9698e-05	0.2972	N/A	N/A
56000	1.229	2.9515e-05	0.3048	N/A	N/A
56500	1.24	2.9333e-05	0.2779	N/A	N/A
57000	1.251	2.9150e-05	0.2782	N/A	N/A
57500	1.262	2.8967e-05	0.3115	N/A	N/A
58000	1.273	2.8784e-05	0.2761	N/A	N/A
58500	1.284	2.8601e-05	0.287	N/A	N/A
59000	1.295	2.8418e-05	0.2967	N/A	N/A
59500	1.306	2.8235e-05	0.2844	N/A	N/A
60000	1.317	2.8052e-05	0.2875	N/A	N/A
60500	1.328	2.7869e-05	0.2998	N/A	N/A
61000	1.339	2.7686e-05	0.2731	N/A	N/A
61500	1.35	2.7504e-05	0.2941	N/A	N/A
62000	1.361	2.7321e-05	0.2899	N/A	N/A
62500	1.372	2.7138e-05	0.305	N/A	N/A
63000	1.383	2.6955e-05	0.295	N/A	N/A
63500	1.394	2.6772e-05	0.2861	N/A	N/A
64000	1.405	2.6589e-05	0.2653	N/A	N/A
64500	1.416	2.6406e-05	0.2749	N/A	N/A
65000	1.427	2.6223e-05	0.2906	N/A	N/A
65500	1.438	2.6040e-05	0.2712	N/A	N/A
66000	1.449	2.5857e-05	0.3095	N/A	N/A
66500	1.46	2.5675e-05	0.3181	N/A	N/A
67000	1.471	2.5492e-05	0.2544	N/A	N/A
67500	1.481	2.5309e-05	0.2504	N/A	N/A
68000	1.492	2.5126e-05	0.3013	N/A	N/A
68500	1.503	2.4943e-05	0.2951	N/A	N/A
69000	1.514	2.4760e-05	0.2822	N/A	N/A
69500	1.525	2.4577e-05	0.2482	N/A	N/A
70000	1.536	2.4394e-05	0.2707	N/A	N/A
70500	1.547	2.4211e-05	0.2601	N/A	N/A
71000	1.558	2.4028e-05	0.3073	N/A	N/A
71500	1.569	2.3846e-05	0.2815	N/A	N/A
72000	1.58	2.3663e-05	0.2587	N/A	N/A
72500	1.591	2.3480e-05	0.2825	N/A	N/A
73000	1.602	2.3297e-05	0.2921	N/A	N/A
73500	1.613	2.3114e-05	0.2775	N/A	N/A
74000	1.624	2.2931e-05	0.2611	N/A	N/A
74500	1.635	2.2748e-05	0.2656	N/A	N/A
75000	1.646	2.2565e-05	0.2757	N/A	N/A
75500	1.657	2.2382e-05	0.2574	N/A	N/A
76000	1.668	2.2199e-05	0.2874	N/A	N/A
76500	1.679	2.2017e-05	0.2507	N/A	N/A
77000	1.69	2.1834e-05	0.2621	N/A	N/A
77500	1.701	2.1651e-05	0.2704	N/A	N/A
78000	1.712	2.1468e-05	0.2748	N/A	N/A
78500	1.723	2.1285e-05	0.2705	N/A	N/A
79000	1.734	2.1102e-05	0.3062	N/A	N/A
79500	1.745	2.0919e-05	0.298	N/A	N/A
80000	1.756	2.0736e-05	0.2788	N/A	N/A
80500	1.767	2.0553e-05	0.2542	N/A	N/A
81000	1.778	2.0370e-05	0.2742	N/A	N/A
81500	1.789	2.0188e-05	0.2559	N/A	N/A
82000	1.8	2.0005e-05	0.2805	N/A	N/A
82500	1.811	1.9822e-05	0.2572	N/A	N/A
83000	1.822	1.9639e-05	0.2701	N/A	N/A
83500	1.833	1.9456e-05	0.2636	N/A	N/A
84000	1.844	1.9273e-05	0.2724	N/A	N/A
84500	1.855	1.9090e-05	0.2652	N/A	N/A
85000	1.866	1.8907e-05	0.2604	N/A	N/A
85500	1.877	1.8724e-05	0.262	N/A	N/A
86000	1.888	1.8541e-05	0.2594	N/A	N/A
86500	1.899	1.8359e-05	0.2609	N/A	N/A
87000	1.909	1.8176e-05	0.2769	N/A	N/A
87500	1.92	1.7993e-05	0.2803	N/A	N/A
88000	1.931	1.7810e-05	0.2635	N/A	N/A
88500	1.942	1.7627e-05	0.2759	N/A	N/A
89000	1.953	1.7444e-05	0.2669	N/A	N/A
89500	1.964	1.7261e-05	0.2887	N/A	N/A
90000	1.975	1.7078e-05	0.2838	N/A	N/A
90500	1.986	1.6895e-05	0.2482	N/A	N/A
91000	1.997	1.6712e-05	0.2344	N/A	N/A
91124	2.0	N/A	N/A	0.3564	0.8288
91500	2.008	1.6529e-05	0.1882	N/A	N/A
92000	2.019	1.6347e-05	0.1598	N/A	N/A
92500	2.03	1.6164e-05	0.1572	N/A	N/A
93000	2.041	1.5981e-05	0.1645	N/A	N/A
93500	2.052	1.5798e-05	0.1709	N/A	N/A
94000	2.063	1.5615e-05	0.1649	N/A	N/A
94500	2.074	1.5432e-05	0.1513	N/A	N/A
95000	2.085	1.5249e-05	0.1468	N/A	N/A
95500	2.096	1.5066e-05	0.2013	N/A	N/A
96000	2.107	1.4883e-05	0.1819	N/A	N/A
96500	2.118	1.4700e-05	0.1817	N/A	N/A
97000	2.129	1.4518e-05	0.1869	N/A	N/A
97500	2.14	1.4335e-05	0.164	N/A	N/A
98000	2.151	1.4152e-05	0.1722	N/A	N/A
98500	2.162	1.3969e-05	0.1798	N/A	N/A
99000	2.173	1.3786e-05	0.1633	N/A	N/A
99500	2.184	1.3603e-05	0.1812	N/A	N/A
100000	2.195	1.3420e-05	0.1719	N/A	N/A
100500	2.206	1.3237e-05	0.1556	N/A	N/A
101000	2.217	1.3054e-05	0.2062	N/A	N/A
101500	2.228	1.2871e-05	0.1641	N/A	N/A
102000	2.239	1.2689e-05	0.1656	N/A	N/A
102500	2.25	1.2506e-05	0.1639	N/A	N/A
103000	2.261	1.2323e-05	0.173	N/A	N/A
103500	2.272	1.2140e-05	0.1895	N/A	N/A
104000	2.283	1.1957e-05	0.1707	N/A	N/A
104500	2.294	1.1774e-05	0.1851	N/A	N/A
105000	2.305	1.1591e-05	0.1782	N/A	N/A
105500	2.316	1.1408e-05	0.1703	N/A	N/A
106000	2.327	1.1225e-05	0.1581	N/A	N/A
106500	2.337	1.1042e-05	0.1648	N/A	N/A
107000	2.348	1.0860e-05	0.1986	N/A	N/A
107500	2.359	1.0677e-05	0.1799	N/A	N/A
108000	2.37	1.0494e-05	0.1611	N/A	N/A
108500	2.381	1.0311e-05	0.1644	N/A	N/A
109000	2.392	1.0128e-05	0.1457	N/A	N/A
109500	2.403	9.9451e-06	0.1498	N/A	N/A
110000	2.414	9.7622e-06	0.1812	N/A	N/A
110500	2.425	9.5793e-06	0.1837	N/A	N/A
111000	2.436	9.3964e-06	0.167	N/A	N/A
111500	2.447	9.2135e-06	0.1736	N/A	N/A
112000	2.458	9.0306e-06	0.1638	N/A	N/A
112500	2.469	8.8477e-06	0.1659	N/A	N/A
113000	2.48	8.6647e-06	0.1822	N/A	N/A
113500	2.491	8.4818e-06	0.1583	N/A	N/A
114000	2.502	8.2989e-06	0.1486	N/A	N/A
114500	2.513	8.1160e-06	0.1723	N/A	N/A
115000	2.524	7.9331e-06	0.1505	N/A	N/A
115500	2.535	7.7502e-06	0.1588	N/A	N/A
116000	2.546	7.5673e-06	0.1401	N/A	N/A
116500	2.557	7.3844e-06	0.1572	N/A	N/A
117000	2.568	7.2015e-06	0.1915	N/A	N/A
117500	2.579	7.0186e-06	0.1555	N/A	N/A
118000	2.59	6.8357e-06	0.1793	N/A	N/A
118500	2.601	6.6528e-06	0.1671	N/A	N/A
119000	2.612	6.4699e-06	0.1405	N/A	N/A
119500	2.623	6.2870e-06	0.1504	N/A	N/A
120000	2.634	6.1041e-06	0.1683	N/A	N/A
120500	2.645	5.9212e-06	0.1562	N/A	N/A
121000	2.656	5.7383e-06	0.1648	N/A	N/A
121500	2.667	5.5554e-06	0.1531	N/A	N/A
122000	2.678	5.3725e-06	0.1582	N/A	N/A
122500	2.689	5.1896e-06	0.1415	N/A	N/A
123000	2.7	5.0067e-06	0.1391	N/A	N/A
123500	2.711	4.8238e-06	0.1349	N/A	N/A
124000	2.722	4.6409e-06	0.1772	N/A	N/A
124500	2.733	4.4580e-06	0.1687	N/A	N/A
125000	2.744	4.2751e-06	0.1536	N/A	N/A
125500	2.754	4.0922e-06	0.1394	N/A	N/A
126000	2.765	3.9093e-06	0.1729	N/A	N/A
126500	2.776	3.7264e-06	0.1582	N/A	N/A
127000	2.787	3.5435e-06	0.176	N/A	N/A
127500	2.798	3.3606e-06	0.1595	N/A	N/A
128000	2.809	3.1777e-06	0.1665	N/A	N/A
128500	2.82	2.9948e-06	0.1628	N/A	N/A
129000	2.831	2.8119e-06	0.1426	N/A	N/A
129500	2.842	2.6290e-06	0.164	N/A	N/A
130000	2.853	2.4461e-06	0.1562	N/A	N/A
130500	2.864	2.2632e-06	0.1632	N/A	N/A
131000	2.875	2.0803e-06	0.1562	N/A	N/A
131500	2.886	1.8974e-06	0.1538	N/A	N/A
132000	2.897	1.7145e-06	0.1649	N/A	N/A
132500	2.908	1.5316e-06	0.1576	N/A	N/A
133000	2.919	1.3487e-06	0.1481	N/A	N/A
133500	2.93	1.1658e-06	0.1535	N/A	N/A
134000	2.941	9.8291e-07	0.1552	N/A	N/A
134500	2.952	8.0001e-07	0.1542	N/A	N/A
135000	2.963	6.1711e-07	0.148	N/A	N/A
135500	2.974	4.3421e-07	0.1502	N/A	N/A
136000	2.985	2.5131e-07	0.1423	N/A	N/A
136500	2.996	6.8405e-08	0.152	N/A	N/A
136686	3.0	N/A	N/A	0.3809	0.8518

Framework Versions

• Transformers: 5.14.0.dev0
• PyTorch: 2.13.0+cu130