"HET1" GPT2 (base) and GPT2MoE
Collection
Trained on est_latn, arb_arab, ell_grek, tur_latn, and eng_latn (heterogeneous set 1). โข 6 items โข Updated
GPT2 (base) trained on a set of languages where English dominates the model and tokenizer training data sets English is dominant in the model training set (3000mb of English and 500mb (both with byte premium scaling) for all other languages) English is dominant in the tokenizer training set (300mb of English and 50mb (both with byte premium scaling) for all other languages)
Based on 1 W&B runs named gptbase_dom_het_tok_1000mb in sjzahra-ri-se/huggingface.
Loss: 1 run(s); warmup: 1; GPU telemetry: 1 run(s).
Training hardware: Trained on a single H100 GPU.
Greek, Turkish, Estonian, English, Arabic
| Parameter | Value |
|---|---|
| Runs included | 1 |
| Training hardware | Trained on a single H100 GPU |
| Total epochs | 9.99975 |
| Total training steps | 360,651 |
| Total training FLOPs | 6.0309e+18 |
| Learning rate | 0.0001 |
| Train batch size | 64 |
| Eval batch size | 64 |
| Gradient accumulation | 1 |
| Weight decay | 0.01 |
| Warmup steps | 36065 |
| Warmup ratio | 0 |
| Optimizer | adamw_torch_fused |
| Scheduler | linear |
| Max sequence length | 20 |
| Precision | False |
| Seed | 43 |
| GPU utilization | mean 97.0%, peak 100.0% |
W&B runs: dy9jdmqz
Generated 2026-11-17T17:24:04+00:00.