My series of fully open, state-of-the-art small mixture-of-experts models.
🔄 In a Training Loop
aquilesfd
aquilesfd
AI & ML interests
research
Recent Activity
reacted to DedeProGames's post with 🔥 about 2 hours ago
Meet Supra2.5-Ultra-Preview-001. 230M parameters, ~40.4B training tokens.
SupraLabs has released the first preview of its Supra2.5-Ultra family: a compact, Llama-style base model for text completion and small-model experimentation.
What's inside?
• 230M parameters
• 32 transformer layers
• 2,048-token context window
• Grouped Query Attention: 12 attention heads, 4 KV heads
• 49,152-token vocabulary
• SwiGLU activation, RoPE and tied embeddings
• BF16 weights and Hugging Face Transformers support
• Apache 2.0 license
Training
The model was trained on approximately 40.4B tokens from FineWeb-Edu, DCLM-Baseline and FineMath using a single A100 SXM.
Reported zero-shot results
• HellaSwag: 37.18% normalized accuracy
• PIQA: 65.78% accuracy
• ARC-Easy: 57.11% accuracy
• ARC-Challenge: 26.28% normalized accuracy
Want to experiment?
Code on REPO.
Model, weights and usage example:
SupraLabs/Supra2.5-Ultra-Preview-001 liked a model about 10 hours ago
SupraLabs/Supra2.1-Ultra-Preview-001 liked a model about 10 hours ago
JetBrains/Mellum2.1-12B-A2.5B-ThinkingOrganizations
models 8
aquilesfd/neo-3-1B-A90M-Instruct
Text Generation • Updated • 12
aquilesfd/neo-3-3B-A400M-Thinking
Text Generation • Updated • 7
aquilesfd/neo-3-1B-A90M-Base
Text Generation • 1.0B • Updated • 281 • 8
aquilesfd/neo-3-3B-A400M-Base
Text Generation • 3B • Updated • 224 • 8
aquilesfd/neo-2-345M-C2
Text Generation • 0.4B • Updated • 24 • 1
aquilesfd/neo-2-345M-C1
Text Generation • 0.4B • Updated • 24 • 1
aquilesfd/neo-64M-C1
Text Generation • 64.1M • Updated • 26
aquilesfd/VLM-MoE-3x124M-Experiment
0.4B • Updated • 17