Post
74
A 0.5B-active base trained on 60B tokens, rivals Qwen3.5-2B.
2.59B total and 553M active parameters. 12-task lm-eval average 58.3, vs Qwen3.5 2B 58.1, trained on only ~60B tokens.
Architecture: 23 GDN-2 + 9 MLA layers, mHC with 4 residual streams, Engram bigram/trigram hash tables before blocks 1 and 14, 120 latent experts per layer with top-6 routing + 2 shared experts, quantile load balancing. No positional encoding. Olmo 3 tokenizer.
Loads with transformers (trust_remote_code) or via
Known gaps: code is the weakest domain, 16K context, no scaling-law sweep yet.
Model: ArsalanZ/Nenya-2.5B-A0.5B-Base
Detailed technical report: https://www.arsalanzafar.ai/nenya-technical-report
I'd love some feedback!
2.59B total and 553M active parameters. 12-task lm-eval average 58.3, vs Qwen3.5 2B 58.1, trained on only ~60B tokens.
Architecture: 23 GDN-2 + 9 MLA layers, mHC with 4 residual streams, Engram bigram/trigram hash tables before blocks 1 and 14, 120 latent experts per layer with top-6 routing + 2 shared experts, quantile load balancing. No positional encoding. Olmo 3 tokenizer.
Loads with transformers (trust_remote_code) or via
pip install nenya without remote code. Mac users: pip install "nenya[mac]" gives a Metal engine at 127 tok/s on M4 and 74 tok/s on M1, ~4 GB memory. An 8-bit Metal variant is in the repo.Known gaps: code is the weakest domain, 16K context, no scaling-law sweep yet.
Model: ArsalanZ/Nenya-2.5B-A0.5B-Base
Detailed technical report: https://www.arsalanzafar.ai/nenya-technical-report
I'd love some feedback!