Genrius AI

Generating Real Intelligence. We build AI models and make them small and fast enough to run on hardware people already own.

New: Qwen3.8-27B-Pingala8-2bit - Qwen3.8-27B in 8.31 GB, running fully on one 12 GB laptop GPU at 37.7 tokens/s. WikiText-2 perplexity 6.89 vs 7.88 for llama.cpp IQ2_XXS (8.43 GB); DASH-Q IQ2_XS is better (6.54, 8.55 GB) and unsloth UD-IQ2_S (7.34, 8.37 GB) is not significantly different. Needs the patched Pingala-8 llama.cpp runtime (CUDA), built from source.

Try it: Will it fit your GPU? (calculator) | One-click build, Windows and Linux, NVIDIA RTX 30/40/50 |

Results across 8 models

77-85% smaller than the 16-bit weights, with 13-70% lower perplexity than llama.cpp IQ2_XXS at the same size. Token generation matches or beats IQ2_XXS on 5 of 6 models measured (Qwen3-0.6B is 30% slower); Qwen3-4B generates 39% faster than 4-bit Q4_K_M at half its size; Qwen3.8-27B runs on one 12 GB GPU. Prompt processing is slower (0.74-0.89x).

Lower is better. The files need our patched llama.cpp build (NVIDIA CUDA); stock llama.cpp, Ollama and LM Studio cannot load them. Qwen3-1.7B and Qwen3-0.6B have no one-click launcher yet; run them with the v2.5 runtime's llama-server. The other five run in the v2.5 one-click build (Windows and Linux); Qwen3.5-2B and Qwen3.5-0.8B need v2.5 or later. On Qwen3-1.7B, ik_llama.cpp's IQ2_KT is better than Pingala-8 (20.89 vs 28.23, in a 3.3% larger file). On Qwen3-0.6B, IQ2_XXS collapses and Pingala-8 generates tokens at 0.70x the speed of IQ2_XXS.

Pingala-8: 2-bit LLMs for llama.cpp, no retraining

• 85% smaller: Qwen3-4B goes from 8.05 GB to 1.24 GB, and Llama 3.1 8B from 16.08 GB to 2.39 GB.
• At the file size of llama.cpp's IQ2_XXS, our Qwen3-4B reaches WikiText-2 perplexity 16.64. IQ2_XXS reaches 28.61 and the best IQ2_KT file we built reaches 18.15 (lower is better).
• Where we lose: Qwen3-14B (IQ2_KT 9.40, ours 9.57), Qwen3-1.7B (IQ2_KT 20.89 in a 3.3% larger file, ours 28.23), prompt processing (0.74x of IQ2_XXS), and retrained models such as Ternary Bonsai 4B.

Links

• VRAM calculator: normal vs Pingala-8
• One-click build (Windows and Linux, launchers for five models): v2.5
• Code
• Paper
• Qwen3-4B-Pingala8-2bit, Llama-3.1-8B-Pingala8-2bit and Qwen3.5-4B-Pingala8-2bit (text only); Qwen3.5-2B-Pingala8-2bit and Qwen3.5-0.8B-Pingala8-2bit (text only; use the v2.5 one-click build); Qwen3-1.7B-Pingala8-2bit (IQ2_KT is better on this model) and Qwen3-0.6B-Pingala8-2bit (generates at 0.70x IQ2_XXS speed); no one-click launcher yet, run with the v2.5 runtime's llama-server
• Benchmark tables