solintellegence/Sol-Lite-Base

Sol Lite Base

Lite Base reuses four transformer blocks on a second pass. That gives it fourteen block applications while storing ten blocks. We trained the 14,995,843-parameter model from scratch on 8,153,333,760 tokens.

The released weights are for text completion. There was no instruction tuning or assistant safety tuning. The repository includes a PyTorch loader and generation helper.

Architecture

Setting	Value
Parameters	14,995,843
Training tokens	8,153,333,760
Context	2,048 tokens
Tokenizer	4,096-entry digit-aware byte-level BPE
Hidden width	256
Stored blocks / effective applications	10 / 14
Recurrent layout	1 prelude, 4 middle blocks used twice, 5 coda blocks
Attention	8 query heads, 2 KV heads, head dimension 32
Q/K normalization	Per-head RMSNorm with RoPE
Recurrence	Learned pass embeddings and channel-wise refresh gates
FFN	Gated SiLU, width 1,465
Local memory	2,048-entry bigram/trigram EngramLite
Token embeddings	Tied to the output head
Weights	FP32 safetensors

Grouped-query attention and loop conditioning support the repeated pass. XSA subtracts attention values, and EngramLite supplies local memory. Learned pass embeddings and channel-wise refresh gates distinguish the two uses of the middle blocks.

Generate text

Install the dependencies, then download the repository and import its standalone implementation:

pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub

from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("solintellegence/Sol-Lite-Base")
sys.path.insert(0, model_dir)

from modeling_sol_lite import load_model, generate

model, tokenizer = load_model(model_dir, device="cpu")

text = generate(
    model,
    tokenizer,
    "The future of efficient language models is",
    max_new_tokens=64,
)

print(text)

Benchmark results

Benchmark	Examples	Normalized accuracy
HellaSwag	10,042	27.72%
ARC-Easy	2,376	34.72%
ARC-Challenge	1,172	22.87%
PIQA	1,838	57.73%
ArithMark-3	1,000	34.20%

The local Intelligence Index score is 8.799. We ran the complete zero-shot language-model splits with lm-eval 0.4.12. ArithMark-3 used independent tokenization and normalized accuracy. Raw evaluation outputs are saved in evals/.

Training sources

The English curriculum combined FineWeb-Edu, DCLM, UltraFineWeb levels 1-3, FineWeb-HQ, FineMath, and FinePhrase. We prepared the stream beforehand and kept the tokenizer fixed. Public benchmark examples weren't used to choose the checkpoint.

Files

The checkpoint is model.safetensors. Use modeling_sol_lite.py for SolForCausalLM, loading, and generation; config.json supplies the architecture settings. tokenizer.json and tokenizer_config.json are the matching tokenizer files. Run provenance is in training_state.json.

Intended use and license

Lite Base is for experiments with small language models, recurrence, memory, and distillation. Generated text can repeat itself, lose coherence, or give wrong answers. Its training doesn't cover instruction following.

The model is licensed under CC BY 4.0. Upstream datasets retain their own terms.