Team-PIXEL/mpixel-base

MPIXEL

MPIXEL, introduced in Scaling Multilingual Pretraining of Pixel Language Models, is a family of multilingual pixel language models, which operate on visually rendered text. Our main models, mpixel-base (86M parameters) and mpixel-large (307M parameters), are pretrained on sixteen languages that span ten writing systems. MPIXEL uses adaptive rendering: it renders text as grapheme bigrams by default, Chinese, Japanese and Korean text one character per patch, and Arabic and Hebrew text word by word.

Basic usage

To use our models, please first install our mpixel package. Here is an example of loading a pretrained model and encoding input text:

from transformers import AutoModel
import mpixel.models.auto_register  # registers MPIXEL with the Auto classes
from mpixel.processing import MPIXELProcessor

processor = MPIXELProcessor.from_pretrained("Team-PIXEL/mpixel-base")
model = AutoModel.from_pretrained("Team-PIXEL/mpixel-base")

inputs = processor(["Hello world!", "The quick brown fox jumps over the lazy dog."])
outputs = model(**inputs)

print(outputs.last_hidden_state.shape)  # torch.Size([2, 513, 768])

For further information, please take a look at our repository: https://github.com/ilkerkesen/mpixel.

Citation

If you find MPIXEL beneficial for your research, please cite it,

@misc{kesen2026scalingmultilingualpretrainingpixel,
      title={Scaling Multilingual Pretraining of Pixel Language Models}, 
      author={Ilker Kesen and Jenny Kunz and Kasra Malihi and Desmond Elliott},
      year={2026},
      eprint={2610.29681},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.29681}, 
}