SmolVLM 500M Instruct for ExecuTorch WebGPU

This repository contains an ExecuTorch conversion of HuggingFaceTB/SmolVLM-500M-Instruct for the ExecuTorch WebGPU backend, so the model runs entirely in a web browser on the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground.

Base model: HuggingFaceTB/SmolVLM-500M-Instruct
Format: ExecuTorch .pte programs delegated to the WebGPU backend
Quantization: 8-bit linears in the vision encoder and decoder (int8 static activations in two passes, int8 per-channel GPTQ weights); int8 token embeddings
Total size: 497.7 MB

About the original model

A compact vision-language model that answers questions about images, describes them and transcribes text in them. It pairs a 93M SigLIP vision encoder with the SmolLM2-360M language model.

Developed by: Hugging Face
Paper: SmolVLM: Redefining small and efficient multimodal models
507M parameters
English
Based on the Idefics3 architecture

For training data, evaluation results and full details, see the original model card: HuggingFaceTB/SmolVLM-500M-Instruct.

Intended use and limitations

Image description, visual question answering and reading text in images.

Not for high-stakes decisions about people (for example identity, employment or credit). It can describe images inaccurately or invent details; see the misuse and out-of-scope section of the upstream card.

What changed in this conversion

All linear layers of the vision encoder and decoder use 8-bit weights (per-channel, GPTQ) and 8-bit activations with one static scale per layer, applied in two passes (the second pass quantizes the first pass's rounding error)
Token embeddings quantized to int8
The key/value of the first prompt token (<|im_start|>) are built into the decoder, so prompts must start with it
Context capped at 512 tokens; quantization changes outputs slightly compared with the original model

Inputs and outputs

smolvlm_vision_q8.pte: pixels float32 [1, 3, 512, 512] (scaled as (x/255 - 0.5) / 0.5) -> image_embeds float32 [1, 64, 960]
smolvlm_decoder_q8.pte: forward(embeds float32 [1, 4, 960], input_pos int64 [1]) -> logits float32 [1, 49280] for the last of the 4 rows; feed windows of 4 consecutive positions, the KV cache lives inside the program
Text embeddings come from embed_tokens.i8.bin with embed_tokens.scale.f32.bin; tokenize with the files in tokenizer/; vlm_meta.json lists the special tokens and sizes

Files

File	Size	SHA-256
`embed_tokens.i8.bin`	45.1 MB	`d3ea47247be0b070…`
`embed_tokens.scale.f32.bin`	192.5 KB	`adefdadaae81d572…`
`smolvlm_decoder_q8.pte`	349.9 MB	`a54d81fbafd289c3…`
`smolvlm_vision_q8.pte`	99.0 MB	`a48d6919aa2067be…`
`tokenizer/special_tokens_map.json`	1.0 KB	`aa0ff906077086df…`
`tokenizer/tokenizer.json`	3.4 MB	`5ece781dc8d2b2f3…`
`tokenizer/tokenizer_config.json`	27.6 KB	`36c6fd44d07d10fd…`
`vlm_meta.json`	201 B	`b36fcf7ef7da8c2a…`

Full checksums are in manifest.json. The playground verifies them after download.

Running it

These programs were exported with the ExecuTorch WebGPU backend (backends/webgpu) and run in a browser (headless Chromium, WebGPU on SwiftShader) with the WebAssembly runtime build used by the ExecuTorch WebGPU Playground, where every generated token (78 of 78 over 4 turns) matched the same programs run on CPU. That runtime build is published in executorch-community/ExecuTorch-WebGPU-Runtime; the playground is the easiest way to run them. It downloads these files once, caches them in the browser, and runs everything locally.

Requirements: a browser with WebGPU enabled.

Provenance

Upstream: HuggingFaceTB/SmolVLM-500M-Instruct at revision a7da5b986cb59b408707209984f360a5f4ad7e47 (official)
Exported with: executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git 3b6068392324)
Reproduce: python recipes/smolvlm-500m-instruct-q8.py --out out/smolvlm-500m-instruct-q8 (recipe in export/)
Re-running the recipe produces byte-identical files.

Citation

If you use this model, please cite the original work:

@article{marafioti2025smolvlm,
  title={SmolVLM: Redefining small and efficient multimodal models},
  author={Andrés Marafioti and Orr Zohar and Miquel Farré and Merve Noyan and Elie Bakouch and Pedro Cuenca and Cyril Zakka and Loubna Ben Allal and Anton Lozhkov and Nouamane Tazi and Vaibhav Srivastav and Joshua Lochner and Hugo Larcher and Mathieu Morlon and Lewis Tunstall and Leandro von Werra and Thomas Wolf},
  journal={arXiv preprint arXiv:2504.05299},
  year={2025}
}

License

This is a derivative of the upstream model and is distributed under the same terms (apache-2.0); see LICENSE. Modifications (conversion to the ExecuTorch format and quantization) by Meta Platforms, Inc.