executorch-community/Qwen2.5-0.5B-Instruct-ExecuTorch-WebGPU

Qwen2.5 0.5B Instruct for ExecuTorch WebGPU

This repository contains an ExecuTorch conversion of Qwen/Qwen2.5-0.5B-Instruct for the ExecuTorch WebGPU backend, so the model runs entirely in a web browser on the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on executorch.ai).

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Format: ExecuTorch .pte programs delegated to the WebGPU backend
• Quantization: 4-bit groupwise weights (q4gsw), fp16 KV cache
• Total size: 296.5 MB

About the original model

The 0.5B instruction-tuned model of the Qwen2.5 family.

• Developed by: Qwen Team, Alibaba Cloud
• Paper: Qwen2.5 Technical Report
• 0.49B parameters, 24 layers
• Context length 32,768 tokens; generation up to 8,192 tokens
• Supports over 29 languages

For training data, evaluation results and full details, see the original model card: Qwen/Qwen2.5-0.5B-Instruct.

Intended use and limitations

Chat and instruction following in a small footprint.

Small language models can state false information confidently, repeat themselves, and reflect biases in their training data. Check important outputs.

What changed in this conversion

• Weights quantized to 4-bit groupwise (q4gsw); the KV cache is fp16
• Context capped at 8,192 tokens (upstream: 32,768)

Inputs and outputs

• forward(tokens int64 [1, 1..512], input_pos int64 [1]) -> logits float32 [1, 151936] (last position only)
• The KV cache lives inside the program; call it with up to 512 new tokens at a time, then one token per step
• Tokenize with the files in tokenizer/ (the upstream Qwen2.5 tokenizer) and the ChatML template

Files

File	Size	SHA-256
qwen2_5_0_5b_instruct_q4gsw.pte	285.6 MB	474274b934978f6c…
tokenizer/config.json	659 B	18e18afcaccafade…
tokenizer/generation_config.json	242 B	e558847a8b440261…
tokenizer/merges.txt	1.6 MB	599bab5407508877…
tokenizer/tokenizer.json	6.7 MB	c0382117ea329cdf…
tokenizer/tokenizer_config.json	7.1 KB	5b5d4f65d0acd3b2…
tokenizer/vocab.json	2.6 MB	ca10d7e9fb3ed185…

Full checksums are in manifest.json. The playground verifies them after download.

Running it

These programs were exported with the ExecuTorch WebGPU backend (backends/webgpu) and validated with the WebAssembly runtime build used by the ExecuTorch WebGPU Playground (launching soon on executorch.ai). That runtime build is not published yet; until it is, the playground is the supported way to run them. It downloads these files once, caches them in the browser, and runs everything locally.

Requirements: a browser with WebGPU enabled. The language models use the shader-f16 GPU feature when available and fall back to fp32 math without it.

Provenance

• Upstream: Qwen/Qwen2.5-0.5B-Instruct at revision 7ae557604adf67be50417f59c2c2f167def9a775 (official)
• Exported with: executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git 3b6068392324)
• Reproduce: python recipes/qwen2.5-0.5b-instruct.py --out out/qwen2.5-0.5b-instruct (recipe in export/)
• Re-running the recipe produces byte-identical files.

Citation

If you use this model, please cite the original work:

@misc{qwen2.5,
  title={Qwen2.5: A Party of Foundation Models},
  url={https://qwenlm.github.io/blog/qwen2.5/},
  author={Qwen Team},
  month={September},
  year={2024}
}

License

This is a derivative of the upstream model and is distributed under the same terms (apache-2.0); see LICENSE. Modifications (conversion to the ExecuTorch format and quantization) by Meta Platforms, Inc.