BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion
Abstract
BYOD converts autoregressive language models into diffusion models via parameter-efficient LoRA adapters, enabling parallel generation with minimal training cost while preserving base-model knowledge.
Unlike autoregressive (AR) models, diffusion language models (DLMs) permit parallel token updates, flexible generation, and bidirectional attention, but leading systems rely on costly pretraining or full-model fine-tuning. We present Build Your Own DLM (BYOD), a generalizable LoRA-only recipe for parameter-efficient AR-to-DLM conversion. Without architecture-specific tuning, BYOD induces diffusion generation in Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B; to our knowledge, this is the first reported AR-to-DLM conversion of a Mistral model. The adapters comprise 4.1-5.8% of final parameters while every base-model parameter remains frozen; each 25,000-iteration conversion needs at most 102.4M tokens, processed in 3.1-4.0 hours on one Google Colab GPU. All four models generate from fully masked sequences, use bidirectional context in a controlled attention test, and retain substantial parent-model knowledge. BYOD-Gemma is the strongest BYOD conversion on our nine-task evaluation, reaching 50.6% average accuracy compared with 51.9% for LLaDA 8B Instruct in the same harness. Open-generation quality at low network function evaluation (NFE) budgets remains the principal limitation: BYOD perplexity is 8.3-11.1 versus 4.1 for LLaDA and 2.2-3.0 for the AR parents when generating 128 tokens in 64 steps. To enable further experimentation and inspection, we provide a standalone training and evaluation implementation and interactive demos for all four conversions. BYOD thus provides a practical route for extending diffusion generation to other existing model families while identifying the gap that efficient conversion must next close.
Community
BYOD: Build your own diffusion language model in Google Colab
Can an existing instruction model learn to generate text through diffusion in a few hours on one GPU?
In BYOD: Build Your Own DLM, we convert Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B into bidirectional masked-diffusion language models. Every pretrained weight stays frozen: training updates only LoRA adapters, representing 4.1–5.8% of the final model parameters. The four primary conversions each took 3.1–4.0 hours on one GPU in Google Colab.
We want other people to be able to train, inspect, and experiment with these models too. The training notebook and interactive demos are available now.
Try the fine-tuned diffusion models before training your own:
These demos run directly in your browser:
| Model | Interactive demo |
|---|---|
| Gemma 2 9B | Try BYOD-Gemma |
| Llama 3.1 8B | Try BYOD-Llama |
| Qwen2.5 7B | Try BYOD-Qwen |
| Ministral 8B | Try BYOD-Ministral |
Instead of appending one token at a time, the model starts with a masked answer and fills positions through successive denoising steps, with a flexible inference budget. Try the same prompt with different step budgets and see how it influences inference speed and quality. The hosted demos also include experimental decoding options beyond the paper’s evaluation settings.
For related demos, try our earlier LAD model, which also enabled the changing of unmasked tokens.
Train your own model
Open the training notebook in Google Colab · Code and instructions
The notebook has been designed to make it as easy as possible to help you train your own DLM. No coding is needed, you only need to run each code cell:
- Choose a model.
- Use the provided tokenized training data, or configure your own instruction–response dataset.
- Train while watching the loss curve to see if the model keeps improving.
- Open the interactive inference interface via the notebook step 6 and try out your model. You can also stop a run early and still use it in the interface.
Quick mode trains smaller Llama, Gemma, or Qwen models and rank-128 LoRA for initial experiments. Paper mode uses the original 7B–9B models and rank-1024 adapters and needs a paid GPU; our paper runs used a 96 GB NVIDIA RTX PRO 6000.
What did we find?
The same conversion recipe works across all four families. All four generate from fully masked answers and use information to the right of a masked position in our controlled attention test.
BYOD-Gemma comes within 1.3 percentage points of LLaDA on our nine-task average: 50.6% versus 51.9%, evaluated in the same harness. The unchanged AR parents still perform better on average, with particularly large conversion gaps in mathematics and code.
Figure 1: Task-level accuracy in the matched evaluation. Performance depends on both the task and the converted model family.
Generation at a low number of inference steps: At 64 denoising steps for 128 output tokens, externally scored perplexity is 8.3–11.1 for BYOD, versus 4.1 for LLaDA and 2.2–3.0 for the AR parents. Increasing the denoising budget improves the results, but our low-cost conversion still leaves a quality gap.
Figure 3: Generation quality across inference budgets. Lower perplexity and higher Distinct-1 are better; crosses mark the AR parents.
We hope BYOD makes diffusion-model experiments easier to start: change a backbone, try different training data, inspect a denoising trajectory, and see what improves.
Start with the Colab notebook · Read the paper · Get the code
BYOD: Build Your Own Diffusion Language Model
Can an existing autoregressive model learn to generate text through diffusion in a few hours on a single GPU?
In BYOD: Build Your Own DLM, we convert Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B into bidirectional masked-diffusion language models. Every pretrained weight stays frozen: training updates only LoRA adapters, representing 4.1–5.8% of the final model parameters. The four primary conversions each took 3.1–4.0 hours on one GPU in Google Colab.
We want other people to be able to train, inspect, and experiment with these models too. The training notebook and interactive demos are available now.
Really nice to see AR-to-diffusion conversion with frozen base weights that fits in 3 to 4 hours on a single Colab GPU; that makes it reproducible on homelab hardware too. I'm curious about the fixed rank-1024 Q/V-only LoRA: did you try lower ranks or adding K, O or MLP targets, to see whether the low-NFE perplexity gap (8.3 to 11.1 versus 2.2 to 3.0 for the AR parents) is mostly an adapter capacity limit or a property of the plain IID absorbing-mask objective? The merged checkpoint as a starting point for a short full diffusion fine-tune, which you mention as future work, sounds like a very natural next experiment.
Thanks @orinmemory ! We did experiment with applying LoRA to additional target modules (K, O, and MLP), but these did not seem to substantially improve performance in our initial experiments.
In our earlier paper, LAD: LoRA-Adapted Diffusion, we also investigated different model sizes and LoRA ranks, although using a somewhat different training and inference approach. We found that increasing the LoRA rank generally improved performance, but with diminishing returns. For the 8B models, increasing the rank beyond 1024 provided little additional benefit.
That said, we haven't systematically studied the effects of adapter capacity, so the exact contribution to the perplexity gap remains an open question. And I agree that a short full-parameter fine-tuning run starting from the merged checkpoint would be a very interesting next experiment. Although there is also a cool benefit from keeping the diffusion adaption separate; it enables the model to instantly switch between autoregressive and diffusive generation!
Get this paper in your agent:
hf papers read 2610.41574 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash