LegoACE: Autoregressive Construction Engine for Expressive LEGO® Assemblies

Official model weights for LegoACE, presented at SIGGRAPH Asia 2025.

LegoACE is an autoregressive transformer that generates LEGO® assemblies as sequences of placed bricks. This repository hosts two pretrained variants:

Subfolder	Conditioning	Encoder	Training steps
mv/	Multi-view images (4 views)	DINOv2-base	520K
text/	Text descriptions	CLIP ViT-B/32	210K

• 📄 Paper: LegoACE @ SIGGRAPH Asia 2025
• 💻 Code: VAST-AI-Research/LegoACE
• 📊 Architecture: 32-layer Llama-style transformer, hidden size 768, vocab ~16K

---

Quick start

Full inference pipeline (LDR tokenizer, multi-view rendering, LDR → GLB conversion) lives in the GitHub repository. The snippets below show only how to load the weights.

git clone https://github.com/VAST-AI-Research/LegoACE.git
cd LegoACE
pip install -e .

Multi-view image conditioned (recommended)

from model.llama_image_condition import ImageConditionModel

model = ImageConditionModel.from_pretrained("VAST-AI/LegoACE", subfolder="mv").to("cuda")

End-to-end usage with the dataset/MVNpzDataset.py loader and Blender-based GLB export is documented in the GitHub README:

python inference/inference_multi_view.py \
    --ckpt_dir VAST-AI/LegoACE \
    --dataset_name <your_dataset> \
    --dataset_class dataset.MVNpzDataset.MVNpzDataset \
    --save_dir ./outputs/inference \
    --save_name mv-demo \
    --infer_number 100 --batch_size 4 --repeat 4 --dataset_split val

Text conditioned

from model.llama_text_condition import TextConditionModel

model = TextConditionModel.from_pretrained("VAST-AI/LegoACE", subfolder="text").to("cuda")

python inference/inference_text_condition.py \
    --ckpt_dir VAST-AI/LegoACE \
    --dataset_name <your_dataset> \
    --save_dir ./outputs/inference --save_name text-demo \
    --prompts "A red sports car" "A modern brick bed" "A bridge over a river"

---

Outputs

Each generation step emits a quintuple (x, y, z, rotation_id, brick_type_id). The full pipeline converts those token sequences into:

1. LDR — text-format LEGO instructions (LDraw)
2. GLB — 3D mesh via Blender + ImportLDraw
3. Normal maps — pyrender renderings of the assembled model

LegoACE supports an LDR vocabulary covering 28 common brick types and 20 discrete rotation classes; see utils/brick_ids.py.

---

Intended uses & limitations

Intended uses

• Research on autoregressive 3D / LEGO® generative models.
• Generating LEGO assemblies for academic and creative exploration.

Limitations

• Outputs are restricted to the 28-brick vocabulary used in training.
• Quality depends on prompt phrasing (text) or image quality (multi-view).
• The model has been trained primarily on small/medium-scale assemblies and may produce structurally unstable or non-buildable arrangements.
• Generation requires the LDR tokenizer files (*_dat_dict.json, *_rot_dict.json) that ship with the dataset, not with these weights.

---

Citation

@inproceedings{xu2025legoace,
  author    = {Hao Xu and Yuqing Zhang and Yiqian Wu and Xinyang Zheng and
               Yutao Liu and Xiangjun Tang and Yunhan Yang and Ding Liang and
               Yingtian Liu and Yuanchen Guo and Yanpei Cao and Xiaogang Jin},
  title     = {LegoACE: Autoregressive Construction Engine for Expressive LEGO{\textregistered}
               Assemblies},
  booktitle = {Proceedings of the {SIGGRAPH} Asia 2025 Conference Papers},
  publisher = {{ACM}},
  year      = {2025},
  pages     = {40:1--40:11},
  doi       = {10.1145/3757377.3763881},
  url       = {https://doi.org/10.1145/3757377.3763881}
}

---

License

Released under the MIT License.

LEGO® is a trademark of the LEGO Group, which does not sponsor, authorize, or endorse this project.