Ling3-Tiny-UBTree-B16

Paper · Target model

Ling3-Tiny-UBTree-B16 is a UBTree speculative-decoding draft model for inclusionAI/Ling-3.0-tiny. It combines a five-layer DFlash backbone with a learned bigram selector to propose multiple candidate continuations for target-model verification.

This checkpoint is used alongside the matching target model; it cannot serve as a standalone text-generation model.

How UBTree works

UBTree pipeline

UBTree pairs a unigram proposer with a bigram selector:

A DFlash backbone proposes candidate tokens for future positions in parallel.
The selector scores transitions between adjacent candidates in parallel.
Tree construction retains promising branches within a verification budget.
The target verifies the tree and commits the accepted continuation.

The paper also introduces tree-native training with renormalized KL supervision on high-temperature target traces, encouraging plausible alternative branches. See the paper for the method and experiments.

Model specifications

Item	Value
Target model	inclusionAI/Ling-3.0-tiny
Draft architecture	`Ling3UBTreeModel`
Draft backbone	5 full-attention Transformer layers
Parameters stored in this checkpoint	183,127,326 (183.1M)
Weight dtype	BF16
Hidden size	1,536
Attention heads / KV heads	16 / 16
Target auxiliary feature layers (zero-based)	4, 8, 12, 16, 22
Draft block size	16 (anchor-inclusive runtime convention)
Candidate top-k	64 per draft position
Default verification tree budget	64 non-root nodes + 1 anchor
Training	DFlash warm start, then joint backbone + UBTree selector training

The parameter count covers tensors distributed in this repository. The draft shares token embeddings and the language-model head with the target; these shared weights are not included here. B16 denotes the draft block size, not the verification tree's node budget.

Serving with SGLang

Use the asahi/ubtree SGLang branch for the commands below. Upstream integration is in progress in sgl-project/sglang#44050.

CUDA_VISIBLE_DEVICES=0 \
SGLANG_OPT_FA3_VERIFY_METADATA_IN_GRAPH=1 \
SGLANG_OPT_UBTREE_FA3_TREE_ONLY_MASK=1 \
python -m sglang.launch_server \
  --model-path /path/to/ling3-tiny \
  --served-model-name ling3-tiny \
  --tp-size 1 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend fa3 \
  --page-size 1 \
  --context-length 4096 \
  --max-total-tokens 4096 \
  --max-running-requests 1 \
  --mem-fraction-static 0.80 \
  --max-prefill-tokens 4096 \
  --chunked-prefill-size 4096 \
  --disable-radix-cache \
  --mamba-backend triton \
  --linear-attn-backend triton \
  --linear-attn-verify-backend triton \
  --mamba-ssm-dtype float32 \
  --cuda-graph-backend-prefill disabled \
  --cuda-graph-backend-decode full \
  --cuda-graph-bs-decode 1 \
  --disable-cuda-graph-padding \
  --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --trust-remote-code \
  --speculative-algorithm UBTREE \
  --speculative-draft-model-path /path/to/ling3-tiny-ubtree \
  --speculative-num-draft-tokens 49

Benchmark results

All methods use block size 16, TP1, concurrency 1, BF16, and greedy no-thinking decoding. UBTree uses a 49-node tree.

Each cell shows acceptance length / E2E speedup over target-only AR.

Dataset	UBTree	DFlash	DSpark
GSM8K	9.53 / 3.055×	6.57 / 2.665×	7.53 / 2.718×
MATH-500	9.86 / 3.518×	5.74 / 2.643×	6.96 / 2.826×
AIME 2025	8.44 / 3.152×	4.76 / 2.261×	5.37 / 2.263×
HumanEval	10.04 / 3.411×	6.39 / 2.770×	7.59 / 2.914×
MBPP	9.35 / 3.062×	5.87 / 2.473×	7.16 / 2.678×
LiveCodeBench	7.46 / 2.794×	4.11 / 1.948×	4.84 / 2.030×

AL and throughput use three-round medians; the matched AR baseline uses one round.

Citation

If you use UBTree in your work, please cite:

@article{liang2026ubtree,
  title   = {UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding},
  author  = {Liang, Chumeng and Wang, Linxuan and Peng, Xinyu and Liu, Huabin and Chen, Yuxin and Liu, Ge and Lin, Guang and Song, Qifan and Li, Jianguo},
  journal = {arXiv preprint arXiv:2609.39972},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.39972}
}

License

The model weights are released under the MIT License. The pipeline figure is from Liang et al., Figure 2, reproduced without modification under CC BY 4.0.