EvoLenTokenizer/base-100k

model_base_5120

This model is a BERT-based masked language model with a 5,120-token vocabulary, trained on human genomic sequences as part of the EvoLen study: EvoLen: Evolution-Guided Tokenization for DNA Language Model.

Model description

The model uses a BERT-base architecture (12 layers, 768 hidden units, 12 attention heads) and was trained with a masked language modeling objective on DNA sequences tokenized into 512-token windows. It can be used as a foundation model for downstream DNA sequence tasks or for feature extraction.

Intended uses & limitations

This model is intended for research on DNA language modeling, including fine-tuning on genomic benchmarks such as regulatory element classification and ATAC-seq tasks. It was trained on human reference genome (hg38) data and may not generalize to other organisms or non-conserved sequences without further adaptation.

Training and evaluation data

The model was pretrained on the human genome (hg38) using masked language modeling. Evaluation was performed on a held-out split of the same corpus. The evaluation results are:

• Loss: 5.0825
• Accuracy: 0.2308
• Perplexity: 161.17

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 4e-05
• train_batch_size: 96
• eval_batch_size: 96
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 768
• total_eval_batch_size: 768
• optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 10000
• training_steps: 100000

Training results

• Train loss: 5.6056
• Eval loss: 5.0825
• Eval accuracy: 0.2308
• Perplexity: 161.17

Framework versions

• Transformers 4.35.2
• Pytorch 2.8.0+cu128
• Datasets 4.4.1
• Tokenizers 0.15.2

Citation

If you use this model, please cite:

@inproceedings{huang2026evolen,
  title     = {EvoLen: Evolution-Guided Tokenization for DNA Language Model},
  author    = {Huang, Nan and Zhou, Xiaoxiao and Cui, Junxia and
               Tapia-Pacheco, Mario and Amariuta, Tiffany and Li, Yang and
               Shang, Jingbo},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}