PersonaDrive

Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation

Mahmoud Srewa · Praneetsai Vasu Iddamsetty · Mohammad Abdullah Al Faruque · Salma Elmalaki

University of California, Irvine · IEEE/CVF WACV 2027

Project page · Paper (arXiv 2606.12616) · Code · Demo video

PersonaDrive: conservative vs. aggressive with the same weights, then PersonaDrive vs. SimLingo at a red light

A driving agent that drives carefully or aggressively, based on how real people drove.

This is the released driving model of PersonaDrive. PersonaDrive conditions a vision-language-action (VLA) driving agent on demonstrations retrieved from people who were asked to drive in a given style, whether aggressive, neutral or conservative. One set of weights serves every style. Switching style only changes which database of human demonstrations the retrieval head searches.

Videos, the method and the full results are on the project page.

PersonaDrive architecture

This checkpoint

This checkpoint is Stage 3 of the PersonaDrive pipeline (in-context fine-tuning). It is SimLingo's driving model, fine-tuned to read K = 2 retrieved context points as few-shot demonstrations before it predicts its own waypoints. Each context point holds an image, control history, command, target points, commentary and executed waypoints. The model gets no style label and infers the style from the demonstrations.

Initialised from RenzKa/simlingo (epoch=013)
Backbone OpenGVLab/InternVL2-1B
Parameters 1.36 B in total, of which about 24 M are trained in this stage
Retrieved context points K = 2
Simulator / benchmark CARLA 0.9.15 / Bench2Drive

The retrieval head (Stages 1–2) is a separate model of about 2 M trained parameters. It is in the code repository as persona_drive_rag/checkpoints/best_model.pt.

Files

Path Size Purpose
personadrive_icl_sft/checkpoints/epoch=000.ckpt/pytorch_model.pt 1.96 GB The model. A single-file PyTorch state dict that loads directly for evaluation.
personadrive_icl_sft/.hydra/config.yaml 2 KB The training config. The agent builds the model from it and finds .hydra/ by walking up from the checkpoint path.
retrieval/normal/driving_index.faiss 221 MB Vectors of the normal (no-style) retrieval index, covering 54,040 frames of the SimLingo dataset and embedded with the released retrieval head.
retrieval/normal/frames.json.gz 1 MB The frames those vectors belong to. The SimLingo data itself is not included, and scripts/retrieval/fill_normal_index.py in the code repository completes the index from your SimLingo download.
retrieval/normal/driving_index.info.json < 1 KB Index type and size.

pytorch_model.pt was consolidated from the run's DeepSpeed ZeRO-2 checkpoint (last.ckpt). It contains only tensors, so it also loads with torch.load(..., weights_only=True).

The checkpoint is the complete model, not just an adapter. You don't need SimLingo's weights to use it. The weights trained in this stage are stored in fp32 and the frozen ones in bf16.

Part Tensors In this stage
LoRA adapters on the language model (r 32, α 64), stored unmerged 336 trained, continuing from SimLingo's adapters
Vision-to-language projector (mlp1) 6 trained
Driving adaptor 10 trained
Waypoint encoder 6 trained
Language-model base weights and language adaptor 294 frozen (as in SimLingo)
Vision encoder (InternViT) 340 frozen (as in SimLingo)

The file is loaded by the PersonaDrive code. It is not a transformers or PEFT checkpoint.

How to use

Requirements

  • Linux x86-64 and CARLA 0.9.15.
  • An NVIDIA Ampere, Ada or Hopper GPU, for bf16 and flash-attention 2. Blackwell GPUs need a newer PyTorch than the pinned 2.2.2.
  • Installation steps are in the code repository.

Run this in a clone of the repository, with the personadrive environment active.

# the model and its config (~2 GB)
hf download PervasiveAutonomyLab/PersonaDrive --local-dir pretrained \
    --include "config.json" "personadrive_icl_sft/.hydra/*" "personadrive_icl_sft/checkpoints/epoch=000.ckpt/*"

# one Bench2Drive route on one GPU, aggressive style, K = 2
export AGENT_CKPT=$PWD/pretrained/personadrive_icl_sft/checkpoints/epoch=000.ckpt/pytorch_model.pt
PERSONADRIVE_ICL_PROFILE=fast \
ROUTE_XML=$PERSONADRIVE_ROOT/leaderboard/data/bench2drive_split/bench2drive_00.xml \
OUT_DIR=/tmp/personadrive_route00 \
bash scripts/eval/test_one_lb1_route.sh
  • Style. Set PERSONADRIVE_ICL_PROFILE to fast (aggressive), default (neutral, normal driving), slow (conservative) or normal (no style, the index built from SimLingo's PDM-lite data). PERSONADRIVE_NUM_ICL=0 turns retrieval off.

  • Retrieval data. Retrieval needs three things.

    • The retrieval head, included in the code repository.
    • The FAISS indices under persona_drive_rag/, built as in Build the retrieval indices (the normal index is completed from retrieval/normal/ in this repository).
    • The camera frames those indices point to.

    The repository's datasets/README.md lists which archives to get for each style. Retrieved frames whose image is missing are skipped. If none can be loaded, the model runs without retrieval.

Results

These are the numbers published in the paper (Bench2Drive, 220 routes, closed loop).

Without style DS ↑ SR (%) ↑
SimLingo 85.07 ± 0.95 67.27 ± 2.11
HiP-AD 86.77 69.09
PersonaDrive (normal index) 88.95 72.29
PersonaDrive with style (one checkpoint) DS ↑ Speed (m/s) Accel. (m/s²)
Conservative 89.62 5.05 3.17
Neutral 87.55 5.52 3.51
Aggressive 88.69 6.05 3.96

From conservative to aggressive, speed rises by 19.8% and acceleration by 24.9%. The project page compares these with SimLingo prompted with the style name, StyleDrive and DMW.

Training

  • Data.

    • Style-instructed driving in all three styles, aggressive, neutral and conservative.
    • 2,848 PDM-lite routes from SimLingo's dataset, which keep the basic driving skill.

    Each frame is paired with its K = 2 context points, retrieved from the index of the matching style.

  • Settings. LoRA on the language model (r 32, α 64, dropout 0.1) with the vision encoder frozen. AdamW at lr 3e-5 (OneCycle, weight decay 0.1), batch 6 per GPU × 4 GPUs with DeepSpeed ZeRO-2 in bf16, and 1 epoch.

  • Hardware. 4× NVIDIA L40S (48 GB), about 12 h.

  • Reproducing the run. python simlingo_training/train.py experiment=personadrive_icl_sft in the code repository. The composed config of this run is personadrive_icl_sft/.hydra/config.yaml.

Intended use and limitations

  • Research use in simulation only (CARLA). The model is not meant to control a real vehicle.
  • Where the styles come from. 15 participants who drove under aggressive, neutral and conservative instructions on a driver-in-the-loop rig. Styles are paired at route level, so style is separated from driver identity.
  • Speed. With K = 2 the model takes about 0.56 s per simulator step (paper, offline), so the agent does not run in real time at CARLA's 20 Hz.

License

Apache 2.0, the same license as SimLingo (code and the RenzKa/simlingo weights). The backbone InternVL2-1B is released under the MIT license.

The human driving data behind PersonaDrive's driving styles was collected by us and is released as the PersonaDrive dataset. SimLingo's PDM-lite data was used to train the SimLingo base model, for part of the Stage-3 fine-tuning data and for the style-agnostic normal index. Its licence asks for the attribution below, and the released weights may only be shared free of charge for academic or scientific use and must not be used to operate, or help operate, a vehicle or robot, or for any high-risk purpose.

This work was made using the SimLingo dataset, provided by Wayve Technologies Ltd under licence terms available at https://github.com/RenzKa/simlingo.

Citation

@inproceedings{srewa2027personadrive,
  title     = {PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation},
  author    = {Srewa, Mahmoud and Iddamsetty, Praneetsai Vasu and Al Faruque, Mohammad Abdullah and Elmalaki, Salma},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

Acknowledgements

This work is supported by the U.S. National Science Foundation under grant 2339266 and partially supported by the UCI ProperAI Institute. PersonaDrive builds directly on SimLingo, CARLA garage, DriveLM / PDM-Lite and Bench2Drive, and we thank their authors for releasing their code.

Downloads last month
78
Video Preview
loading

Model tree for PervasiveAutonomyLab/PersonaDrive

Base model

RenzKa/simlingo
Finetuned
(1)
this model

Paper for PervasiveAutonomyLab/PersonaDrive