NVIDIA RoboLab

RoboLab is a benchmark of 120 tabletop manipulation tasks for a DROID robot (a Franka Panda arm with a Robotiq 2F-85 gripper), simulated with NVIDIA Isaac Lab and rendered with Isaac Sim’s RTX renderer. Tasks span pick-and-place, stacking, sorting, reorientation and language-conditioned variants (spatial relations, colors, counting, vague instructions), graded simple, moderate or complex.

• Paper: RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• GitHub: NVlabs/RoboLab
• Project page and leaderboard: research.nvidia.com/labs/srl/projects/robolab

Requirements and licences

• GPU: an NVIDIA RTX GPU (Isaac Sim renders with RTX). One env of one task takes about 7 GB of VRAM, Isaac Sim included; each further env in the same task adds about 1 to 2 GB (RoboLab’s guide fits 30 to 100 envs on a 48 GB GPU, depending on the task). Your policy needs its own memory on top.
• Isaac Sim: RoboLab runs on Isaac Sim 6.0.1 and Isaac Lab 3.0 beta 2 here, RoboLab’s Isaac Sim 6 stack. Using Isaac Sim accepts the NVIDIA Isaac Sim license; the Docker image accepts it for you, as RoboLab’s own image does.
• Assets: RoboLab’s scenes, objects and robot models (about 6 GB, from its git LFS) are mostly licensed CC BY-NC-SA 4.0, with some under other terms; see RoboLab’s THIRD_PARTY_NOTICES.md. LeRobot does not distribute them: the image downloads them from NVlabs when you build it.
• Docker, for the served setup below: BuildKit (docker buildx) to build the image, and the NVIDIA Container Toolkit so that the container can use the GPU.

Run it served (recommended)

Isaac Lab 3.0 beta pins transformers==4.57.6 (hence huggingface-hub<1.0), numpy==2.3.1 and packaging==26.0, which LeRobot’s own dependencies don’t accept. Run RoboLab in its own Docker image, behind lerobot-env-server, and your policy in your usual LeRobot environment:

# Build the image on first use (Isaac Sim and the assets: a long first build), then serve every task
lerobot-env-server --config_path=docker/sim/configs/robolab.yaml --docker=true

# In your LeRobot environment
lerobot-eval \
  --policy.path=<your DROID joint-position policy> \
  --env.type=robolab \
  --env.task=BananaInBowlTask \
  --env.server.endpoint=tcp/127.0.0.1:7448 \
  --rename_map='{"observation.images.over_shoulder_left_camera": "observation.images.<exterior>", "observation.images.wrist_cam": "observation.images.<wrist>"}' \
  --eval.batch_size=1 \
  --eval.n_episodes=10

The server runs any of its configured tasks: the client opens the one it asks for, and the server builds it in the running Isaac Sim (a few seconds). Starting Isaac Sim itself takes from seconds to a few minutes. If it finds no GPU in the container (ERROR_INCOMPATIBLE_DRIVER), see the Vulkan note in Simulation.

Run it locally

With Isaac Sim and RoboLab installed in your environment, drop --env.server.endpoint and lerobot-eval runs RoboLab in its own process. Install RoboLab’s stack first, then LeRobot without its dependencies, as the Docker image does:

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/NVlabs/RoboLab.git && git -C RoboLab lfs pull
uv venv --python 3.12 && source .venv/bin/activate
uv pip install "isaaclab[isaacsim]==3.0.0b2.post1" -e ./RoboLab \
  --extra-index-url https://pypi.nvidia.com --index-strategy unsafe-best-match
uv pip install --no-deps -e ./lerobot
uv pip install draccus termcolor eclipse-zenoh  # what the rest of LeRobot's simulation code imports
export OMNI_KIT_ACCEPT_EULA=YES                 # accepts the Isaac Sim license

This environment has no room for policies that need transformers 5 (π₀.₅, SmolVLA, …): run those served.

Tasks

Each of RoboLab’s 120 tasks is its own suite with one task id, so lerobot-eval reports a success rate per task and overall. --env.task takes a comma-separated list of task names, e.g. BananaInBowlTask,RubiksCubeAndBananaTask; leave it empty for all 120. The task names, instructions and difficulty are listed in RoboLab’s task metadata.

All envs of a task share one Isaac Lab world on the GPU: --eval.batch_size=N runs N episodes at once in that world.

Observations and actions

The robot runs at 15 Hz (physics at 120 Hz, 8 physics steps per action), and LeRobot records it as robot_type="Franka", as in DROID. Each episode has its task’s time limit (20 to 300 s, so 300 to 4,500 steps); an episode that succeeds ends as terminated, one that runs out of time as truncated. The task’s instruction is the env’s task description.

Sim robot

RoboLab can also stand in for a real DROID robot in lerobot-rollout, lerobot-record and the other robot tools (see Simulation: local or served):

lerobot-rollout \
  --robot.type=sim --robot.env.type=robolab --robot.env.task=BananaInBowlTask --robot.task_id=0 \
  --robot.env.server.endpoint=tcp/127.0.0.1:7448 \
  --policy.path=<your policy> --fps=15 ...

Environment config reference

Comparing with the leaderboard

RoboLab’s leaderboard evaluates 10 episodes per task with the default instruction, the WRIST_LEFT cameras and joint-position actions, which are LeRobot’s defaults. It doesn’t state which Isaac Sim version produced its numbers, and RoboLab’s default is Isaac Sim 5.0: physics and rendering differ between Isaac Sim versions, so numbers from this Isaac Sim 6.0.1 setup are not directly comparable.