vanishingradient/deepagents-rl-coding

deepagents-rl

Train / evaluate a coding agent in the deepagents harness on long-horizon SWE-bench tasks, with official Docker evaluation (not proxy rewards).

Protocol

1. Load SWE-bench/SWE-bench_Lite (image, eval_script, FAIL_TO_PASS, PASS_TO_PASS)
2. Full git checkout of repo @ base_commit as the agent sandbox
3. Agent edits via deepagents tools (ls / read_file / edit_file / execute / …)
4. Export git diff as model_patch
5. Grade inside the instance Docker image:
   • apply patch → run official eval_script (applies test_patch, runs pytest)
   • parse >>>>> Start/End Test Output
   • resolved = FULL iff all FAIL_TO_PASS pass and all PASS_TO_PASS maintained
6. Record trajectory (messages, tools, patch, Langfuse, W&B, JSONL)

Quick start

# Requires Docker Desktop running
python scripts/verify_swe_docker_grader.py   # gold patch must FULL-resolve

python scripts/prepare_swe_lite_and_push.py  # push metadata to HF

python scripts/run_deepagent_rollouts.py \
  --split dev --limit 1 --mode tool_loop --log-level DEBUG

Env: MODEL_BASE_URL, MODEL_NAME, Langfuse + W&B keys, Docker engine up.

Logging (every step)

Every stage is logged: repo clone/fetch, each LLM turn, every tool call (args + result), every docker … command, patch apply, eval_script, F2P/P2P grade.

Under --artifacts (default artifacts/swe_lite/):

File	Contents
run.log	Human-readable step log
events.jsonl	One JSON event per line (full replay)
trajectories.jsonl	Full episode trajectories
evals/<episode_id>/	model.patch, eval.sh, test_output.txt, report.json

Env: DARL_LOG_LEVEL, DARL_LOG_FILE, DARL_EVENTS_JSONL.

Langfuse trace shape

Each episode is one agent trace with nested children:

deepagent-swe-episode          (agent)
├── agent-loop                 (span)
│   ├── turn-0                 (span)
│   │   ├── llm-turn-0         (generation via CallbackHandler)
│   │   ├── ls / read_file …   (tool)
│   │   └── …
│   ├── turn-1 …
│   └── …
└── reward_grader              (evaluator)
    ├── git_diff_model_patch   (span)
    └── docker_swe_eval        (span)

Scores on the trace: pass_frac, reward_total, resolved, n_tool_calls.

Grader

src/deepagents_rl/harness/swe_docker_eval.py — same protocol as the SWE-bench harness (instance image + eval script + F2P/P2P). No file-overlap shortcuts.