Sajjad Pakdamansavoji

savoji

https://sajjadpsavoji.github.io

AI & ML interests

VLM, VLA, LLM, Robotics, Computer Vision

Recent Activity

posted an update about 9 hours ago

APEX removes DMD's separate fake-score network for streaming video distillation: 4.4% less peak memory, 9.2% shorter iterations, and +30.0% net human preference on instruction following. DMD needs an auxiliary network that tracks the generator's output distribution. APEX uses the generator itself. The original text condition generates, and the negated projected text (c_shift = -c) selects fake-velocity estimation with the same weights. The frozen teacher, Self-Replay Forcing and perceptual loss from S2-Avatar stay. The price is weaker motion. 🔍 What changes • One set of weights and, by default, shared AdamW states • Inference uses only the generation path, then a one-step 2x Refiner • The fake estimator is fitted to replay outputs; its clean-latent gap to the teacher gives a detached generator correction 📊 The numbers • Vs DMD at matched settings: 102.7 to 98.2 GiB per GPU, 35.9 to 32.6 s per iteration • Vs S2-Avatar, 70 paired 180 s cases at 3 NFE: instruction 22 wins, 47 ties, 1 loss; motion -8.6% • Progressive VIF 3.856 vs 3.831, aesthetics 0.642 vs 0.633, pooled AVSync error 1.319 to 1.341 s ⚠️ Caveats • One cost measurement with no variance reported; 70-case rounds with no significance tests • Only 180 s avatar-style video at 1344x768, and the baseline is the authors' own recipe Paper: https://huggingface.co/papers/2611.14033 Would you accept a few percent in training savings and better prompt following in exchange for weaker motion? 📚 Read it elsewhere • The deep dive on Medium: https://medium.com/p/fd4facc13c3f • The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/apex-drops-dmds-fake-score-network • The thread on X: https://x.com/saavoji/status/2120949703953289381 • Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#apex-drops-dmds

repliedto their post 1 day ago

A 284B-parameter DeepSeek-V4-Flash now runs on a 64 GB desktop with 0 GPUs: llama-modchip shrinks the checkpoint to 56.56 GiB (1.71 bpw) and decodes at 9.1 to 14.7 tok/s. The standard UD-IQ1_S build is 76.87 GiB, above the ~62.5 GiB usable on a 64 GB Linux box. The repo mixes precisions (routed experts mostly IQ1_S, MLA projections in a custom Fractal-64 type) and patches the CPU kernels. The model plus a 256K KV cache then sits at 57.61 to 58.9 GiB RSS with zero swap. The repo was created on 2026-10-29 and already has 744 stars and 470 points on Hacker News. 📊 The numbers • Size: 56.56 GiB, 20.31 GiB smaller than UD-IQ1_S • Prefill: 25.0 to 44.36 tok/s (+77%) on a 16-core Ryzen 9 9955HX • Decode: 5.91 tok/s unpatched, 9.06 to 9.95 tok/s at Top-6, 11.5 to 14.7 tok/s with the n-gram drafter ⚠️ Caveats • "Lossless" means bit-exact against the same re-quantized checkpoint. Versus UD-IQ1_S the author reports 0.9964 top-logit cosine similarity • The 98.5/100 quality score comes from the author's own gate, and the headline speeds come from one machine • The custom tensor type 40 is rejected by stock llama.cpp, Ollama and LM Studio 🛠️ Try it • Run scripts/check_cpu_readiness.py from the repo: it needs no model download • Run ./modchip bench for speed, then ./modchip eval to check the quality claims yourself Repo: https://github.com/synchopate/llama-modchip HN thread: https://news.ycombinator.com/item?id=50411961 Would you accept a custom runtime for 9 to 15 tok/s from a 284B model on a 64 GB box? 📚 Read it elsewhere • The deep dive on Medium: https://medium.com/p/e47ab3581f58 • The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/llama-modchip-runs-a-284b-deepseek • The thread on X: https://x.com/saavoji/status/2119858773280141526 • Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#llama-modchip

repliedto their post 1 day ago

View all activity

Organizations

None yet

posted an update about 9 hours ago

Post

APEX removes DMD's separate fake-score network for streaming video distillation: 4.4% less peak memory, 9.2% shorter iterations, and +30.0% net human preference on instruction following.

DMD needs an auxiliary network that tracks the generator's output distribution. APEX uses the generator itself. The original text condition generates, and the negated projected text (c_shift = -c) selects fake-velocity estimation with the same weights. The frozen teacher, Self-Replay Forcing and perceptual loss from S2-Avatar stay. The price is weaker motion.

🔍 What changes
• One set of weights and, by default, shared AdamW states
• Inference uses only the generation path, then a one-step 2x Refiner
• The fake estimator is fitted to replay outputs; its clean-latent gap to the teacher gives a detached generator correction

📊 The numbers
• Vs DMD at matched settings: 102.7 to 98.2 GiB per GPU, 35.9 to 32.6 s per iteration
• Vs S2-Avatar, 70 paired 180 s cases at 3 NFE: instruction 22 wins, 47 ties, 1 loss; motion -8.6%
• Progressive VIF 3.856 vs 3.831, aesthetics 0.642 vs 0.633, pooled AVSync error 1.319 to 1.341 s

⚠️ Caveats
• One cost measurement with no variance reported; 70-case rounds with no significance tests
• Only 180 s avatar-style video at 1344x768, and the baseline is the authors' own recipe

Paper:
Few-Step Streaming Video Generation via Condition Shifting (2611.14033)

Would you accept a few percent in training savings and better prompt following in exchange for weaker motion?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/fd4facc13c3f
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/apex-drops-dmds-fake-score-network
• The thread on X: https://x.com/saavoji/status/2120949703953289381
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#apex-drops-dmds

replied to their post 1 day ago

Fair point. This targets boxes with no GPU, and 9 to 15 tok/s is well under your 45 to 60 bar. Your setup already beats it. The README's hybrid CPU plus GPU result is only for 30B models, not the 284B one, so for agent work a smaller MoE looks like the closer fit.

replied to their post 1 day ago

Good point on the memory access pattern, and the README doesn't test it. The headline speeds all come from one Ryzen 9 9955HX. The leaderboard has an Apple M4 Max row at 77.95 tok/s, but it mixes models so it isn't a clean test of efficiency cores. The 30B MoEs in the repo may fit your size limit better.

posted an update 1 day ago

Post

Comfy-Org's repackaged MiniMax-H3-Image has 351,803 downloads last month, more than the 142,989 of the official repo, which was created five days earlier.

@MiniMaxAI 's 33.1B dense DiT generates about 16 MP directly (4096x4096 up to 6272x2688), with no upscaler, in 6 steps and no classifier-free guidance. One checkpoint covers text-to-image and editing with up to 16 reference images. The card lists recipes for SGLang, vLLM-Omni, diffusers and ComfyUI, and @Comfy-Org added quantized files and workflow templates within days.

🔍 What's new
• The H3 design without audio layers, 50 layers, with the H3 Qwen3-VL text encoder unchanged
• Only 6 steps and guidance 1 are accepted; other values return HTTP 400 and negative_prompt is rejected

🛠️ Try it
• Official BF16 DiT is 68.65 GB, plus a separate 66.7 GB text encoder
• Comfy-Org offers a pruned int8 DiT, int8 or NVFP4 text encoders, an fp16 VAE, and T2I and Edit templates
• It expects a detailed English prompt: use the MiniMax API or the local MiniMax-H3-Image-PE (17.9 GB, NVFP4)

⚠️ Caveats
• The card gives no benchmark scores; "strongest instruction following among open-source models" is a vendor claim
• Community license: written authorization is needed above 1 million USD annualized revenue or for hosted services

MiniMaxAI/MiniMax-H3-Image
Comfy-Org/MiniMax-H3-Image
MiniMaxAI/minimax-h3-image

Which route would you take for a local run: ComfyUI with the quantized files, or SGLang, vLLM-Omni or diffusers with the official weights?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/f338de809dcd
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/minimax-h3-image-is-a-16-mp-base
• The thread on X: https://x.com/saavoji/status/2120585989798207554
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#minimax-h3-image

replied to their post 1 day ago

It's usable for chat and light agent work. Decode runs 9 to 15 tok/s on one 16-core Ryzen box. On quality the README reports 98.5 out of 100, but that comes from the author's own gate, so nothing independent yet. Running ./modchip eval on your own machine is the way to check it.

replied to their post 2 days ago

Fair point on the hybrid cores. The README only reports a 16 core Ryzen and nothing on chips with efficiency cores, so I can't say whether you'd land at half speed. ./modchip bench is a quick way to find out on your machine. For iterating, the KV snapshot restore helps with repeated long prompts, but decode stays slow.

posted an update 2 days ago

Post

One frozen encoder now maps 3 tactile sensor types (piezoresistive, magnetic, visuotactile) into the same token space, and the Tiny checkpoint runs on CPU.

LingBot-Touch bets that touch progresses by standardizing the representation, not by building a better sensor. A JEPA-style model uses modality-specific tokenizers and one shared Transformer, predicting masked tactile content in latent space. The authors say the tokens plug into diffusion policy and VLAs such as pi0.5 and LingBot-VLA. The paper has 91 upvotes on HF Daily Papers.

🔍 What's released
• Frozen target encoder plus utilities that output tokens [B,N,C] with time, finger and position addresses
• Tiny and Small checkpoints, Apache-2.0, with 120 sample files for Sharpa and XHand
• Inference only: pretraining code and policy integrations are not described as released

🛠️ Try it
• Run infer.py on the Sharpa or XHand sample with --device cpu
• Read out with cross-attention queries, not mean/max pooling: each token mixes information from other taxels and time steps

⚠️ Caveats
• A new raw taxel layout needs verified geometry calibration; matching tensor shapes is not enough
• The abstract gives no per-task numbers, so check the paper's tables before trusting the gains
• Model files need an account with access to the model repo

Learning Heterogeneous Tactile Representation for General Dexterous Manipulation (2611.12806)
https://github.com/Robbyant/lingbot-touch

Would you put a frozen cross-sensor tactile encoder in front of your policy, or keep per-sensor pipelines?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/0d2028d8fc73
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/lingbot-touch-argues-touch-needs
• The thread on X: https://x.com/saavoji/status/2120209644392194083
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#lingbot-touch

posted an update 3 days ago

Post

3,630

A 284B-parameter DeepSeek-V4-Flash now runs on a 64 GB desktop with 0 GPUs: llama-modchip shrinks the checkpoint to 56.56 GiB (1.71 bpw) and decodes at 9.1 to 14.7 tok/s.

The standard UD-IQ1_S build is 76.87 GiB, above the ~62.5 GiB usable on a 64 GB Linux box. The repo mixes precisions (routed experts mostly IQ1_S, MLA projections in a custom Fractal-64 type) and patches the CPU kernels. The model plus a 256K KV cache then sits at 57.61 to 58.9 GiB RSS with zero swap. The repo was created on 2026-10-29 and already has 744 stars and 470 points on Hacker News.

📊 The numbers
• Size: 56.56 GiB, 20.31 GiB smaller than UD-IQ1_S
• Prefill: 25.0 to 44.36 tok/s (+77%) on a 16-core Ryzen 9 9955HX
• Decode: 5.91 tok/s unpatched, 9.06 to 9.95 tok/s at Top-6, 11.5 to 14.7 tok/s with the n-gram drafter

⚠️ Caveats
• "Lossless" means bit-exact against the same re-quantized checkpoint. Versus UD-IQ1_S the author reports 0.9964 top-logit cosine similarity
• The 98.5/100 quality score comes from the author's own gate, and the headline speeds come from one machine
• The custom tensor type 40 is rejected by stock llama.cpp, Ollama and LM Studio

🛠️ Try it
• Run scripts/check_cpu_readiness.py from the repo: it needs no model download
• Run ./modchip bench for speed, then ./modchip eval to check the quality claims yourself

Repo:
https://github.com/synchopate/llama-modchip

HN thread:
https://news.ycombinator.com/item?id=50411961

Would you accept a custom runtime for 9 to 15 tok/s from a 284B model on a 64 GB box?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/e47ab3581f58
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/llama-modchip-runs-a-284b-deepseek
• The thread on X: https://x.com/saavoji/status/2119858773280141526
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#llama-modchip

8 replies

replied to their post 4 days ago

It looks like a crop setting only. The model card says the mode sets how much room is left around the object, so real leaves more margin than synthetic. Nothing there says it changes the model, and the card gives no training details, so I can't say how the data was split. Glad the game asset worked.

replied to their post 4 days ago

Thanks Jackson, glad the Space worked well for you. The 7.2 GB figure is peak VRAM for a full request, so it fits cards like a 10 GB RTX 3080, though that one takes 46.2 s per image. What kind of objects did you try?

posted an update 4 days ago

Post

121

SLA reaches 72.3% on RoboTwin 2.0 Clean2Random against 46.0% for the pi0.5 it starts from. It does this by aligning VLA visual features to a frozen 3D model in both continual pre-training and post-training.

2D VLAs inherit vision from image-text pretraining. In the authors' depth probe, unaligned VLA embeddings show no meaningful spatial structure. SLA adds a cosine alignment loss to OmniVGGT features: L = L_action + alpha * L_align. The teacher, projector and gates are dropped after training, so inference needs no depth or calibration.

🔍 The recipe
• Gates mix visual embeddings from all VLA layers; state dropout stops the policy shortcutting on robot state
• Data: 6 public datasets, about 1.7M episodes, 14,330 h; delta end-effector actions as rotation vectors (Euler subtraction gives outliers near +/-2pi); episode downsampling; no dexterous hands or base motion
• 36k steps, batch 4,608, about 110 h on 48 H100s

📊 The numbers
• Clean2Random: SLA 72.3, Spatial Forcing (post-training only) 62.6, pi0.5 46.0
• LIBERO-Plus 87.4 vs 84.4; RoboDojo-Real 33.9 vs 12.8
• RoboCasa-Panda 68.4 with 50 demos per task

⚠️ Caveats
• Per-proportion ablation numbers are only in Fig 4; the text says both stages win at every tested proportion
• ME-Dex-1.0 beats it on Clean2Clean (89.6 vs 86.0); the LIBERO-Plus lead over Qwen-RobotManip is 0.3
• The eight real-world conditions are private

SLA: Scaling Spatial-Language-Action Model via Continual Pre-training (2611.08373)
https://arxiv.org/abs/2611.08373

Would you add a training-only 3D alignment loss to your VLA, and in which stage?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/32363a577404
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/sla-aligns-a-vla-to-a-frozen-3d-model
• The thread on X: https://x.com/saavoji/status/2119503826579542064
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#sla-teaches-pi0

posted an update 5 days ago

Post

2,502

Idol, a new Apache-2.0 image-to-3D model from @speridlabs , makes a 3D Gaussian splat from one image in 7.2 GB of peak VRAM. It is the #1 trending model on the Hub with 1,057 likes.

It writes a standard 3DGS .ply with up to 1,048,576 Gaussians, including sides the picture never showed, and does background removal and framing itself. The catch: the only quality evidence is the authors' own benchmark, on a repo created 2026-10-28.

📊 The numbers
• Authors' benchmark, 1 x H100: best in 6 of 8 columns vs TripoSplat, TRELLIS and LGM (SSIM 0.726, LPIPS 0.213, DINOv2 0.781, CLIP 0.885)
• LGM wins PSNR (12.46 vs 12.25) and speed (1.1 s). Idol takes 6.8 s vs 7.1 s for TRELLIS and 9.9 s for TripoSplat
• 262,144 Gaussians, 50 steps: 3.4 s on B300, 6.7 s on H100, 19.1 s on RTX 4090, 46.2 s on a 10 GB RTX 3080

🛠️ Try it
• Install idol-sdk, accept the terms of the gated DINOv3 and FLUX.2-dev models, run hf auth login
• python -m idol.sample --image photo.jpg --mode real
• --num_gaussians runs from 32,768 (0.4 MB) to 1,048,576 (12.2 MB)

⚠️ Caveats
• Test set and size are not named, and no independent evaluation exists yet
• The card says "under 2 seconds", but the timing table shows 3.4 s at best
• Single objects only, NVIDIA CUDA required, no failure cases shown

speridlabs/idol
https://speridlabs.com/research/idol

If you run it on your own messy photos, how do the unseen sides hold up?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/5f660ae6ae56
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/speridlabs-idol-builds-3d-splats
• The thread on X: https://x.com/saavoji/status/2119137739580858545
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#speridlabs-idol

4 replies

posted an update 6 days ago

Post

VFlash cuts per-step latency 5.1 to 5.2x on Jetson Thor and DGX Spark, and its one-pass drafter beats the autoregressive 4B model it drafts for on 3D robot trajectories.

A planner does not have to write points, boxes and trajectories one coordinate token at a time. A primitive's type, dimensionality and length fix its layout before decoding, so a lightweight drafter fills the whole primitive in one forward pass. The decoder accepts it if it is structurally valid, with no token agreement with the target. Text keeps verified speculative decoding.

🔍 Why token matching fails
• DFlash commits 5.5 tokens per iteration on subtask text but only 2.2 inside trajectory blocks, so it only about halves decoding time
• A coordinate one bin off fails the check even when the geometric error is small

📊 The numbers
• AgiBot waypoints within 5 cm: 50% to 73% (3D only) and 40% to 74% (2D+3D); ADE falls 18 to 61% across InternA1, AgiBot and MolmoBot
• Under AR, decoding is 75 to 87% of a step on Thor; VFlash cuts it 12 to 42x. H100 gets 3.6x
• MolmoSpaces success: 70.2% with VFlash plans, 62.6% baseline, 61.0% with AR plans

⚠️ Caveats
• Latency is a pipeline measurement at batch size one, with a 4B target and 36 to 84 token primitives
• Boxes need an AR fallback: without it, mean F1 drops about 20 points on Dense200
• The 25x dense point motion figure is preliminary, and accuracy there is mixed

VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding (2611.08449)
https://arxiv.org/abs/2611.08449

If your planner's cost is mostly coordinate decoding, would you swap token-level verification for a structure validator?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/26ed0ca0bb84
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/vflash-drafts-whole-robot-plans-in
• The thread on X: https://x.com/saavoji/status/2118773042503623062
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#vflash-drafts

posted an update 7 days ago

Post

Cutting an agent's context window from 256K to 16K decodes 3.6x to 7x more tokens per second on identical H100s (GLM 5.2 3.6x, Kimi K2.6 5.6x, Qwen 3 Coder Next 7x), and in the paper's tests accuracy barely moved.

"You don't need long context" (arXiv 2611.05306) argues that long-horizon agents mostly need memory management, not bigger windows. Frequent compaction into a compact state, plus selective reading of inputs with bash tools, kept accuracy close to full-history baselines while contexts stayed short.

📊 The numbers
• Compaction over 6 benchmarks: Opus 4.5 -3.7 to +5.6 ppts, GPT 5.6 Sol -3.3 to +4.4 versus full history
• One action per context: SWE-bench Verified 70.2 for Opus 4.5 (baseline 70.0) and 82.4 for GPT 5.6 Sol (baseline 84.4), median peak context 7K to 9K tokens
• Long inputs stored as files and read selectively: Opus 4.5 -0.4 to +40.0 ppts, estimated cost savings 4.4x to 233.3x across both models

🛠️ Try it
• Set a compaction threshold near 12K to 36K tokens
• Add a tool that reads files or history on demand
• Compare accuracy, total tokens and cost against full history

⚠️ Caveats
• Speedups come from open models, accuracy results from closed ones (Opus 4.5, GPT 5.6 Sol)
• Cost savings are estimates, the harness is bash-only, and the preprint has no independent replication yet

You don't need long context (2611.05306)
https://arxiv.org/abs/2611.05306

What would your agent need to carry in its handoff state, and what could it safely fetch later?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/41f69cdbe3b3
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/qwen-3-coder-next-decodes-7x-faster
• The thread on X: https://x.com/saavoji/status/2118402118927438127
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#claude-opus-4

replied to their post 8 days ago

8 GB is plenty here. At 4-bit LFM3-1.7B takes about 1 GB, so it could fit your Intake or JSON router slot, since the card targets tool calling and structured output. Whether it beats your current picks is untested, the numbers are vendor reported. The card says to keep coding and knowledge-heavy work on a bigger model.

replied to their post 8 days ago

Agreed, the matched row is the fairer comparison, which is why the post says a gap to closed models remains. The paper only reports Astra under RoboProbe and uni-robot-agent. It doesn't report Astra under RPent or GAP, so I can't say whether the harness order holds for both models.

posted an update 8 days ago

Post

All 1,125 turns in eight weight-space experiments with OpenAI's gpt-6-astra are logged and audited as tool-free, and the repo already has 877 stars and 109 forks.

The logs support "weights are an interface worth probing". They do not support "LLMs natively program neural networks". Every prompt, reply, code run and evaluation is published, so you can check where the line falls. The paper has 12,564 alphaXiv visits and 511 votes this week.

🔍 What was tested
• Reading: which game a network was trained on (Exp 1), what a diffusion model generates (Exp 2), which language an LM learned (Exp 3)
• Editing: fix a failing network (Exp 4), improve Qwen3.5-0.8B at math with 10, 25 and 50 passes (Exp 8)
• Writing: MinAtar game networks (Exp 5), a 1.5M-weight Go network (Exp 6)

⚠️ Caveats
• Exps 1, 2, 4, 5 had no tools. Exps 3, 6, 7, 8 used a no-network bubblewrap sandbox, one Python block per turn
• Effort differs: high (1, 2, 4), xhigh (3, 5, 6, 7), max (8), so results are not like-for-like
• One closed model, the authors' own runs, and trial runs of earlier prompts are left out

🛠️ Audit it yourself
• Check audit.tool_free in each turn's run.json and the raw events.jsonl
• Open exp8_math: prompts, tool_calls.jsonl, eval files. Edits ship as exact diffs against the public base weights, with an apply script

https://github.com/RikiyaT/lm_understand-logs

https://huggingface.co/papers/2611.weight-space-programming

Which experiment would you rerun first with an open-weight model?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/3ed2af66a0c9
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/openais-gpt-6-astra-reads-network
• The thread on X: https://x.com/saavoji/status/2118036030452650268
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#openais-gpt-6-10

posted an update 9 days ago

Post

126

Open-weight Qwen3.8-Flash-Next hits a RoboDojo Score of 31.39 with no depth or task-verifier feedback, above GPT-6 Astra's published 28.97 and every listed VLA/WAM baseline.

The lever is the harness. The same Qwen scores 1.90 with RoboProbe, 7.75 with RPent, 21.01 with GAP (Hybrid) and 31.39 with uni-robot-agent. Under that harness GPT-6 Astra reaches 51.90, so a gap to closed models remains.

📊 The numbers
• Qwen costs $1.09 per episode; Astra costs $21.29
• Best VLA/WAM baseline, Xiaomi-Robotics-1: 20.07 Score
• Real robot, 19 tabletop tasks: Qwen 49.50, fine-tuned pi0.5 47.75, zero-shot Dream-1 35.50
• No depth keeps 87.5% of Qwen's score; RGB only without camera parameters drops it to 9.13

🛠️ Try it
• pip install uni-robot-agent, then uni-robot-agent setup --integration robodojo (MIT, Python 3.11, Linux GPU host)
• Reusable ideas: immutable observation snapshots, duplicate-action rejection, pipelined execution (84.5% to 93.8% of motion overlaps model calls)

⚠️ Caveats
• Scores are low in absolute terms; Qwen needs 95.5 min to reach a score of 25, Astra 42.6
• The package has no physical-robot support yet; the real-world benchmark is in-house on one dual-arm platform

Democratizing Robot-Use Agents: A Unified Harness with Open Models (2611.01560)
https://github.com/Open-RUA/uni-robot-agent
https://arxiv.org/abs/2611.01560

Before buying depth sensors or a frontier API, how much of your robot agent's gap is really the harness?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/51efcc243344
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/open-qwen38-flash-next-beats-gpt
• The thread on X: https://x.com/saavoji/status/2117670449111007255
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#qwen3-8-flash

2 replies

replied to their post 10 days ago

That is a fair hit and it matches the card, which says the model is not recommended for agentic coding. Even in Liquid's own table it trails Qwen3.5 9B on SWE-bench Verified, 43.40 against 51.20. The strengths they claim are tool calling and structured output, so I would keep coding on a larger model.

replied to their post 10 days ago

Thanks, the PMI result is a good check. The model card and release post don't say how the BFCL run orders its tools, so I can't rule out position effects. LFM3 writes tool calls as XML function calls rather than picking a letter, which may help, but nothing I have tests that. The card gives no Min-P guidance, only top-k and a repetition penalty.

Sajjad Pakdamansavoji

AI & ML interests

Recent Activity

Organizations

savoji's activity