Jonna Matthiesen

Good points! The order is a bit project-dependent though, and where the interest is strongest. From my point of view, Qwen3-1.7B-W4A16 is a bit more interesting as I see a bit more use around it and it can move an actual deploy decision.

But it is not necessary about which accuracy row lands next, but which model x hardware combination we optimize next. Being able to replace an existing deployed model fast with a newer, better one is quite a strong argument, or moving to a cheaper hardware.

In terms of what models may come next, I could see VLAs becoming more interesting going forward since low-latency real-time deployment scenarios are key there.
For example, we recently did a project focusing on VLA (π0.5) optimization on AMD. Some of this was in the AMD keynote last week. We did the GPU VLA inference-latency optimization behind their sub-100ms robot-reasoning number on the Kria AI SOM.

And FYI, missing accuracy benchmarks for the non-LLM/VLM models have been added now. 😊

Some models (like https://huggingface.co/embedl/sam3) have more benchmarks in the model page - will push for updating the Benchmarks page with those results.

Yes, accuracy numbers are a bit thin! For FlashHead you are correct. We have written a paper on FlashHead (see https://arxiv.org/abs/2603.14591) where we show that it is consistently accuracy-preserving: typically within ~99.9% top-1 containment.

For the GenAI models on Jetson: We are actually using the TensorRT vLLM Docker image, so lm-eval has a path and is not blocked by the TRT engine. For example, for Qwen3.6 we have internal accuracy evaluations but not yet pushed.

For the more classical DL models, the benchmark suite mostly compares to quantized variants where we do need to push for more on-device accuracy evaluations.

The notion of picking the largest model that fits the latency (and memory) budget is quite common and that's one focus of our GenAI models . On the classical DL side it is a bit different, edge deployment use-cases are often around making an existing, well performing model faster.

Any plans on releasing flashhead for qwen3.5 models?

Yes! FlashHead-enabled Qwen3.5 models are coming soon. We are currently finalizing accuracy and latency evaluations.

Check it out here: https://github.com/JonnaMat/huggingface-slack-app

🤓 Want to learn more about FlashHead? Check out this blog post: https://huggingface.co/blog/JonnaMat/flashhead

Additional Jetson Orin Nano Super benchmarks will be added continuously.