Jie Jin

Good catch, and it's already how the loop is set up:

1. We only train where the judge is strong. A skill starts only once DuplexJev is above 80% on held-out human-labeled speech for it. The first skill is word stress, where DuplexJev is above 90% on held-out speakers. For some skills we use a specialist judge: a DuplexJev trained for that one task, which scores higher than the balanced release.
2. No forced winner. A candidate is kept only if its target probability clears an absolute floor and beats the group median by a margin. If no candidate does, the item contributes nothing that round.
3. People check it before every release. Before a version ships, blind human ratings compare it with the previous one, alongside DuplexJev-vs-human agreement on sophon audio and on real speech. If the judge's picks don't hold up with people on generated audio, it shows up there.

Fair point on the denominator: on the unseen LISTEN corpora v1.4 gave 63 off-four answers out of 107 (12 correct); on the training corpora, 3 of 93.

On the reward side, the setup is built so that path doesn't pay. Expression is trained one skill at a time (stress, each emotion, laughter …), and every training item names its target up front ("say this fearfully", "stress this word", "laugh here"). We sample several candidates for the same item and DuplexJev only ranks them against that target, so the reward is relative within the group, not DuplexJev deciding on its own that a clip is "fear". And a label only enters the reward after DuplexJev is shown to judge it well on real human speech; if it doesn't, we train the ear first. It's the same philosophy as the input side: learn from the correct answer. The DuplexJev-vs-human agreement will also be reported per label, so a label where the judge is unreliable shows up.

Fair point, and thank you, that's exactly the split to make. The plan has three layers, so DuplexJev is never the final judge of its own student:

1. Training: DuplexJev's judgments are the reward signal for reinforcement learning.
2. Checkpoints: progress is checked on purpose-built test sets with fixed answers (held-out items, never trained on), not on DuplexJev's opinion of the output.
3. Final evaluation: human ratings are the headline, scored blind: raters don't see DuplexJev's judgments or which system made a clip. Next to them we'll report how often DuplexJev agrees with the human raters, on sophon audio and on real speech, so if the student found the teacher's blind spots it shows up as a drop on sophon audio.

On the 10-class report I promised: v1.4 (Oct 22) added surprise, fear, anxiety, excitement and more. It now uses those labels and gets more of them right (on the unseen LISTEN corpora, correct answers outside the basic four went from 3 to 12), but some other emotion items slipped a little, so we're still looking for the best recipe there. v1.4's real gains are in languages: Japanese error 32.7% → 11.6% (FLEURS), Korean 31.8% → 7.6%, Vietnamese 41.7% → 14.6%.

That's a sharp read, and a better framing than mine. Thank you! "Never trained" is right; "never reached" isn't. The model reserves the untrained labels for audio unlike anything it was trained on: 0 of 93 on its training corpora, 24 of 107 on unseen ones. That's a free out-of-distribution signal, and it fits how we use DuplexJev anyway: every answer comes with its probability, so an agent can already treat a low-confidence or off-four answer as "not sure, don't act on it".

Agreed on the 10-class report. We'll publish off-vocabulary answers on seen vs. unseen corpora, so it's clear whether the gain is hearing or just vocabulary.

On sophon-realtime, a quick preview, since it's the other half of this story. It's an end-to-end spoken dialogue model, and DuplexJev is its paralinguistic teacher: we use DuplexJev's judgments as the training signal for expression, so the model doesn't just say the words but laughs, coughs and clears its throat where a person naturally would, with the tone to match. If DuplexJev is about hearing how something is said, sophon-realtime is about saying it that way. Our goal is state of the art in paralinguistic expression.

For the duplex side: both. Turn-taking and barge-in on Full-Duplex-Bench (pause handling, turn-taking latency, user interruption, backchannels) with the public scripts, plus expression results judged by DuplexJev and by people, and an end-to-end latency breakdown. Answer files included, same as here, so you can take it apart again.

You're right again, and thank you! Two reviews in, you've made our benchmark reporting better than when we posted it.

To your question: v1.3's emotion supervision stops at those four labels (angry, happy, neutral, sad). Fear, disgust, surprise and calm were never answer targets, so nothing is lost at readout; the model was simply never taught them. Your split shows it cleanly:

• On the four labels it was trained on: 83/116 (72%), vs 46 for Qwen3-Omni and 39 for Gemini 3.8 Flash.
• Same four labels, unseen corpora only (TESS, RAVDESS, SAVEE): 29/54, vs 22 and 16.
• On labels it never saw: 3/84. That's the gap to close.

Expanding to 10 emotion classes is scheduled for next week. When it ships, we'll report LISTEN neutral-text your way: trained labels vs. the rest.

And good catch on IEMOCAP. PARA alone is 26.0 vs 25.0 for GPT-audio-1.5, and on the 51 EM clips Qwen3-Omni and GPT-audio lead. The README reports per-condition numbers, so no pooled figure there.

One more thing to watch for this week: we're preparing to open-source sophon-realtime, our full-duplex conversational model, with DuplexJev as its ears. Would love your eyes on it too.

This is a fantastic review, thank you! You reproduced the whole pipeline from our scripts and then went further than we did. That's exactly why we published every model's raw answers, and it's great to see someone put them to work.

You're right that the headline should stand on clips we never trained on, so we're adopting your split as the main number. On those 1,035 items, DuplexJev still comes out on top:

• DuplexJev-32B-Para v1.3: 50.2
• Gemini 3.8 Flash: 47.8
• Qwen3-Omni-30B: 45.0

Your split also shows how broad the lead is. Of the 11 corpora in the 1,035, DuplexJev is first (or tied first) on 8, with big margins where only the voice carries the answer:

• TESS (emotion with neutral words): 38.3 vs 17.0 Flash / 25.5 Qwen
• MSPB (prosody): 77.3 vs 67.2 / 71.9
• IEMOCAP non-verbal (laughs, sighs, breaths): 27.1 vs 21.1 / 21.1

And as your bootstrap shows, the 5.2-point lead over Qwen3-Omni holds firmly (95% CI 2.2 to 8.2).

MELD is a great catch. It's TV dialogue where the words themselves carry most of the emotion, the one setting where models that read the words have the edge. We're adding conversational dialogue emotion data in our next training round and will report MELD separately so everyone can follow the progress.

We'll update the README and scripts to report the 1,035 split as the headline, with the trained-corpus items listed separately. Thanks again, reviews like this make the work better!

Good idea, so we just ran it: the answers don't depend on option position. Same clips, options shuffled, letters mapped back to labels (DuplexJev-32B-Para v1.3, one order per request, no averaging):

• 29 clips; 4-way emotion and 4-way turn state with 15 distinct option orders each, gender with both orders.
• 98.5% of answers are identical across all orders (emotion 686/696, turn 685/696, gender 58/58).
• When the model is confident (≥ 0.8), zero flips, and the winning label's probability barely moves (median change 0.01 for emotion, 0.04 for turn).
• The few flips all come from 2 genuinely ambiguous clips per question, where the model was only 0.55–0.67 sure to begin with.

For benchmarks we still average over a few orders, which settles even those borderline clips (29/29 match) at almost no cost, since the orders share the audio prefix. We'll run the same check on the full benchmark set and add it to the repo. Thanks for the suggestion!