BEST-RQ-2

Recommended: BEST-RQ-2.2-base-2M has the highest reported overall X-ARES scores in this comparison (2M training steps, overlap50 evaluation).

BEST-RQ-2 is a self-supervised audio encoder trained on AudioSet for a configured budget of 200,000 optimizer steps. It produces 768-dimensional clip and frame embeddings from mono 16 kHz waveforms, and supports downstream fine-tuning.

This repository contains the trained encoder, its preprocessing configuration, and the custom Transformers implementation. No installation of the research repository is needed.

X-ARES results

Scores on X-ARES (0–100, higher is better). Audio-JEPA, BEST-RQ (Conformer), BEST-RQ (ViT), and the original BEST-RQ-2 variants below use the same AudioSet split for 200,000 steps. BEST-RQ-2.2 (2M) uses 2,000,000 steps, with batch size 128 per GPU on two GPUs (effective batch 256). The pretrained baselines are shown for comparison.

Model	Speech	Music	Environment	Global Mean	Mean of Means	Hugging Face model
data2vec	50.62	23.24	15.41	37.83	29.76	data2vec-audio-base
wav2vec 2.0	41.79	34.94	29.52	37.84	35.42	wav2vec2-large-100k-voxpopuli
Whisper	49.19	38.67	28.61	42.75	38.82	whisper-base
Audio-JEPA	29.64	44.27	25.61	31.18	33.17	Audio-JEPA-base
BEST-RQ (Conformer)	40.43	35.58	30.81	37.43	35.60	Separate codebase (recipe unavailable)
BEST-RQ (ViT)	32.87	41.62	34.50	34.88	36.33	BEST-RQ-ViT
BEST-RQ-2 (Interspeech 2026)	38.49	54.40	46.39	43.21	46.43	BEST-RQ-2
BEST-RQ-2.1	52.60	62.23	53.38	54.59	56.07	BEST-RQ-2.1-base
BEST-RQ-2.2	53.78	63.90	55.71	56.11	57.80	BEST-RQ-2.2-base
BEST-RQ-2.2 (2M)	58.62	65.74	61.80	58.77	62.05	BEST-RQ-2.2-base-2M
BEST-RQ-2.2 (2M, overlap50)	60.87	65.78	65.90	61.13	64.18	BEST-RQ-2.2-base-2M

Global Mean averages all benchmark task scores. Mean of Means gives equal weight to the Speech, Music, and Environment category means.

The two 2M rows report the same training checkpoint with different extraction settings: original evaluation and 50% overlapping windows. Their scores use the logged Combined metric: equal weight for the linear-probe and k-NN aggregates, excluding unavailable tasks separately for each method. The earlier rows retain previously reported scores. Extraction and evaluation settings differ across runs, so this is not a controlled training-duration ablation.

Audio-JEPA scores were supplied by the author for run jp6l70l6. The other 200k-model and pretrained-baseline scores are reported in the project README. These are reported research results, not a new benchmark run of the Transformers exports.

The Audio-JEPA row refers to the newer 16 kHz, 200,000-step run, not the original ICME model (32 kHz, 100,000 steps). The author reports better results for this newer checkpoint.

HEAR results

Scores on HEAR (0–100, higher is better), using overlap50_two_phase extraction. All five project models cover all 19 tasks. BEST-RQ-2.2 (2M) was trained for 2,000,000 steps; the other rows use 200,000 steps.

Model	Speech	Music	Environment	Global Mean	Mean of Means	Coverage
Audio-JEPA	36.53	74.20	51.85	56.42	54.19	19/19
BEST-RQ-2	50.74	79.46	69.21	67.69	66.47	19/19
BEST-RQ-2.1	64.74	86.47	70.83	75.50	74.02	19/19
BEST-RQ-2.2	65.76	85.13	76.19	76.66	75.69	19/19
BEST-RQ-2.2 (2M)	70.18	85.38	81.82	79.64	79.13	19/19

The 2M scores come from this W&B evaluation; its checkpoint hash matches the released 2M checkpoint. The earlier rows retain the reported 2026-09-19 W&B comparison. Score sources and task-level 2M results. These are arithmetic means of primary task scores, with equal task weighting for Global Mean and equal domain weighting for Mean of Means; they are not the challenge's standardized ranking. These evaluations precede the Transformers export.

Model and training

Property	Value
Architecture	12-layer Transformer, 768 dimensions, 12 attention heads
Input frontend	128-bin mel spectrogram with a linear patch projection
Patch shape	16 mel bins by 16 time frames
Transformer	Sinusoidal positional embeddings, LayerNorm, GELU MLP
Training data	AudioSet
Training objective	Masked prediction of frozen codebook targets
Masking ratio	40–60%
Checkpoint step metadata	Not recorded in this older safetensors file
Exported weight dtype	float32
Extraction policy	`overlap50_two_phase`

Load the model

Install the runtime dependencies in your Python environment:

pip install "torch>=2.9.1" "torchaudio>=2.9.1" "timm>=0.9" "einops>=0.7" "transformers>=4.57,<6"

Use matching PyTorch and torchaudio versions. GPU installations may require the appropriate PyTorch build for your CUDA version.

import torch
from transformers import AutoFeatureExtractor, AutoModel

model_name = "ltuncay/BEST-RQ-2"
processor = AutoFeatureExtractor.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()
audio = torch.zeros(processor.sampling_rate)  # Replace with real mono audio.
inputs = processor(audio, sampling_rate=processor.sampling_rate, return_tensors="pt")
with torch.inference_mode():
    outputs = model(**inputs)
clip_embeddings = outputs.pooler_output
frame_embeddings = outputs.last_hidden_state
output_dim = model.config.encoder_kwargs["embed_dim"]

Resample to processor.sampling_rate and downmix stereo before preprocessing. The extractor performs padding only; spectrogram/convolution features are computed inside the model. Pass the returned sample attention mask for variable durations. Outputs include a frame attention mask and timestamps in milliseconds (-1 for padding). pooler_output uses the saved HEAR extraction preset, including phase-balanced pooling. Frame features average frequency patches; they are not the raw frequency-time ViT grid.

Fine-tuning

For fine-tuning call model.train(), attach a task head and optimize its parameters alongside the model. Save with model.save_pretrained(path) and processor.save_pretrained(path). This encoder export excludes pretraining predictors, quantizers, teachers and optimizer state. Continue self-supervised research training with the original Lightning code and checkpoints.

Reproducibility and provenance

revision is optional; pin both loaders to the same full commit hash for reproducibility. Custom Python code is included in this repository and requires trust_remote_code=True. The weights use safetensors. See export_manifest.json for source and validation details. Source: the existing AECC 2026 submission in this repository, pinned at a03eeb5c4433f4bf7a7e6e8b4724af862789959c. The export uses its BEST-RQ-2.safetensors and matching config.yaml. The saved configuration specifies a 200,000-step training budget, but this older safetensors file has no saved-step metadata. The actual checkpoint step therefore remains unspecified in the export manifest. Source file hashes are recorded there.

AECC 2026 compatibility

The original submission is preserved at revision aecc-2026-submission, including its model card, checkpoint, technical report and xares-llm integration. The legacy files also remain available on main for existing consumers.

This Transformers export uses the same pretrained encoder weights. Its public outputs follow the shared BEST-RQ-2 family extraction policy: frequency-averaged frame embeddings and phase-balanced clip embeddings with overlapping windows. The original AECC wrapper returned spectrogram patch tokens with sequential audio chunking, so the two public interfaces are not interchangeable for reproducing benchmark results. Use the archived revision for the original evaluation setup.

A comparison against the archived frontend and encoder on a 10-second waveform found exact spectrogram and patch-projection agreement, with maximum encoder absolute difference 1.67e-6 (within floating-point tolerance). The validation manifest records this comparison.

The exported model matched the pre-export encoder exactly on the exporter's variable-duration test batch. These are integration checks, not a downstream benchmark evaluation of this release.

Intended uses and limitations

Use these embeddings as features for audio research, classification, retrieval, or downstream fine-tuning. This is an encoder: it does not generate transcripts, audio, or class labels without a downstream model.

AudioSet training does not establish performance on every language, acoustic domain, demographic group, or downstream task. Evaluate the model on your target data. No new downstream benchmark scores are claimed for this export.

The feature extractor pads waveforms and checks the sample rate; it does not resample or downmix. Unequal-length clips are processed individually after padding is removed. The output embeddings follow the saved windowing policy rather than exposing the raw spectrogram token grid.

Browse the BEST-RQ-2 family collection.

Source and citation

Research code: audio-embeddings. BEST-RQ-2 belongs to the BEST-RQ-2 family. Cite the family paper when using it:

@inproceedings{tuncay2026best,
  title={BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations},
  author={Tuncay, Ludovic K and Labb{\'e}, Etienne and Pellegrini, Thomas},
  booktitle={Interspeech 2026},
  year={2026}
}