Instructions to use moonbox-dev/stable-audio-3-medium-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use moonbox-dev/stable-audio-3-medium-MLX with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download moonbox-dev/stable-audio-3-medium-MLX --local-dir stable-audio-3-medium-MLX
- Stable Audio 3
How to use moonbox-dev/stable-audio-3-medium-MLX with Stable Audio 3:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Stable Audio 3 Medium MLX
MLX builds of Stable Audio 3 Medium, made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's files together. Only the parts listed below are converted: the rest of the pipeline comes from the original repository unless the section below says otherwise.
Files
The last column is how far each build is from the original PyTorch model on the same inputs (see the section below for what was measured).
| Precision | File | Size | sha256 | Checked against the original |
|---|---|---|---|---|
| 8bit | 8bit/conditioner.safetensors |
0.8 MB | 21ceba16b12f |
first step 0.008-0.025, latents cosine >= 0.995, log-mel 0.26-0.82 dB, SNR 17-30 dB, CLAP 0.972-0.990 vs fp16, prompt match 0.61 |
| 8bit | 8bit/config.json |
0.8 kB | aaa946aa7833 |
first step 0.008-0.025, latents cosine >= 0.995, log-mel 0.26-0.82 dB, SNR 17-30 dB, CLAP 0.972-0.990 vs fp16, prompt match 0.61 |
| 8bit | 8bit/decoder.safetensors |
1.70 GB | 7bc0f3e1c345 |
first step 0.008-0.025, latents cosine >= 0.995, log-mel 0.26-0.82 dB, SNR 17-30 dB, CLAP 0.972-0.990 vs fp16, prompt match 0.61 |
| 8bit | 8bit/dit.safetensors |
1.63 GB | 1d924a73294c |
first step 0.008-0.025, latents cosine >= 0.995, log-mel 0.26-0.82 dB, SNR 17-30 dB, CLAP 0.972-0.990 vs fp16, prompt match 0.61 |
| 8bit | 8bit/mlx_model.py |
40.0 kB | 839674ae00d1 |
first step 0.008-0.025, latents cosine >= 0.995, log-mel 0.26-0.82 dB, SNR 17-30 dB, CLAP 0.972-0.990 vs fp16, prompt match 0.61 |
| 8bit | 8bit/text_encoder.safetensors |
563.2 MB | 4e13c975e6dc |
first step 0.008-0.025, latents cosine >= 0.995, log-mel 0.26-0.82 dB, SNR 17-30 dB, CLAP 0.972-0.990 vs fp16, prompt match 0.61 |
| 8bit | 8bit/tokenizer.model |
4.2 MB | 61a7b147390c |
first step 0.008-0.025, latents cosine >= 0.995, log-mel 0.26-0.82 dB, SNR 17-30 dB, CLAP 0.972-0.990 vs fp16, prompt match 0.61 |
| 6bit | 6bit/conditioner.safetensors |
0.8 MB | 21ceba16b12f |
first step 0.023-0.088, latents cosine >= 0.963, log-mel 0.60-1.65 dB, SNR 8-20 dB, CLAP 0.943-0.988 vs fp16, prompt match 0.61 |
| 6bit | 6bit/config.json |
0.8 kB | 7d0769f28af0 |
first step 0.023-0.088, latents cosine >= 0.963, log-mel 0.60-1.65 dB, SNR 8-20 dB, CLAP 0.943-0.988 vs fp16, prompt match 0.61 |
| 6bit | 6bit/decoder.safetensors |
1.70 GB | 7bc0f3e1c345 |
first step 0.023-0.088, latents cosine >= 0.963, log-mel 0.60-1.65 dB, SNR 8-20 dB, CLAP 0.943-0.988 vs fp16, prompt match 0.61 |
| 6bit | 6bit/dit.safetensors |
1.29 GB | 74b2ccd30f94 |
first step 0.023-0.088, latents cosine >= 0.963, log-mel 0.60-1.65 dB, SNR 8-20 dB, CLAP 0.943-0.988 vs fp16, prompt match 0.61 |
| 6bit | 6bit/mlx_model.py |
40.0 kB | 839674ae00d1 |
first step 0.023-0.088, latents cosine >= 0.963, log-mel 0.60-1.65 dB, SNR 8-20 dB, CLAP 0.943-0.988 vs fp16, prompt match 0.61 |
| 6bit | 6bit/text_encoder.safetensors |
563.2 MB | 4e13c975e6dc |
first step 0.023-0.088, latents cosine >= 0.963, log-mel 0.60-1.65 dB, SNR 8-20 dB, CLAP 0.943-0.988 vs fp16, prompt match 0.61 |
| 6bit | 6bit/tokenizer.model |
4.2 MB | 61a7b147390c |
first step 0.023-0.088, latents cosine >= 0.963, log-mel 0.60-1.65 dB, SNR 8-20 dB, CLAP 0.943-0.988 vs fp16, prompt match 0.61 |
| 4bit | 4bit/conditioner.safetensors |
0.8 MB | 21ceba16b12f |
below threshold (experimental): first step 0.13-0.37, latents cosine 0.82-0.97, log-mel 2.1-3.9 dB, SNR 1-9 dB, CLAP 0.967-0.986 vs fp16, prompt match 0.62 |
| 4bit | 4bit/config.json |
0.8 kB | a5ccbeff6063 |
below threshold (experimental): first step 0.13-0.37, latents cosine 0.82-0.97, log-mel 2.1-3.9 dB, SNR 1-9 dB, CLAP 0.967-0.986 vs fp16, prompt match 0.62 |
| 4bit | 4bit/decoder.safetensors |
1.70 GB | 7bc0f3e1c345 |
below threshold (experimental): first step 0.13-0.37, latents cosine 0.82-0.97, log-mel 2.1-3.9 dB, SNR 1-9 dB, CLAP 0.967-0.986 vs fp16, prompt match 0.62 |
| 4bit | 4bit/dit.safetensors |
952.9 MB | 528b3c10d818 |
below threshold (experimental): first step 0.13-0.37, latents cosine 0.82-0.97, log-mel 2.1-3.9 dB, SNR 1-9 dB, CLAP 0.967-0.986 vs fp16, prompt match 0.62 |
| 4bit | 4bit/mlx_model.py |
40.0 kB | 839674ae00d1 |
below threshold (experimental): first step 0.13-0.37, latents cosine 0.82-0.97, log-mel 2.1-3.9 dB, SNR 1-9 dB, CLAP 0.967-0.986 vs fp16, prompt match 0.62 |
| 4bit | 4bit/text_encoder.safetensors |
563.2 MB | 4e13c975e6dc |
below threshold (experimental): first step 0.13-0.37, latents cosine 0.82-0.97, log-mel 2.1-3.9 dB, SNR 1-9 dB, CLAP 0.967-0.986 vs fp16, prompt match 0.62 |
| 4bit | 4bit/tokenizer.model |
4.2 MB | 61a7b147390c |
below threshold (experimental): first step 0.13-0.37, latents cosine 0.82-0.97, log-mel 2.1-3.9 dB, SNR 1-9 dB, CLAP 0.967-0.986 vs fp16, prompt match 0.62 |
How to run
Powered by Stability AI. Stable Audio 3 Medium makes instrumental music, sound beds and sound effects from a description: 44.1 kHz stereo, up to 380 seconds, in 8 steps. It has no lyrics input.
Each folder is a whole pipeline for mlx_model.py, the one-file MLX runtime that sits in the folder (mlx, numpy,
sentencepiece; no PyTorch):
| File | What |
|---|---|
dit.safetensors |
the diffusion transformer (1.45 B parameters); its attention and feed-forward layers MLX affine-quantized to 8, 6 or 4 bits, group size 64, the rest float16 |
decoder.safetensors |
the SAME-L decoder, latents to audio (float32); the same file in every folder |
text_encoder.safetensors, tokenizer.model |
the encoder half of T5Gemma b-b-ul2 (float16) and its vocabulary; the same in every folder |
conditioner.safetensors |
the prompt padding embedding and the seconds embedder |
config.json, mlx_model.py |
sizes and quantization; the runtime |
pip install mlx numpy sentencepiece soundfile
hf download moonbox-dev/stable-audio-3-medium-MLX --include "8bit/*" --local-dir sa3
import sys, soundfile as sf
sys.path.insert(0, "sa3/8bit")
from mlx_model import Pipeline
pipe = Pipeline("sa3/8bit") # one part in memory at a time: text encoder, transformer, decoder
audio, info = pipe.generate("Warm lofi hip hop with dusty drums, mellow Rhodes piano and deep bass, 85 BPM",
seconds=30, seed=11)
sf.write("lofi.wav", audio.T, pipe.sample_rate)
On an Apple M5 a 20-second clip takes about 3 seconds and 4 GB of memory. The model code is
mlx-audio's stable_audio_3 (MIT); the weights are Stability AI's own MLX
export (stabilityai/stable-audio-3-optimized), renamed
and, for the transformer, quantized. There is no fp16 folder here: that is Stability's export itself.
How they were checked. Each build against the fp16 export in the same runtime from the same noise: 4 clips
(lofi, orchestral, folk, rain; 12-47 s, seed 11). "First step" is the relative error of the transformer's first
output; "log-mel" the mean difference of the two clips' log-mel spectrograms; CLAP
(laion/clap-htsat-unfused) scores a clip against the fp16 clip and against its prompt. For scale, a different seed
of the fp16 build scores log-mel 3.8-10.2 dB and CLAP 0.90-0.95 against the first, and the same file scores CLAP
0.98-0.99 against itself. Pass: first step <= 0.10 and log-mel <= 2.0 dB on every clip. The export itself wasn't
compared with a PyTorch run.
| Precision | Size (transformer) | First step | Log-mel vs fp16 | CLAP vs fp16 / vs prompt | Status |
|---|---|---|---|---|---|
| fp16 export | 5.18 GB (2.91) | reference | - | 0.98-0.99 / 0.61 | Stability's export, not in this repository |
8bit/ |
3.90 GB (1.63) | 0.008-0.025 | 0.26-0.82 dB | 0.972-0.990 / 0.61 | passes: the same clip |
6bit/ |
3.57 GB (1.29) | 0.023-0.088 | 0.60-1.65 dB | 0.943-0.988 / 0.61 | passes: small differences |
4bit/ |
3.23 GB (0.95) | 0.13-0.37 | 2.1-3.9 dB | 0.967-0.986 / 0.62 | below threshold, experimental: a different rendition of the prompt, same character |
The decoder's output can pass full scale on loud material; generate() clips it to [-1, 1] (clip=False returns it
as decoded).
How these were made
From Stable Audio 3 Medium at revision 30cb2644da: the original weights loaded in MLX and stored as bf16 or fp16, or
quantized with MLX's affine quantization (8bit, 6bit, 4bit, group size 64; layers MLX can't quantize, such as
convolutions and norms, stay bf16). Every build was checked against the original before upload; the sha256 column
above is what the files must hash to.
License
Stability AI Community License (text encoder: Gemma Terms of Use). These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.
Original model card
Stable Audio 3 Optimized
Note: This repository contains experimental checkpoints optimised for acceleration on specific hardware. For standard checkpoints, please use Stable Audio 3 Medium instead.
Please note: For commercial use, please refer to https://stability.ai/license
Model Description
Stable Audio 3 is a family of fast latent diffusion models (small, medium, large) for variable length audio generation and editing. Since our models can generate several minutes of audio,
variable-length generations are key to avoid the cost of producing full-length generations for short
sounds. We also support inpainting, enabling targeted audio editing and the continuation of short
recordings. Our latent diffusion models operate on top of a novel semantic-acoustic autoencoder that
projects audio into a compact latent space, enabling efficient diffusion-based generation while preserving audio fidelity and encouraging semantic structure in the latent. Finally, we run adversarial
post-training to both accelerate inference and improve generation quality, reducing the number of inference steps while improving fidelity and prompt adherence. Stable Audio 3 models are trained on
licensed and Creative Commons data to generate music and sounds in less than a 2s on an H200 GPU
and less than a few seconds on a MacBook Pro M4. We release the weights of small and medium,
that can run on consumer-grade hardware, together with their training and inference pipeline.
Usage
This model can be used with:
- the
stable-audio-3inference and fine-tuning library - the
stable-audio-toolsresearch library
Using with stable-audio-3
from stable_audio_3 import StableAudioModel
model = StableAudioModel.from_pretrained("medium")
audio = model.generate(
prompt=(
"House music that encapsulates the feeling of being at a festival "
"in the sunny weather with all your friends 124 BPM"
),
duration=180
)
Using with stable-audio-tools
import torch
import torchaudio
from einops import rearrange
from stable_audio_tools import get_pretrained_model
from stable_audio_tools.inference.generation import generate_diffusion_cond_inpaint
device = "cuda" if torch.cuda.is_available() else "cpu"
if device == "cuda":
model_half = True
# Download model
model, model_config = get_pretrained_model("stabilityai/stable-audio-3-medium")
sample_rate = model_config["sample_rate"]
sample_size = model_config["sample_size"]
model = model.to(device)
if model_half:
model = model.to(torch.float16)
# Set up text and timing conditioning
conditioning = [{
"prompt": (
"A dream-like Synthpop instrumental that would accompany "
"a dream-sequence in a surrealist movie 120 BPM"
),
"seconds_total": 380
}]
# Generate stereo audio
output = generate_diffusion_cond_inpaint(
model,
steps=8,
cfg_scale=1.0,
conditioning=conditioning,
sample_size=sample_size,
sampler_type="pingpong",
device=device
)
# Rearrange audio batch to a single sequence
output = rearrange(output, "b d n -> d (b n)")
# Peak normalize, clip, convert to int16, and save to file
output = output.to(torch.float32).div(torch.max(torch.abs(output))).clamp(-1, 1).mul(32767).to(torch.int16).cpu()
torchaudio.save("output.wav", output, sample_rate)
Optimized CPU runtime (TFLite / LiteRT)
Portable CPU builds live in optimized/tflite. The SAME autoencoder ships as static rung .tflite codecs at tflite/same-{s,l}/{enc,dec}_{fp32,w8a8}.tflite — one weight-shared file per encoder/decoder that dispatches a fixed-size subgraph per length, keeping RAM flat and avoiding a chunk-size knob. They require ai_edge_litert>=2.2.0 and are driven by the RungEncoder / RungDecoder runtime in that directory (w8a8 is the default; fp32 is the bit-exact reference). The earlier dynamic-length codecs are preserved under tflite/same-*/legacy/.
Runtime LoRA on TensorRT (NVIDIA)
dit_fp16_lora.trt is a DiT with a runtime low-rank branch on each adapted linear. All
three DiTs have one:
| engine | DiT | targets | size |
|---|---|---|---|
tensorRT/sm_90/sa3-m/dit_fp16_lora.trt |
medium, 24 × 1536 | 229 | 2.96 GB |
tensorRT/sm_90/sa3-sm-music/dit_fp16_lora.trt |
small-music, 20 × 1024 | 193 | 0.95 GB |
tensorRT/sm_90/sa3-sm-sfx/dit_fp16_lora.trt |
small-sfx, 20 × 1024 | 193 | 0.95 GB |
A TensorRT plan has its weights compiled in, so an adapter cannot be merged into it the way
the CPU runtime does; these engines instead take the adapter as network inputs — A, Bt
and a per-output-row rescale P are fed in like activations, with the rank as a dynamic
dimension — and compute y = W₀·x·(1+P) + Btᵀ·(srow ⊙ (A·x)).
So loading an adapter is a buffer write rather than a rebuild: adapters swap in milliseconds,
several stack by concatenating along the rank axis (up to a total of 512), and srow gives
each one its own strength. P is what DoRA needs — its per-row magnitude rescale is a change
to the base path that no low-rank branch can express.
The branch costs a flat few ms per step, so its relative price depends on which DiT it is
attached to. On medium that is +13% at L=4096 and +37% at 1292; on the small DiTs the same
absolute cost lands on a base step about 2.7× cheaper, so at stack rank 16 on an H200 it is
+34% at L=4096, +60% at 1292 and +83% at 323. It is paid even with an empty stack — in fact
rank 1 is the most expensive configuration, since TensorRT picks a worse tactic below rank 8
— so use dit_fp16.trt until an adapter is actually wanted. That matters more on the small
DiTs than on medium.
Adapters are not portable between the three DiTs: different depth and width, so a medium adapter will not load against a small engine.
sm_90 only; other architectures build locally in ~5–8 min, since TensorRT bakes the GPU
architecture into the plan. The runtime, the adapter row-norm baking step and the layer maps
all live in
optimized/tensorRT/scripts/lora
— this repository holds only the compiled plans.
Model Details
- Model type:
Stable Audio 3is a latent diffusion model based on a transformer architecture. - Language(s): English
- License: Stability AI Community License.
- Research Paper: https://arxiv.org/abs/2605.17991
We use a publicly available pre-trained T5Gemma model (t5gemma-b-b-ul2) for text conditioning. T5Gemma is redistributed under the Gemma Terms of Use.
Training dataset
Datasets Used
Our dataset consists of 1,278,902 audio recordings, where 806,284 recordings are licensed from AudioSparx and a further 472,618 are from Freesound. The Freesound portion consists of recordings licensed under CC-0, CC-BY, or CCSampling+. To ensure no copyrighted content was present in the Freesound data, music recordings were identified using the PANNs [89] tagger. We flagged audio that activated music-related tags for at least 30s (threshold of 0.15), that was sent to a trusted content detection company to verify the absence of copyrighted material. All identified copyrighted content was removed. After filtering, the Freesound part includes 266,324 CC-0, 194,840 CC-BY, and 11,454 CC-Sampling+ recordings. The same subset of Freesound audio we used to train Stable Audio Open: https://info.stability.ai/attributions.
Quantized
Model tree for moonbox-dev/stable-audio-3-medium-MLX
Base model
stabilityai/stable-audio-3-medium-base