Low generation speed.
#36
by Yurkoff - opened
Generation speed for MiniMax-M2 :14 tokens/sec.
minimax-m2 | (APIServer pid=1) INFO 11-07 09:22:55 [loggers.py:221] Engine 000: Avg prompt throughput: 1139.4 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:23:05 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:23:15 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:23:25 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:23:35 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.0%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:23:45 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:23:55 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:24:05 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.3%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:24:15 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.4%, Prefix cache hit rate: 16.5%
minimax-m2 | (APIServer pid=1) INFO 11-07 09:24:25 [loggers.py:221] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.4%, Prefix cache hit rate: 16.5%
Generation speed for Qwen3-235B-A22B-Instruct-2507: 24 tokens/sec.
Inference was performed on 4 x H100 GPUs. In both cases, "chat_template_kwargs": {"enable_thinking": False} was used.
It seems that adding the parameter "chat_template_kwargs": {"enable_thinking": False} does not disable reasoning. Each model response begins with the <think> tag.
This raises an additional question: how can I disable reasoning during inference?
The parameters I used to run VLLM:
--gpu-memory-utilization 0.90 \
--max-model-len 32768 \
--tensor-parallel-size 4 \
--trust-remote-code \
--enable-auto-tool-choice \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2_append_think
For optimal performance on H100 and H20 GPUs with this MoE model, we recommend using DeepGEMM for linear layers while disabling it for MoE expert layers.
https://docs.vllm.ai/projects/recipes/en/latest/MiniMax/MiniMax-M2.html#deepgemm-usage