Inference models

tencent/Hy3-FP8

Lambda — Wed, 08 Jul 2026 14:38:23 GMT

TL;DR: token throughput (vLLM)

Hardware configuration	Requests/s	Generation throughput (tok/s)	Total throughput (tok/s)	TTFT (ms)	ITL (ms)	Prompts	Tokens in	Tokens out	Parallel requests
4× NVIDIA B200 GPUs	0.86	1,756.55	8,782.76	1,607.07	17.44	512	4,194,304	1,048,576	32

zai-org/GLM-5.2

Lambda — Wed, 01 Jul 2026 15:52:24 GMT

TL;DR: token throughput

Measured on NVIDIA HGX B200, CUDA 12.8. 8192 in / 2048 out tokens, 32 concurrent requests.

moonshotai/Kimi-K2.7-Code

Lambda — Wed, 01 Jul 2026 15:52:18 GMT

TL;DR: token throughput

Measured on NVIDIA HGX B200, CUDA 12.8. 8192 in / 2048 out tokens, 32 concurrent requests.

MiniMaxAI/MiniMax-M3

Lambda — Tue, 30 Jun 2026 16:16:01 GMT

TL;DR: token throughput

Measured on NVIDIA HGX B200, CUDA 12.8. 8192 in / 2048 out tokens, 32 concurrent requests.

nvidia/nemotron-3-ultra

Lambda — Sat, 06 Jun 2026 15:42:26 GMT

TL;DR: token throughput

All benchmarks use the single NVFP4 checkpoint (nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4) on a decode-heavy 8K-input / 64K-output workload at 256 concurrent requests. Per-user generation throughput is aggregate generation tok/s divided by the concurrency level — the rate each individual user sees their response stream back.

LiquidAI/LFM2.5-8B-A1B

Lambda — Wed, 03 Jun 2026 14:44:31 GMT

TL;DR: token throughput

SGLang vLLM

Hardware	Gen. throughput	Per-user gen	Total throughput	TTFT (mean)	ITL (mean)
1× NVIDIA B200 GPU	6,098 tok/s	206 tok/s	30,489 tok/s	792 ms	4.9 ms
1× NVIDIA H100 GPU	3,714 tok/s	125 tok/s	18,572 tok/s	1,248 ms	8.0 ms
1× NVIDIA A100 GPU	1,950 tok/s	68 tok/s	9,751 tok/s	3,594 ms	14.7 ms

Hardware	Gen. throughput	Per-user gen	Total throughput	TTFT (mean)	ITL (mean)
1× NVIDIA B200 GPU	7,253 tok/s	238 tok/s	36,267 tok/s	433 ms	4.5 ms
1× NVIDIA H100 GPU	3,787 tok/s	123 tok/s	18,937 tok/s	568 ms	8.2 ms
1× NVIDIA A100 GPU	1,971 tok/s	64 tok/s	9,853 tok/s	962 ms	15.7 ms

HuggingFaceBio/Carbon-3B

Lambda — Wed, 03 Jun 2026 14:44:28 GMT

TL;DR: nucleotide throughput

Both Carbon-500M and Carbon-3B run on a single NVIDIA A10 GPU, served with SGLang or vLLM. Because Carbon tokenizes DNA as non-overlapping 6-mers, each token carries roughly 6 base pairs, so the token rates below correspond to roughly 6× as many base pairs per second.

stepfun-ai/Step-3.7-Flash

Lambda — Wed, 03 Jun 2026 14:44:24 GMT

TL;DR: token throughput

vLLM with MTP speculative decoding, workload 8192 in / 2048 out tokens at 32 concurrent requests.

moonshotai/Kimi-K2.6

Lambda — Thu, 30 Apr 2026 14:02:36 GMT

TL;DR: token throughput

vLLM

Hardware	Gen. throughput	Per-user gen	Total throughput	TTFT (mean)	ITL (mean)
NVIDIA HGX B200	1408 tok/s	44 tok/s	7046 tok/s	2264 ms	44.5 ms

deepseek-ai/DeepSeek-V4-Pro

Lambda — Tue, 28 Apr 2026 16:51:18 GMT

TL;DR: token throughput

vLLM

Hardware	Gen. throughput	Per-user gen	Total throughput	TTFT (mean)	ITL (mean)
NVIDIA HGX B200	911.92 tok/s	28.50 tok/s	4,561.38 tok/s	1,186.15 ms	55.79 ms