26 models with qualified results

Models

Measured configurations by model. Open a model to compare GPUs on one exact test.

Get your startup command

Choose your operating system and enter a model repository. Launch support is checked separately from performance results.

All tasks · All measured GPUs

Speeds compare only within one exact test. Recipe availability is checked on each report against the published recipe executor.

WeVisOne-VL-2B

tencent · 6 tested configurations · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBBF16 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 153tok/s
Best measured · 16 concurrent requests: 319tok/s
Observed GPU occupancy: 85.3GiB
First token: 753 ms

Use this recipe · 16 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	153.44	13.39 s	753 ms	85.25	343.4	1	d2914dd8 ↗
C2	208.65	19.64 s	1.1 s	85.25	360.4	1	d2914dd8 ↗
C4	259.19	33.21 s	1.6 s	85.25	398.9	1	d2914dd8 ↗
C8	295.37	59.5 s	2.1 s	85.25	422	1	d2914dd8 ↗
C12	311.33	86.14 s	2.7 s	85.25	432.4	1	d2914dd8 ↗
C16	319.17	112.4 s	3.3 s	85.25	438	1	d2914dd8 ↗

5 other test configurations on this GPU

BF16 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 162tok/s
Best measured · 18 concurrent requests: 857tok/s
Observed GPU occupancy: 85.3GiB
First token: 71 ms

Use this recipe · 18 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	161.57	12.72 s	71 ms	85.25	327.8	1	75198c63 ↗
C2	269.43	15.21 s	86 ms	85.25	349.1	1	75198c63 ↗
C4	460.41	17.83 s	106 ms	85.25	427.3	1	75198c63 ↗
C8	644.45	25.52 s	127 ms	85.25	486.9	1	75198c63 ↗
C16	771.04	42.77 s	184 ms	85.25	533.9	1	75198c63 ↗
C17	812.79	43.14 s	195 ms	85.25	550.4	1	75198c63 ↗
C18	856.64	43.41 s	196 ms	85.25	568.9	1	75198c63 ↗

BF16 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 101tok/s
Best measured · 8 concurrent requests: 151tok/s
Observed GPU occupancy: 85.3GiB
First token: 2.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 5 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	101.26	20.26 s	2.3 s	85.25	369.1	1	c5c9cf6f ↗
C2	123.11	33.41 s	3.6 s	85.25	404.8	1	c5c9cf6f ↗
C4	140.19	64.26 s	4.5 s	85.25	430	1	c5c9cf6f ↗
C6	142.08	97.79 s	5.3 s	85.25	430.4	1	c5c9cf6f ↗
C8	150.73	122.37 s	6.3 s	85.25	446.7	1	c5c9cf6f ↗

BF16 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 114tok/s
Best measured · 9 concurrent requests: 433tok/s
Observed GPU occupancy: 85.3GiB
First token: 135 ms

Use this recipe · 9 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	113.51	18.06 s	135 ms	85.25	341.3	1	5fee1315 ↗
C2	153.3	26.73 s	161 ms	85.25	369.5	1	5fee1315 ↗
C4	281.86	29.1 s	202 ms	85.25	483.8	1	5fee1315 ↗
C6	293.74	41.93 s	214 ms	85.25	459.2	1	5fee1315 ↗
C7	340.1	42.27 s	227 ms	85.25	496.1	1	5fee1315 ↗
C8	387.93	42.41 s	237 ms	85.25	534.9	1	5fee1315 ↗
C9	433.07	42.78 s	246 ms	85.25	575.8	1	5fee1315 ↗

BF16 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 56.1tok/s
Best measured · 4 concurrent requests: 66.8tok/s
Observed GPU occupancy: 85.3GiB
First token: 8 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.09	36.63 s	8 s	85.25	418.9	1	2c661c7e ↗
C2	62.52	67.49 s	12 s	85.25	444.3	1	2c661c7e ↗
C3	63.59	109.31 s	13.5 s	85.25	446.8	1	2c661c7e ↗
C4	66.75	144.63 s	15 s	85.25	461.6	1	2c661c7e ↗

BF16 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 71tok/s
Best measured · 4 concurrent requests: 165tok/s
Observed GPU occupancy: 85.3GiB
First token: 268 ms

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	71.05	28.83 s	268 ms	85.25	356.6	1	1ff0c98a ↗
C2	89.43	45.82 s	323 ms	85.25	399.6	1	1ff0c98a ↗
C3	135.43	45.41 s	379 ms	85.25	470.9	1	1ff0c98a ↗
C4	165.29	49.61 s	420 ms	85.25	524	1	1ff0c98a ↗

Qwen3.8-27B-NVFP4

unsloth · 11 tested configurations · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBNVFP4 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 52.8tok/s
Best measured · 8 concurrent requests: 215tok/s
Observed GPU occupancy: 87.4GiB
First token: 3.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.78	38.8 s	3.3 s	87.37	337.3	1	aee8b33e ↗
C2	87.2	48.71 s	4.9 s	87.37	343.4	1	aee8b33e ↗
C4	142.14	64.91 s	7 s	87.37	375.3	1	aee8b33e ↗
C6	187.69	79.85 s	8.1 s	87.37	402.8	1	aee8b33e ↗
C7	202.31	85.21 s	8.7 s	87.37	411.8	1	aee8b33e ↗
C8	215.05	94.52 s	9.3 s	87.37	421.4	1	aee8b33e ↗

10 other test configurations on this GPU

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57tok/s
Best measured · 8 concurrent requests: 374tok/s
Observed GPU occupancy: 87.4GiB
First token: 222 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.96	36.17 s	222 ms	87.37	322.6	1	bced0642 ↗
C2	104.14	39.34 s	355 ms	87.37	305.9	1	bced0642 ↗
C4	198.42	41.3 s	704 ms	87.37	319.3	1	bced0642 ↗
C6	291.12	42.23 s	1 s	87.37	342.1	1	bced0642 ↗
C7	333.32	43.04 s	1.1 s	87.37	349.1	1	bced0642 ↗
C8	374.04	43.82 s	1.3 s	87.37	353.2	1	bced0642 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.1tok/s
Best measured · 8 concurrent requests: 373tok/s
Observed GPU occupancy: 86.6GiB
First token: 231 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.09	35.88 s	231 ms	86.63	319	1	37f29252 ↗
C4	197.8	41.44 s	650 ms	86.63	330.2	1	37f29252 ↗
C8	372.88	43.98 s	1.3 s	86.63	367.3	1	37f29252 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 56.4tok/s
Best measured · 8 concurrent requests: 371tok/s
Observed GPU occupancy: 86.4GiB
First token: 244 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.4	36.51 s	244 ms	86.35	319.1	1	791b887d ↗
C4	196.45	41.73 s	654 ms	86.35	315.5	1	791b887d ↗
C8	370.68	44.24 s	1.3 s	86.36	348.1	1	791b887d ↗

NVFP4 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 44.9tok/s
Best measured · 4 concurrent requests: 96.8tok/s
Observed GPU occupancy: 87.4GiB
First token: 8.6 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	44.86	45.67 s	8.6 s	87.37	360.5	1	d2ff32ec ↗
C2	67.59	63.21 s	12.7 s	87.37	379.1	1	d2ff32ec ↗
C3	84.95	84.07 s	14.6 s	87.37	398.5	1	d2ff32ec ↗
C4	96.77	105.62 s	16.3 s	87.37	415.8	1	d2ff32ec ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.7tok/s
Best measured · 8 concurrent requests: 340tok/s
Observed GPU occupancy: 87.4GiB
First token: 329 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.72	37.56 s	329 ms	87.37	322.4	1	d3f6b770 ↗
C2	99.95	40.99 s	522 ms	87.37	315.9	1	d3f6b770 ↗
C4	189.07	43.34 s	971 ms	87.37	337.8	1	d3f6b770 ↗
C6	264.7	46.44 s	1.3 s	87.37	364	1	d3f6b770 ↗
C7	303.08	47.32 s	1.4 s	87.37	373.4	1	d3f6b770 ↗
C8	339.61	48.27 s	1.7 s	87.37	381.2	1	d3f6b770 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54tok/s
Best measured · 8 concurrent requests: 337tok/s
Observed GPU occupancy: 86.4GiB
First token: 361 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.04	38.07 s	361 ms	86.36	320.7	1	417f69be ↗
C4	187.12	43.81 s	879 ms	86.36	333.2	1	417f69be ↗
C8	336.95	48.66 s	1.6 s	86.36	379.7	1	417f69be ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.5tok/s
Best measured · 8 concurrent requests: 339tok/s
Observed GPU occupancy: 86.6GiB
First token: 359 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.54	37.55 s	359 ms	86.63	324.9	1	7d1a7c8e ↗
C4	188.48	43.48 s	878 ms	86.63	350.2	1	7d1a7c8e ↗
C8	338.65	48.43 s	1.6 s	86.63	396.9	1	7d1a7c8e ↗

NVFP4 · 128K → 2K tokens · No cache reuse
Output throughput · includes prefill · 1 request
31.5tok/s
Best measured · 2 concurrent requests
41.3tok/s
Observed GPU occupancy
87.4GiB
First token
24.9 s
Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 concurrency levels
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm · vLLM 0.28.0
Protocol
wbench-search-http-v1
Dataset
migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload
131072in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 31.54 64.93 s 24.9 s 87.37 402.7 1 73fcb7c1 ↗
C2 41.27 111.14 s 34 s 87.37 426.3 1 73fcb7c1 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.54	64.93 s	24.9 s	87.37	402.7	1	73fcb7c1 ↗
C2	41.27	111.14 s	34 s	87.37	426.3	1	73fcb7c1 ↗

NVFP4 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 31.5tok/s
Best measured · 8 concurrent requests: 57.7tok/s
Observed GPU occupancy: 86.6GiB
First token: 24.9 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.45	65.12 s	24.9 s	86.63	415.7	1	448b0777 ↗
C4	50.67	220.6 s	45.8 s	86.63	469.4	1	448b0777 ↗
C8	57.74	421.65 s	61.9 s	86.63	494.5	1	448b0777 ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 50tok/s
Best measured · 8 concurrent requests: 287tok/s
Observed GPU occupancy: 86.4GiB
First token: 506 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	49.98	40.98 s	506 ms	86.36	322.7	1	68d1f2c6 ↗
C4	171.33	47.85 s	1 s	86.36	365.5	1	68d1f2c6 ↗
C8	287.31	57.12 s	1.9 s	86.36	425.4	1	68d1f2c6 ↗

Qwen3.8-27B-NVFP4

RadixArk · 14 tested configurations · 2 GPUs

RTX 5090 · 32 GBNVFP4 · 8K → 2K tokens
Output throughput · includes prefill · 1 request
70.8tok/s
Observed GPU occupancy
27.3GiB
First token
633 ms
Use this recipe · 1 request ↗Evidence report and run availability ↗Same test on other GPUs
Test details
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm
Protocol
wbench-fixed-http-v1
Dataset
synthetic-tokenizer-v1:repeats=2
Workload
8192in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 70.82 29.01 s 633 ms 27.29 405.5 1 bad294c0 ↗
1 other test configurations on this GPU
- NVFP4 · 32K → 2K tokens
  Output throughput · includes prefill · 1 request
  62.5tok/s
  Observed GPU occupancy
  27.3GiB
  First token
  3.6 s
  Use this recipe · 1 request ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  vllm
  Protocol
  wbench-fixed-http-v1
  Dataset
  synthetic-tokenizer-v1:repeats=2
  Workload
  32768in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 62.48 32.79 s 3.6 s 27.29 433.3 1 746bbc39 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	70.82	29.01 s	633 ms	27.29	405.5	1	bad294c0 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	62.48	32.79 s	3.6 s	27.29	433.3	1	746bbc39 ↗

RTX PRO 6000 Blackwell Server Edition · 96 GBNVFP4 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 55.6tok/s
Best measured · 8 concurrent requests: 225tok/s
Observed GPU occupancy: 86GiB
First token: 3.2 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	55.59	36.91 s	3.2 s	85.98	341	1	dca51ce5 ↗
C2	91.37	46.48 s	4.6 s	85.98	342.9	1	dca51ce5 ↗
C4	149.21	62.63 s	6.5 s	85.98	372.9	1	dca51ce5 ↗
C6	196.73	76.24 s	7.8 s	85.98	398.9	1	dca51ce5 ↗
C7	211.87	81.41 s	8.3 s	85.98	408.7	1	dca51ce5 ↗
C8	224.61	90.5 s	8.9 s	85.98	414.8	1	dca51ce5 ↗

11 other test configurations on this GPU

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 60.1tok/s
Best measured · 8 concurrent requests: 393tok/s
Observed GPU occupancy: 86GiB
First token: 217 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	60.1	34.34 s	217 ms	85.98	323	1	9cdf45b7 ↗
C2	109.3	37.48 s	349 ms	85.98	310.8	1	9cdf45b7 ↗
C4	209.23	39.18 s	686 ms	85.98	324.8	1	9cdf45b7 ↗
C6	306.94	40.06 s	992 ms	85.98	336	1	9cdf45b7 ↗
C7	351.24	40.83 s	1.1 s	85.98	342.6	1	9cdf45b7 ↗
C8	392.93	41.71 s	1.3 s	85.98	348.2	1	9cdf45b7 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 59.7tok/s
Best measured · 8 concurrent requests: 394tok/s
Observed GPU occupancy: 86.8GiB
First token: 230 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	59.68	34.59 s	230 ms	86.82	323.3	1	83d37469 ↗
C4	207.31	39.54 s	664 ms	86.82	320	1	83d37469 ↗
C8	394.04	41.6 s	1.2 s	86.82	346.1	1	83d37469 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 60.5tok/s
Best measured · 8 concurrent requests: 395tok/s
Observed GPU occupancy: 86.9GiB
First token: 228 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	60.47	33.87 s	228 ms	86.86	320.3	1	cbf1431c ↗
C4	207.86	39.44 s	636 ms	86.86	333.6	1	cbf1431c ↗
C8	394.52	41.6 s	1.3 s	86.86	361.6	1	cbf1431c ↗

NVFP4 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 47.1tok/s
Best measured · 4 concurrent requests: 101tok/s
Observed GPU occupancy: 86GiB
First token: 8.4 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	47.15	43.45 s	8.4 s	85.98	364.4	1	e5a201fa ↗
C2	70.39	60.71 s	12.3 s	85.98	381.3	1	e5a201fa ↗
C3	88.36	80.85 s	14.2 s	85.98	403.6	1	e5a201fa ↗
C4	100.66	101.65 s	15.8 s	85.98	419.2	1	e5a201fa ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.5tok/s
Best measured · 8 concurrent requests: 355tok/s
Observed GPU occupancy: 86GiB
First token: 326 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.45	35.8 s	326 ms	85.98	326.1	1	67a5e6d3 ↗
C2	104.7	39.14 s	515 ms	85.98	321	1	67a5e6d3 ↗
C4	198.72	41.24 s	954 ms	85.98	344.2	1	67a5e6d3 ↗
C6	277.65	44.28 s	1.3 s	85.98	358.8	1	67a5e6d3 ↗
C7	317.41	45.18 s	1.4 s	85.98	369.2	1	67a5e6d3 ↗
C8	354.95	46.19 s	1.6 s	85.98	377.1	1	67a5e6d3 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.1tok/s
Best measured · 8 concurrent requests: 356tok/s
Observed GPU occupancy: 86.8GiB
First token: 358 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.08	36.15 s	358 ms	86.82	326	1	275a096d ↗
C4	196.91	41.63 s	870 ms	86.83	338.7	1	275a096d ↗
C8	355.89	46.08 s	1.6 s	86.83	375.7	1	275a096d ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.7tok/s
Best measured · 8 concurrent requests: 356tok/s
Observed GPU occupancy: 86.9GiB
First token: 357 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.67	35.51 s	357 ms	86.86	325.8	1	fe8141d2 ↗
C4	197.42	41.52 s	870 ms	86.86	354.3	1	fe8141d2 ↗
C8	356.25	46.05 s	1.6 s	86.86	392.4	1	fe8141d2 ↗

NVFP4 · 128K → 2K tokens · No cache reuse
Output throughput · includes prefill · 1 request
32.7tok/s
Best measured · 2 concurrent requests
42.5tok/s
Observed GPU occupancy
86GiB
First token
24.3 s
Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 concurrency levels
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm · vLLM 0.28.0
Protocol
wbench-search-http-v1
Dataset
migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload
131072in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 32.72 62.59 s 24.3 s 85.99 406.5 1 28f51cf1 ↗
C2 42.54 106.24 s 33.5 s 85.99 428.9 1 28f51cf1 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	32.72	62.59 s	24.3 s	85.99	406.5	1	28f51cf1 ↗
C2	42.54	106.24 s	33.5 s	85.99	428.9	1	28f51cf1 ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 53tok/s
Best measured · 4 concurrent requests: 181tok/s
Observed GPU occupancy: 86GiB
First token: 435 ms

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.97	38.72 s	435 ms	85.99	333.9	1	78fc908c ↗
C2	97.25	42.14 s	646 ms	85.99	340.8	1	78fc908c ↗
C3	138.66	44.36 s	822 ms	85.99	358.8	1	78fc908c ↗
C4	181.16	45.25 s	1.1 s	85.99	375.3	1	78fc908c ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 52.9tok/s
Best measured · 8 concurrent requests: 301tok/s
Observed GPU occupancy: 86.9GiB
First token: 508 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.9	38.72 s	508 ms	86.86	338	1	7cea351c ↗
C4	179.81	45.65 s	1 s	86.86	387.5	1	7cea351c ↗
C8	301.16	54.48 s	1.9 s	86.86	443.7	1	7cea351c ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 52.6tok/s
Best measured · 8 concurrent requests: 301tok/s
Observed GPU occupancy: 86.8GiB
First token: 509 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.59	38.95 s	509 ms	86.83	328.8	1	d14962c1 ↗
C4	179.42	45.72 s	1.1 s	86.83	371.9	1	d14962c1 ↗
C8	300.98	54.52 s	1.9 s	86.83	426	1	d14962c1 ↗

StartLux-Decision-v1-0.8B
startlux-models · 2 tested configurations · 1 GPU
- RTX PRO 6000 Blackwell Server Edition · 96 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  50.3decisions/sec
  Observed GPU occupancy
  3.2GiB
  P95 latency
  20 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  startlux-native · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 50.25 20 ms — 3.17 161.3 1 df141700 ↗
- 1 other test configurations on this GPU
  - bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
    Throughput · Batch 1
    50.3decisions/sec
    Observed GPU occupancy
    2.3GiB
    P95 latency
    20 ms
    Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
    Test details
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    startlux-native · PyTorch 2.11.0+cu130
    Protocol
    native-forward-v1
    Dataset
    typed-invoices-v1
    Workload
    decision
    Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 50.28 20 ms — 2.29 165 1 8c1985ea ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.25	20 ms	—	3.17	161.3	1	df141700 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.28	20 ms	—	2.29	165	1	8c1985ea ↗

Intern-Decision-0.8B

internlm · 4 tested configurations · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
Throughput · Batch 1
50.8decisions/sec
Best measured · Batch 17
347decisions/sec
Observed GPU occupancy
2.4GiB
P95 latency
20 ms
Use this recipe · Batch 17 ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 batch sizes
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
intern-decision-native · PyTorch 2.11.0+cu130
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 50.83 20 ms — 2.36 169 2 18b06f63 ↗
B17 346.72 50 ms — 3.07 378 1 6ccb367c ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.83	20 ms	—	2.36	169	2	18b06f63 ↗
B17	346.72	50 ms	—	3.07	378	1	6ccb367c ↗

3 other test configurations on this GPU

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 399 s

Throughput · Batch 1: 52decisions/sec
Best measured · Batch 16: 334decisions/sec
Observed GPU occupancy: 2.4GiB
P95 latency: 20 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 21 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: intern-decision-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	52.04	20 ms	—	2.36	162.6	1	156910e7 ↗
B2	96.21	21 ms	—	2.38	213.4	1	156910e7 ↗
B4	177.08	23 ms	—	2.43	243.4	1	156910e7 ↗
B8	299.06	27 ms	—	2.57	341.2	1	156910e7 ↗
B12	306.24	40 ms	—	36.98	349.6	1	156910e7 ↗
B14	322.36	43 ms	—	36.98	365.9	1	156910e7 ↗
B15	328.14	46 ms	—	36.98	369.8	1	156910e7 ↗
B16	333.92	48 ms	—	3.19	368.4	1	156910e7 ↗
B17	331.9	52 ms	—	36.98	379.5	1	156910e7 ↗
B18	310.84	58 ms	—	36.98	365.4	1	156910e7 ↗
B20	316.13	64 ms	—	36.98	375.1	1	156910e7 ↗
B24	312.87	77 ms	—	36.98	377.5	1	156910e7 ↗
B28	309.61	91 ms	—	36.98	379.4	1	156910e7 ↗
B32	319.44	101 ms	—	3.44	376.3	1	156910e7 ↗
B40	307.24	131 ms	—	36.98	379.6	1	156910e7 ↗
B48	306.75	159 ms	—	36.98	380.4	1	156910e7 ↗
B64	304.06	213 ms	—	4.2	383.7	1	156910e7 ↗
B128	285.17	453 ms	—	6.23	387.8	1	156910e7 ↗
B256	284.61	904 ms	—	10.63	390.3	1	156910e7 ↗
B512	282.22	2.04 s	—	19.42	390.1	1	156910e7 ↗
B1024	286.53	3.58 s	—	36.98	396.2	1	156910e7 ↗

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
Throughput · Batch 1
51.5decisions/sec
Observed GPU occupancy
2.6GiB
P95 latency
20 ms
Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
Test details
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
intern-decision-native · PyTorch 2.11.0
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 51.51 20 ms — 2.6 164.2 1 7f2e07c0 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	51.51	20 ms	—	2.6	164.2	1	7f2e07c0 ↗

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 396 s

Throughput · Batch 1: 52.8decisions/sec
Best measured · Batch 17: 350decisions/sec
Observed GPU occupancy: 2.4GiB
P95 latency: 19 ms

Use this recipe · Batch 17 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 21 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: intern-decision-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	52.8	19 ms	—	2.36	172.8	1	5eef633e ↗
B2	95.72	21 ms	—	2.38	186.5	1	5eef633e ↗
B4	177.64	23 ms	—	2.43	242	1	5eef633e ↗
B8	306.42	26 ms	—	2.57	344.2	1	5eef633e ↗
B12	322.68	37 ms	—	37.46	371.9	1	5eef633e ↗
B13	336	39 ms	—	37.46	380.8	1	5eef633e ↗
B14	344.61	41 ms	—	37.46	392.6	1	5eef633e ↗
B15	345.75	44 ms	—	37.46	389.5	1	5eef633e ↗
B16	345.28	47 ms	—	2.85	383.9	1	5eef633e ↗
B17	349.83	49 ms	—	37.46	397.8	1	5eef633e ↗
B18	327.34	55 ms	—	37.46	383.4	1	5eef633e ↗
B20	333.21	60 ms	—	37.46	402.6	1	5eef633e ↗
B24	331.98	73 ms	—	37.46	401.2	1	5eef633e ↗
B28	327.01	86 ms	—	37.46	399.2	1	5eef633e ↗
B32	331.67	97 ms	—	3.41	397.7	1	5eef633e ↗
B48	326.46	148 ms	—	37.46	405.9	1	5eef633e ↗
B64	318.22	202 ms	—	4.51	403.2	1	5eef633e ↗
B128	298.77	430 ms	—	6.71	405.8	1	5eef633e ↗
B256	298.58	859 ms	—	11.11	407.3	1	5eef633e ↗
B512	299.67	1.71 s	—	19.9	411.2	1	5eef633e ↗
B1024	297.46	3.64 s	—	37.46	410.6	1	5eef633e ↗

clef-flash

Cloudflare · 6 tested configurations · 2 GPUs

RTX PRO 6000 Blackwell Server Edition · 96 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 33 s

Throughput · Batch 1: 31.8decisions/sec
Best measured · Batch 4: 39.3decisions/sec
Observed GPU occupancy: 18.9GiB
P95 latency: 32 ms

Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	31.83	32 ms	—	18.95	388.5	1	10c64aff ↗
B4	39.27	93 ms	—	18.95	449.5	1	10c64aff ↗
B16	5.41	8.21 s	—	19.28	233.4	1	10c64aff ↗
B64	15.43	9.66 s	—	21.89	324.3	1	10c64aff ↗

1 other test configurations on this GPU
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 16
  47.7decisions/sec
  Observed GPU occupancy
  19.5GiB
  P95 latency
  338 ms
  Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  clef-native · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B16 47.71 338 ms — 19.55 499.5 2 82d98b36 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B16	47.71	338 ms	—	19.55	499.5	2	82d98b36 ↗

vGPU-32GB · 32 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 1 s
Throughput · Batch 1
10.3decisions/sec
Best measured · Batch 4
10.9decisions/sec
Observed GPU occupancy
18.2GiB
P95 latency
104 ms
Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 batch sizes
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
clef-native
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 10.26 104 ms — 18.17 67.9 1 1d1661ad ↗
B4 10.89 389 ms — 18.49 156 1 1d1661ad ↗
3 other test configurations on this GPU
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  10.6decisions/sec
  Best measured · Batch 4
  11decisions/sec
  Observed GPU occupancy
  18.2GiB
  P95 latency
  99 ms
  Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  clef-native · PyTorch 2.13.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 10.56 99 ms — 18.17 206.1 2 63f44036 ↗
  B4 11.04 402 ms — 18.55 244.4 2 63f44036 ↗
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  10.6decisions/sec
  Best measured · Batch 4
  11.1decisions/sec
  Observed GPU occupancy
  18.2GiB
  P95 latency
  103 ms
  Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  clef-native
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 10.56 103 ms — 18.17 196.8 1 4eb73a4d ↗
  B4 11.1 390 ms — 18.56 235.9 1 4eb73a4d ↗
- bf16 · 1K → 1K tokens
  Throughput · Batch 1
  10.2decisions/sec
  Best measured · Batch 4
  10.9decisions/sec
  Observed GPU occupancy
  18.2GiB
  P95 latency
  104 ms
  Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  vLLM
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 10.19 104 ms — 18.17 69.8 1 caef7154 ↗
  B4 10.88 388 ms — 18.49 152.8 1 caef7154 ↗

StartLux-Decision-v1.1-0.8B-Instant
startlux-models · 2 tested configurations · 1 GPU
- RTX PRO 6000 Blackwell Server Edition · 96 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  18.7decisions/sec
  Observed GPU occupancy
  2.7GiB
  P95 latency
  54 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  startlux-transformers · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 18.69 54 ms — 2.73 134.5 2 a4a69815 ↗
- 1 other test configurations on this GPU
  - bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
    Throughput · Batch 1
    18.8decisions/sec
    Observed GPU occupancy
    2.5GiB
    P95 latency
    54 ms
    Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
    Test details
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    startlux-transformers · PyTorch 2.11.0
    Protocol
    native-forward-v1
    Dataset
    typed-invoices-v1
    Workload
    decision
    Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 18.83 54 ms — 2.54 125.5 1 c342a9ca ↗
Intern-Decision-2B
internlm · 1 tested configuration · 1 GPU
- RTX PRO 6000 Blackwell Server Edition · 96 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  50.3decisions/sec
  Observed GPU occupancy
  4.9GiB
  P95 latency
  21 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  intern-decision-native · PyTorch 2.11.0
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 50.29 21 ms — 4.9 216.2 1 e4bcd6a3 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	10.26	104 ms	—	18.17	67.9	1	1d1661ad ↗
B4	10.89	389 ms	—	18.49	156	1	1d1661ad ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	10.56	99 ms	—	18.17	206.1	2	63f44036 ↗
B4	11.04	402 ms	—	18.55	244.4	2	63f44036 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	10.56	103 ms	—	18.17	196.8	1	4eb73a4d ↗
B4	11.1	390 ms	—	18.56	235.9	1	4eb73a4d ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	10.19	104 ms	—	18.17	69.8	1	caef7154 ↗
B4	10.88	388 ms	—	18.49	152.8	1	caef7154 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	18.69	54 ms	—	2.73	134.5	2	a4a69815 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	18.83	54 ms	—	2.54	125.5	1	c342a9ca ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.29	21 ms	—	4.9	216.2	1	e4bcd6a3 ↗

clef-omni

Cloudflare · 1 tested configuration · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 44 s

Throughput · Batch 1: 7.5decisions/sec
Best measured · Batch 64: 7.5decisions/sec
Observed GPU occupancy: 60.9GiB
P95 latency: 134 ms

Use this recipe · Batch 64 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	7.45	134 ms	—	60.88	232.8	1	16597f91 ↗
B4	7.47	537 ms	—	60.89	249.7	1	16597f91 ↗
B16	7.48	2.14 s	—	60.89	252.8	1	16597f91 ↗
B64	7.49	8.56 s	—	60.89	257.2	1	16597f91 ↗

clef

Cloudflare · 1 tested configuration · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBbf16 · Text → Structured data · typed-invoices-v1 · Test duration 46 s

Throughput · Batch 1: 12.3decisions/sec
Observed GPU occupancy: 52.4GiB
P95 latency: 82 ms

Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	12.32	82 ms	—	52.4	469.1	1	243b7f64 ↗
B4	1.4	8.09 s	—	52.4	225.7	1	243b7f64 ↗
B16	4.33	9.1 s	—	52.98	308.5	1	243b7f64 ↗
B64	9.39	12.46 s	—	57.95	435.9	1	243b7f64 ↗

Qwen3-4B-AWQ
Qwen · 5 tested configurations · 3 GPUs
- L4 · 24 GBAWQ · 8K → 2K tokens · No cache reuse
  Output throughput · includes prefill · 1 request
  55.3tok/s
  Best measured · 4 concurrent requests
  109tok/s
  Observed GPU occupancy
  19.5GiB
  First token
  1.6 s
  Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 concurrency levels
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  vllm · vLLM 0.29.0
  Protocol
  wbench-fixed-http-v3
  Dataset
  migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 55.3 37.17 s 1.6 s 19.54 71.9 1 9301a9ba ↗
  C4 109.43 78.88 s 3.1 s 19.55 71.9 1 9301a9ba ↗
- 1 other test configurations on this GPU
  - AWQ · 8K → 2K tokens · Prefix cache reuse
    Output throughput · includes prefill · 1 request
    57.4tok/s
    Best measured · 4 concurrent requests
    175tok/s
    Observed GPU occupancy
    19.5GiB
    First token
    60 ms
    Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 concurrency levels
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    vllm · vLLM 0.29.0
    Protocol
    wbench-fixed-http-v3
    Dataset
    migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 57.39 35.7 s 60 ms 19.55 71.8 1 7e2cbfda ↗
    C4 175.39 46.75 s 114 ms 19.55 71.9 1 7e2cbfda ↗
- RTX PRO 6000 Blackwell Server Edition · 96 GBAWQ · 8K → 2K tokens · No cache reuse
  Output throughput · includes prefill · 1 request
  215tok/s
  Best measured · 4 concurrent requests
  517tok/s
  Observed GPU occupancy
  86.3GiB
  First token
  298 ms
  Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 concurrency levels
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  vllm · vLLM 0.29.0
  Protocol
  wbench-fixed-http-v3
  Dataset
  migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 214.6 9.63 s 298 ms 86.28 303.8 1 86d36e4f ↗
  C4 516.94 16.14 s 872 ms 86.28 339.5 1 86d36e4f ↗
- 1 other test configurations on this GPU
  - AWQ · 8K → 2K tokens · Prefix cache reuse
    Output throughput · includes prefill · 1 request
    223tok/s
    Best measured · 4 concurrent requests
    742tok/s
    Observed GPU occupancy
    86.3GiB
    First token
    28 ms
    Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 concurrency levels
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    vllm · vLLM 0.29.0
    Protocol
    wbench-fixed-http-v3
    Dataset
    migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 222.74 9.19 s 28 ms 86.28 288.8 1 adbfef8a ↗
    C4 741.85 11.05 s 35 ms 86.28 335 1 adbfef8a ↗
- T4 · 16 GBAWQ · 8K → 2K tokens · No cache reuse
  Output throughput · includes prefill · 1 request
  15.4tok/s
  Best measured · 4 concurrent requests
  20.3tok/s
  Observed GPU occupancy
  12.7GiB
  First token
  28.5 s
  Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 concurrency levels
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  vllm · vLLM 0.29.0
  Protocol
  wbench-fixed-http-v3
  Dataset
  migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 15.39 133.36 s 28.5 s 12.74 67.9 1 fcfe2d64 ↗
  C4 20.31 468.13 s 53.1 s 12.74 67.5 1 fcfe2d64 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	55.3	37.17 s	1.6 s	19.54	71.9	1	9301a9ba ↗
C4	109.43	78.88 s	3.1 s	19.55	71.9	1	9301a9ba ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.39	35.7 s	60 ms	19.55	71.8	1	7e2cbfda ↗
C4	175.39	46.75 s	114 ms	19.55	71.9	1	7e2cbfda ↗

embeddinggemma-2

google · 11 tested configurations · 4 GPUs

L4 · 24 GBnf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 175vectors/sec
Best measured · Batch 16: 1,639vectors/sec
Observed GPU occupancy: 0.9GiB
P95 latency: 6 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	174.88	6 ms	—	0.85	53.1	6	7ad6a821 ↗
B4	571.03	7 ms	—	0.85	57	6	7ad6a821 ↗
B16	1,639.08	10 ms	—	0.85	59.8	6	7ad6a821 ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 177vectors/sec
Best measured · Batch 16: 2,039vectors/sec
Observed GPU occupancy: 1GiB
P95 latency: 6 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	177.46	6 ms	—	0.99	49.8	6	e0700e04 ↗
B4	650.16	6 ms	—	0.99	56.9	6	e0700e04 ↗
B16	2,039.17	8 ms	—	0.99	58.4	6	e0700e04 ↗

RTX PRO 6000 Blackwell Server Edition · 96 GBnf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 189vectors/sec
Best measured · Batch 16: 2,572vectors/sec
Observed GPU occupancy: 1.3GiB
P95 latency: 6 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	189.23	6 ms	—	1.33	145.5	1	c673556b ↗
B4	713.32	6 ms	—	1.33	142.8	1	c673556b ↗
B16	2,571.53	7 ms	—	1.33	162.7	1	c673556b ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 207vectors/sec
Best measured · Batch 16: 2,939vectors/sec
Observed GPU occupancy: 1.5GiB
P95 latency: 5 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	206.79	5 ms	—	1.48	119.3	1	196678d3 ↗
B4	764.95	6 ms	—	1.48	141.4	1	196678d3 ↗
B16	2,938.63	6 ms	—	1.48	161.8	1	196678d3 ↗

T4 · 16 GBnf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 90vectors/sec
Best measured · Batch 16: 815vectors/sec
Observed GPU occupancy: 0.7GiB
P95 latency: 11 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	90.02	11 ms	—	0.75	54.6	2	43c67a74 ↗
B4	303.35	13 ms	—	0.75	60.1	2	43c67a74 ↗
B16	814.64	20 ms	—	0.75	64.1	2	43c67a74 ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 109vectors/sec
Best measured · Batch 16: 1,004vectors/sec
Observed GPU occupancy: 0.9GiB
P95 latency: 9 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	108.65	9 ms	—	0.89	49.3	2	2846de88 ↗
B4	365.65	11 ms	—	0.89	57.9	2	2846de88 ↗
B16	1,004.11	16 ms	—	0.89	63.9	2	2846de88 ↗

Show 1 more GPUs

vGPU-32GB · 32 GBbf16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s

Throughput · Batch 1: 251vectors/sec
Best measured · Batch 16: 2,710vectors/sec
Observed GPU occupancy: 1GiB
P95 latency: 4 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.13.0+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	251.11	4 ms	—	1.05	99.4	1	f489c748 ↗
B4	924.83	4 ms	—	1.05	111.5	1	f489c748 ↗
B16	2,710.19	6 ms	—	1.05	114.9	1	f489c748 ↗

4 other test configurations on this GPU

bf16 · Text → Vectors · embedding-sanity-v1 · Test duration 6 s
Throughput · Batch 1
38.3vectors/sec
Best measured · Batch 4
139vectors/sec
Observed GPU occupancy
0.8GiB
P95 latency
31 ms
Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 batch sizes
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
sentence-transformers · PyTorch 2.13.0+cu130
Protocol
native-forward-v1
Dataset
embedding-sanity-v1
Workload
embedding
Batch size Mean throughput (vectors/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 38.33 31 ms — 0.82 50.5 4 2c17d254 ↗
B4 138.64 33 ms — 0.82 54.2 4 2c17d254 ↗

bf16 · Text → Vectors · embedding-sanity-v1 · Test duration 12 s

Throughput · Batch 1: 38.8vectors/sec
Best measured · Batch 64: 1,766vectors/sec
Observed GPU occupancy: 0.8GiB
P95 latency: 27 ms

Use this recipe · Batch 64 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.13.0+cu130
Protocol: native-forward-v1
Dataset: embedding-sanity-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	38.8	27 ms	—	0.82	48.8	1	dc5960cd ↗
B4	141.65	29 ms	—	0.82	52.4	1	dc5960cd ↗
B16	553.54	30 ms	—	0.87	56.4	1	dc5960cd ↗
B64	1,766.47	38 ms	—	1.03	65.3	1	dc5960cd ↗

bf16 · Text → Vectors · embedding-sanity-v1 · Test duration 6 s
Throughput · Batch 1
38.3vectors/sec
Best measured · Batch 4
138vectors/sec
Observed GPU occupancy
0.8GiB
P95 latency
32 ms
Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 batch sizes
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
sentence-transformers
Protocol
native-forward-v1
Dataset
embedding-sanity-v1
Workload
embedding
Batch size Mean throughput (vectors/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 38.27 32 ms — 0.82 48.6 1 47f71b64 ↗
B4 137.82 35 ms — 0.82 51.9 1 47f71b64 ↗
bf16 · Text → Vectors · embedding-sanity-v1
Throughput · Batch 1
39.4vectors/sec
Best measured · Batch 4
120vectors/sec
Observed GPU occupancy
Not recorded
P95 latency
26 ms
Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 batch sizes
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
sentence-transformers
Protocol
native-forward-v1
Dataset
embedding-sanity-v1
Workload
embedding
Batch size Mean throughput (vectors/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 39.4 26 ms — — — 1 2c97c869 ↗
B4 119.85 44 ms — — — 1 2c97c869 ↗

embeddinggemma-2

unsloth · 4 tested configurations · 2 GPUs

L4 · 24 GBnf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 175vectors/sec
Best measured · Batch 16: 1,650vectors/sec
Observed GPU occupancy: 0.9GiB
P95 latency: 6 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	174.95	6 ms	—	0.85	54.8	5	29b19870 ↗
B4	572.6	7 ms	—	0.85	58.4	5	29b19870 ↗
B16	1,649.62	10 ms	—	0.85	60.7	5	29b19870 ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 177vectors/sec
Best measured · Batch 16: 2,036vectors/sec
Observed GPU occupancy: 1GiB
P95 latency: 6 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	177.21	6 ms	—	0.99	50.5	4	d74f69cc ↗
B4	648.08	6 ms	—	0.99	57.1	4	d74f69cc ↗
B16	2,035.67	8 ms	—	0.99	58.1	4	d74f69cc ↗

T4 · 16 GBnf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 90.1vectors/sec
Best measured · Batch 16: 809vectors/sec
Observed GPU occupancy: 0.7GiB
P95 latency: 11 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	90.07	11 ms	—	0.75	56.2	2	2f9465c4 ↗
B4	302.9	13 ms	—	0.75	61.5	2	2f9465c4 ↗
B16	809.43	20 ms	—	0.75	64	2	2f9465c4 ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 109vectors/sec
Best measured · Batch 16: 1,002vectors/sec
Observed GPU occupancy: 0.9GiB
P95 latency: 9 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	108.76	9 ms	—	0.89	50.3	2	cfe051ff ↗
B4	365.82	11 ms	—	0.89	58.1	2	cfe051ff ↗
B16	1,002.27	16 ms	—	0.89	63.9	2	cfe051ff ↗

IndexTTS-2.5
IndexTeam · 1 tested configuration · 1 GPU
- RTX PRO 6000 Blackwell Server Edition · 96 GBNative · Not recorded → Audio · tts_prompts · Test duration 9 s
  Throughput · Batch 1
  4.2audio-sec/sec
  Observed GPU occupancy
  6.5GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  indextts · PyTorch 2.8.0+cu128
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 4.24 — — 6.48 221.7 1 a46ec2c4 ↗

granite-speech-5.0-470m-turboctc

ibm-granite · 2 tested configurations · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBNative · Audio → Text · asr_prompts · Test duration 16 s

Throughput · Batch 1: 2,662audio-sec/sec
Best measured · Batch 32: 9,194audio-sec/sec
Observed GPU occupancy: 1.6GiB
P95 latency: Not recorded

Evidence report and run availability ↗Same test on other GPUs

Test details · 11 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: transformers-ctc · PyTorch 2.13.0
Protocol: audio-native-v1
Dataset: asr_prompts
Workload: asr

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	2,662.33	—	—	1.64	122.1	1	3be7f67e ↗
B2	4,844.87	—	—	1.67	134.9	1	3be7f67e ↗
B4	6,237.58	—	—	1.75	148	1	3be7f67e ↗
B8	7,907.41	—	—	1.9	165.1	1	3be7f67e ↗
B16	9,083.29	—	—	2.33	194.8	1	3be7f67e ↗
B32	9,193.57	—	—	3.25	243.5	1	3be7f67e ↗
B64	8,622.98	—	—	5.09	345.5	1	3be7f67e ↗
B128	8,355.43	—	—	10.1	451	1	3be7f67e ↗
B256	8,343	—	—	20.09	477.6	1	3be7f67e ↗
B512	8,391.94	—	—	40.63	492.3	1	3be7f67e ↗
B1024	8,387.34	—	—	80.89	501.3	1	3be7f67e ↗

1 other test configurations on this GPU

Native · Audio → Text · asr_prompts

Throughput · Batch 1: 3,143audio-sec/sec
Best measured · Batch 16: 8,581audio-sec/sec
Observed GPU occupancy: 1.6GiB
P95 latency: Not recorded

Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: transformers-ctc · PyTorch 2.13.0
Protocol: audio-native-v1
Dataset: asr_prompts
Workload: asr

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	3,143.16	—	—	1.64	95.8	1	d77a0cfd ↗
B8	8,162.53	—	—	1.64	105.2	1	d77a0cfd ↗
B16	8,581.45	—	—	1.92	111.9	1	d77a0cfd ↗

Qwen3-TTS-12Hz-0.6B-CustomVoice
Qwen · 1 tested configuration · 1 GPU
- RTX PRO 6000 Blackwell Server Edition · 96 GBNative · Not recorded → Audio · tts_prompts · Test duration 43 s
  Throughput · Batch 1
  1.6audio-sec/sec
  Best measured · Batch 4
  5.2audio-sec/sec
  Observed GPU occupancy
  3.3GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  qwen-tts · PyTorch 2.14.1
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 1.6 — — 3.34 154.4 1 ebb272f8 ↗
  B4 5.22 — — 4.92 161 1 ebb272f8 ↗
magpie_tts_multilingual_357m
nvidia · 1 tested configuration · 1 GPU
- RTX PRO 6000 Blackwell Server Edition · 96 GBNative · Not recorded → Audio · tts_prompts · Test duration 19 s
  Throughput · Batch 1
  3.6audio-sec/sec
  Observed GPU occupancy
  4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  nemo-magpie · PyTorch 2.13.0
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 3.57 — — 4.01 178.8 1 666efb0f ↗

bge-small-zh-v1.5

BAAI · 1 tested configuration · 1 GPU

T4 · 16 GBnf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 1,170vectors/sec
Best measured · Batch 16: 5,188vectors/sec
Observed GPU occupancy: 0.3GiB
P95 latency: 1 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	1,169.81	1 ms	—	0.29	39	1	0fb03772 ↗
B4	2,301.06	2 ms	—	0.29	32.5	1	0fb03772 ↗
B16	5,188.01	3 ms	—	0.29	32.5	1	0fb03772 ↗

granite-embedding-97m-multilingual-r2

ibm-granite · 2 tested configurations · 2 GPUs

L4 · 24 GBnf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 792vectors/sec
Best measured · Batch 16: 8,881vectors/sec
Observed GPU occupancy: 0.5GiB
P95 latency: 1 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	791.85	1 ms	—	0.51	37.4	1	d396c54b ↗
B4	2,816.45	1 ms	—	0.51	37.9	1	d396c54b ↗
B16	8,880.86	2 ms	—	0.51	38.9	1	d396c54b ↗

T4 · 16 GBfp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 464vectors/sec
Best measured · Batch 16: 4,084vectors/sec
Observed GPU occupancy: 0.5GiB
P95 latency: 2 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	464.26	2 ms	—	0.47	45.4	1	37c260e4 ↗
B4	1,713.26	2 ms	—	0.47	49.4	1	37c260e4 ↗
B16	4,083.56	4 ms	—	0.47	38.3	1	37c260e4 ↗

Qwen3.8-27B-UD-Q4_K_M.gguf
unsloth/Qwen3.8-27B-GGUF · 2 tested configurations · 1 GPU
- L4 · 24 GBQ4_K_M · 8K → 2K tokens
  Output throughput · includes prefill · 1 request
  14.1tok/s
  Observed GPU occupancy
  17GiB
  First token
  279 ms
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  llamacpp
  Protocol
  wbench-fixed-http-v1
  Dataset
  synthetic-tokenizer-v1:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 14.09 145.5 s 279 ms 17.05 71.9 1 4bc32a6c ↗
- 1 other test configurations on this GPU
  - Q4_K_M · 32K → 2K tokens
    Output throughput · includes prefill · 1 request
    12.9tok/s
    Observed GPU occupancy
    17GiB
    First token
    371 ms
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    llamacpp
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 12.94 158.32 s 371 ms 17.05 71.8 1 d8a4b66b ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA
Qwen/Qwen3.8-Flash-Next · 6 tested configurations · 1 GPU
- RTX PRO 6000 Blackwell Server Edition · 96 GBQ4_K_XL · 8K → 2K tokensCPU + GPU hybrid
  Output throughput · includes prefill · 1 request
  125tok/s
  Observed GPU occupancy
  79.7GiB
  First token
  1.8 s
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  strata
  Protocol
  wbench-fixed-http-v1
  Dataset
  synthetic-tokenizer-v1:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 125.06 23.75 s 1.8 s 79.71 279.5 1 18c8631f ↗
- 5 other test configurations on this GPU
  - NVFP4 · 8K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    151tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    1.1 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 151.39 15.46 s 1.1 s 71.22 292.2 1 60a45fd1 ↗
  - NVFP4 · 8K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    135tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    1.3 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 134.63 19.87 s 1.3 s 71.22 280 1 f2b52c15 ↗
  - NVFP4 · 32K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    114tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    4.8 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 114.25 20.98 s 4.8 s 71.22 316.1 1 4342000e ↗
  - Q4_K_XL · 32K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    106tok/s
    Observed GPU occupancy
    79.7GiB
    First token
    6.7 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 106.11 23.97 s 6.7 s 79.71 308 1 66369e97 ↗
  - NVFP4 · 32K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    124tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    4.1 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 124.44 18.26 s 4.1 s 71.22 319.5 1 4a41258c ↗

Qwen3-Embedding-0.6B

Qwen · 5 tested configurations · 3 GPUs

RTX 4090 · 24 GBfp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 259vectors/sec
Best measured · Batch 16: 2,018vectors/sec
Observed GPU occupancy: 1.8GiB
P95 latency: 4 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	258.52	4 ms	—	1.84	151.6	1	3ba3d2cb ↗
B4	921.44	4 ms	—	1.84	159.5	1	3ba3d2cb ↗
B16	2,017.74	8 ms	—	1.84	133	1	3ba3d2cb ↗

RTX PRO 6000 Blackwell Server Edition · 96 GBfp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 232vectors/sec
Best measured · Batch 32: 3,763vectors/sec
Observed GPU occupancy: 2.2GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	232.18	5 ms	—	2.22	155	1	65594583 ↗
B8	1,601.21	5 ms	—	2.22	191.2	1	65594583 ↗
B32	3,763.07	9 ms	—	2.22	208.7	1	65594583 ↗

2 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s

Throughput · Batch 1: 218vectors/sec
Best measured · Batch 32: 3,617vectors/sec
Observed GPU occupancy: 2.2GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	217.92	5 ms	—	2.22	156.5	1	a7d85bcf ↗
B8	1,494.89	5 ms	—	2.22	185.4	1	a7d85bcf ↗
B32	3,616.74	9 ms	—	2.22	206.8	1	a7d85bcf ↗

nf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s

Throughput · Batch 1: 196vectors/sec
Best measured · Batch 32: 3,091vectors/sec
Observed GPU occupancy: 1.6GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	195.8	5 ms	—	1.61	174.8	1	b9501833 ↗
B8	1,156.9	7 ms	—	1.61	196	1	b9501833 ↗
B32	3,091.08	10 ms	—	1.61	207.4	1	b9501833 ↗

V100-PCIE-32GB · 32 GBfp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 95.3vectors/sec
Best measured · Batch 16: 771vectors/sec
Observed GPU occupancy: 1.7GiB
P95 latency: 11 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	95.33	11 ms	—	1.75	70.8	1	f91802d1 ↗
B4	308.67	13 ms	—	1.75	90.1	1	f91802d1 ↗
B16	771.26	21 ms	—	1.75	99.2	1	f91802d1 ↗

Kokoro-82M
hexgrad · 6 tested configurations · 4 GPUs
- RTX 3080 Ti · 12 GBfp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
  Throughput · Batch 1
  49.2audio-sec/sec
  Best measured · Batch 2
  52.8audio-sec/sec
  Observed GPU occupancy
  1GiB
  P95 latency
  73 ms
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  kokoro · PyTorch 2.12.1+cu130
  Protocol
  audio-native-v1
  Dataset
  kokoro-english-v1
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 49.18 73 ms — 1 158.9 1 3c7db092 ↗
  B2 52.8 140 ms — 1.06 194.7 1 3c7db092 ↗
- RTX PRO 6000 Blackwell Server Edition · 96 GBfp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
  Throughput · Batch 1
  120audio-sec/sec
  Best measured · Batch 4
  125audio-sec/sec
  Observed GPU occupancy
  2GiB
  P95 latency
  29 ms
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  kokoro · PyTorch 2.13.0+cu130
  Protocol
  audio-native-v1
  Dataset
  kokoro-english-v1
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 120.3 29 ms — 2 152.1 2 061d047a ↗
  B4 125.19 124 ms — 2.03 169.7 2 061d047a ↗
- V100-PCIE-32GB · 32 GBfp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
  Throughput · Batch 1
  36audio-sec/sec
  Best measured · Batch 2
  37.8audio-sec/sec
  Observed GPU occupancy
  1.4GiB
  P95 latency
  105 ms
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  kokoro · PyTorch 2.12.1+cu126
  Protocol
  audio-native-v1
  Dataset
  kokoro-english-v1
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 35.95 105 ms — 1.45 76.3 11 b40f8533 ↗
  B2 37.83 237 ms — 1.45 82.6 11 b40f8533 ↗
Show 1 more GPUs
- vGPU-32GB · 32 GBfp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
  Throughput · Batch 1
  43.4audio-sec/sec
  Best measured · Batch 4
  44.9audio-sec/sec
  Observed GPU occupancy
  1GiB
  P95 latency
  101 ms
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  kokoro
  Protocol
  audio-native-v1
  Dataset
  kokoro-english-v1
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 43.43 101 ms — 0.99 64.1 1 d8e4b3ce ↗
  B4 44.88 396 ms — 1.03 69.1 1 d8e4b3ce ↗
- 2 other test configurations on this GPU
  - fp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
    Throughput · Batch 1
    40.4audio-sec/sec
    Best measured · Batch 4
    41.8audio-sec/sec
    Observed GPU occupancy
    1GiB
    P95 latency
    104 ms
    Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 batch sizes
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    kokoro · PyTorch 2.13.0+cu130
    Protocol
    audio-native-v1
    Dataset
    kokoro-english-v1
    Workload
    tts
    Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 40.36 104 ms — 0.99 60.5 1 c0488108 ↗
    B4 41.8 426 ms — 1.03 65.9 1 c0488108 ↗
  - fp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
    Throughput · Batch 1
    76.7audio-sec/sec
    Best measured · Batch 2
    79.8audio-sec/sec
    Observed GPU occupancy
    1GiB
    P95 latency
    45 ms
    Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 batch sizes
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    kokoro · PyTorch 2.13.0+cu130
    Protocol
    audio-native-v1
    Dataset
    kokoro-english-v1
    Workload
    tts
    Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 76.72 45 ms — 1 83.5 1 d90a5c29 ↗
    B2 79.84 100 ms — 1.03 100.8 1 d90a5c29 ↗
whisper-tiny
openai · 7 tested configurations · 4 GPUs
- RTX 3080 Ti · 12 GBfp16 · Audio → Text · asr_prompts · Test duration 2 s
  Throughput · Batch 1
  65.7audio-sec/sec
  Best measured · Batch 4
  91.8audio-sec/sec
  Observed GPU occupancy
  0.4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  faster-whisper · PyTorch 2.12.1+cu130
  Protocol
  audio-native-v1
  Dataset
  asr_prompts
  Workload
  asr
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 65.65 — — 0.41 134.9 1 3f8b97e6 ↗
  B4 91.84 — — 0.4 143.6 1 3f8b97e6 ↗
- RTX PRO 6000 Blackwell Server Edition · 96 GBfp16 · Audio → Text · asr_prompts · Test duration 2 s
  Throughput · Batch 1
  233audio-sec/sec
  Best measured · Batch 4
  364audio-sec/sec
  Observed GPU occupancy
  1.4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  faster-whisper · PyTorch 2.14.1
  Protocol
  audio-native-v1
  Dataset
  asr_prompts
  Workload
  asr
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 232.81 — — 1.42 113.5 2 ba5a45ac ↗
  B4 364.19 — — 1.43 115.8 2 ba5a45ac ↗
- V100-PCIE-32GB · 32 GBfp16 · Audio → Text · asr_prompts · Test duration 1 s
  Throughput · Batch 1
  105audio-sec/sec
  Best measured · Batch 4
  156audio-sec/sec
  Observed GPU occupancy
  0.5GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  faster-whisper · PyTorch 2.12.1+cu126
  Protocol
  audio-native-v1
  Dataset
  asr_prompts
  Workload
  asr
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 104.96 — — 0.51 51.5 1 64035c7b ↗
  B4 155.93 — — 0.52 50.1 1 64035c7b ↗
- 1 other test configurations on this GPU
  - fp16 · Audio → Text · asr_prompts · Test duration 5 s
    Throughput · Batch 1
    90.9audio-sec/sec
    Best measured · Batch 4
    132audio-sec/sec
    Observed GPU occupancy
    0.5GiB
    P95 latency
    Not recorded
    Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 batch sizes
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    faster-whisper · PyTorch 2.12.1+cu126
    Protocol
    audio-native-v1
    Dataset
    asr_prompts
    Workload
    asr
    Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 90.94 — — 0.51 51.5 10 e4ef00c1 ↗
    B4 132.16 — — 0.52 51.2 10 e4ef00c1 ↗
Show 1 more GPUs
- vGPU-32GB · 32 GBfp16 · Audio → Text · asr_prompts
  Throughput · Batch 1
  73.7audio-sec/sec
  Observed GPU occupancy
  0.4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  faster-whisper · PyTorch 2.13.0+cu130
  Protocol
  audio-native-v1
  Dataset
  asr_prompts
  Workload
  asr
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 73.68 — — 0.38 46.8 1 7d182de2 ↗
  B4 68.2 — — 0.38 49.2 1 7d182de2 ↗
- 2 other test configurations on this GPU
  - fp16 · Audio → Text · asr_prompts · Test duration 4 s
    Throughput · Batch 1
    122audio-sec/sec
    Best measured · Batch 4
    168audio-sec/sec
    Observed GPU occupancy
    0.4GiB
    P95 latency
    Not recorded
    Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 batch sizes
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    faster-whisper
    Protocol
    audio-native-v1
    Dataset
    asr_prompts
    Workload
    asr
    Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 121.9 — — 0.42 73.6 1 7aaa1f0d ↗
    B4 168.25 — — 0.42 77.4 1 7aaa1f0d ↗
  - fp16 · Audio → Text · asr_prompts · Test duration 1 s
    Throughput · Batch 1
    137audio-sec/sec
    Best measured · Batch 4
    205audio-sec/sec
    Observed GPU occupancy
    0.4GiB
    P95 latency
    Not recorded
    Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 batch sizes
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    faster-whisper · PyTorch 2.13.0+cu130
    Protocol
    audio-native-v1
    Dataset
    asr_prompts
    Workload
    asr
    Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 137.01 — — 0.38 54.3 1 4b83f25d ↗
    B4 205.27 — — 0.38 66.3 1 4b83f25d ↗

Qwen3-Embedding-4B

Qwen · 4 tested configurations · 2 GPUs

RTX 4090 · 24 GBfp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 3 s
Throughput · Batch 16
498vectors/sec
Observed GPU occupancy
8.2GiB
P95 latency
33 ms
Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
Test details
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
sentence-transformers · PyTorch 2.12.1+cu126
Protocol
native-forward-v1
Dataset
embedding-mteb-tasks-v1
Workload
embedding
Batch size Mean throughput (vectors/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B16 497.84 33 ms — 8.22 262.8 1 0d6d756e ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 81.2vectors/sec
Best measured · Batch 16: 507vectors/sec
Observed GPU occupancy: 8.3GiB
P95 latency: 12 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	81.19	12 ms	—	8.34	258.9	1	4f6f3fbb ↗
B4	279.31	14 ms	—	8.34	270.5	1	4f6f3fbb ↗
B16	507.04	32 ms	—	8.34	264.4	1	4f6f3fbb ↗

V100-PCIE-32GB · 32 GBfp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s

Throughput · Batch 1: 41vectors/sec
Best measured · Batch 16: 229vectors/sec
Observed GPU occupancy: 8.2GiB
P95 latency: 25 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	40.96	25 ms	—	8.25	119.3	1	65a1050f ↗
B4	100.37	40 ms	—	8.25	149.9	1	65a1050f ↗
B16	228.92	70 ms	—	8.25	178.1	1	65a1050f ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 40.9vectors/sec
Best measured · Batch 16: 228vectors/sec
Observed GPU occupancy: 8.2GiB
P95 latency: 25 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu126
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	40.94	25 ms	—	8.25	123.4	1	cd3861d6 ↗
B4	100.23	40 ms	—	8.25	146.9	1	cd3861d6 ↗
B16	228.35	70 ms	—	8.25	180.4	1	cd3861d6 ↗

SenseVoiceSmall
FunAudioLLM · 1 tested configuration · 1 GPU
- RTX 3080 Ti · 12 GBfp16 · Audio → Text · asr_prompts
  Throughput · Batch 1
  360audio-sec/sec
  Best measured · Batch 2
  365audio-sec/sec
  Observed GPU occupancy
  1.3GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  funasr · PyTorch 2.12.1+cu130
  Protocol
  audio-native-v1
  Dataset
  asr_prompts
  Workload
  asr
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 359.69 — — 1.27 101.7 1 c6b5f31d ↗
  B2 364.87 — — 1.27 101.6 1 c6b5f31d ↗

Sortable tables by exact test condition · 335 rows

Decision · 1K → 1K tokens · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	vLLM	10.19	104 ms	—	18.17	69.8	1	caef7154 ↗

Decision · 1K → 1K tokens · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	vLLM	10.88	388 ms	—	18.49	152.8	1	caef7154 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 1 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	clef-native	10.26	104 ms	—	18.17	67.9	1	1d1661ad ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 1 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	clef-native	10.89	389 ms	—	18.49	156	1	1d1661ad ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-nativePyTorch 2.11.0+cu130	50.25	20 ms	—	3.17	161.3	1	df141700 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-nativePyTorch 2.11.0+cu130	50.28	20 ms	—	2.29	165	1	8c1985ea ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0+cu130	50.83	20 ms	—	2.36	169	2	18b06f63 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	51.51	20 ms	—	2.6	164.2	1	7f2e07c0 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	clef-nativePyTorch 2.13.0+cu130	10.56	99 ms	—	18.17	206.1	2	63f44036 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	clef-native	10.56	103 ms	—	18.17	196.8	1	4eb73a4d ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1.1-0.8B-Instant	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-transformersPyTorch 2.11.0+cu130	18.69	54 ms	—	2.73	134.5	2	a4a69815 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1.1-0.8B-Instant	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-transformersPyTorch 2.11.0	18.83	54 ms	—	2.54	125.5	1	c342a9ca ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	50.29	21 ms	—	4.9	216.2	1	e4bcd6a3 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	clef-nativePyTorch 2.13.0+cu130	11.04	402 ms	—	18.55	244.4	2	63f44036 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	vGPU-32GB · 32 GB	bf16	clef-native	11.1	390 ms	—	18.56	235.9	1	4eb73a4d ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0+cu130	47.71	338 ms	—	19.55	499.5	2	82d98b36 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 17 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0+cu130	346.72	50 ms	—	3.07	378	1	6ccb367c ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	31.83	32 ms	—	18.95	388.5	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	39.27	93 ms	—	18.95	449.5	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	5.41	8.21 s	—	19.28	233.4	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	15.43	9.66 s	—	21.89	324.3	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.45	134 ms	—	60.88	232.8	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.47	537 ms	—	60.89	249.7	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.48	2.14 s	—	60.89	252.8	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.49	8.56 s	—	60.89	257.2	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	12.32	82 ms	—	52.4	469.1	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	1.4	8.09 s	—	52.4	225.7	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	4.33	9.1 s	—	52.98	308.5	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	9.39	12.46 s	—	57.95	435.9	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	52.8	19 ms	—	2.36	172.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 2 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	95.72	21 ms	—	2.38	186.5	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	177.64	23 ms	—	2.43	242	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 8 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	306.42	26 ms	—	2.57	344.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 12 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	322.68	37 ms	—	37.46	371.9	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 13 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	336	39 ms	—	37.46	380.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 14 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	344.61	41 ms	—	37.46	392.6	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 15 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	345.75	44 ms	—	37.46	389.5	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	345.28	47 ms	—	2.85	383.9	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 17 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	349.83	49 ms	—	37.46	397.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 18 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	327.34	55 ms	—	37.46	383.4	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 20 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	333.21	60 ms	—	37.46	402.6	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 24 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	331.98	73 ms	—	37.46	401.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 28 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	327.01	86 ms	—	37.46	399.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 32 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	331.67	97 ms	—	3.41	397.7	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 48 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	326.46	148 ms	—	37.46	405.9	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	318.22	202 ms	—	4.51	403.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 128 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	298.77	430 ms	—	6.71	405.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 256 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	298.58	859 ms	—	11.11	407.3	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 512 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	299.67	1.71 s	—	19.9	411.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 1024 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	297.46	3.64 s	—	37.46	410.6	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	52.04	20 ms	—	2.36	162.6	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 2 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	96.21	21 ms	—	2.38	213.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	177.08	23 ms	—	2.43	243.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 8 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	299.06	27 ms	—	2.57	341.2	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 12 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	306.24	40 ms	—	36.98	349.6	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 14 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	322.36	43 ms	—	36.98	365.9	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 15 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	328.14	46 ms	—	36.98	369.8	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	333.92	48 ms	—	3.19	368.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 17 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	331.9	52 ms	—	36.98	379.5	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 18 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	310.84	58 ms	—	36.98	365.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 20 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	316.13	64 ms	—	36.98	375.1	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 24 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	312.87	77 ms	—	36.98	377.5	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 28 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	309.61	91 ms	—	36.98	379.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 32 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	319.44	101 ms	—	3.44	376.3	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 40 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	307.24	131 ms	—	36.98	379.6	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 48 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	306.75	159 ms	—	36.98	380.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	304.06	213 ms	—	4.2	383.7	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 128 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	285.17	453 ms	—	6.23	387.8	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 256 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	284.61	904 ms	—	10.63	390.3	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 512 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	282.22	2.04 s	—	19.42	390.1	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 1024 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	286.53	3.58 s	—	36.98	396.2	1	156910e7 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 3 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	RTX 4090 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	497.84	33 ms	—	8.22	262.8	1	0d6d756e ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	251.11	4 ms	—	1.05	99.4	1	f489c748 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	40.96	25 ms	—	8.25	119.3	1	65a1050f ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	924.83	4 ms	—	1.05	111.5	1	f489c748 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	100.37	40 ms	—	8.25	149.9	1	65a1050f ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	2,710.19	6 ms	—	1.05	114.9	1	f489c748 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	228.92	70 ms	—	8.25	178.1	1	65a1050f ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	217.92	5 ms	—	2.22	156.5	1	a7d85bcf ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	195.8	5 ms	—	1.61	174.8	1	b9501833 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 8 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	1,494.89	5 ms	—	2.22	185.4	1	a7d85bcf ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 8 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	1,156.9	7 ms	—	1.61	196	1	b9501833 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 32 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	3,616.74	9 ms	—	2.22	206.8	1	a7d85bcf ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 32 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	3,091.08	10 ms	—	1.61	207.4	1	b9501833 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 3 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	174.88	6 ms	—	0.85	53.1	6	7ad6a821 ↗
embeddinggemma-2	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	90.02	11 ms	—	0.75	54.6	2	43c67a74 ↗
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	189.23	6 ms	—	1.33	145.5	1	c673556b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 3 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	108.65	9 ms	—	0.89	49.3	2	2846de88 ↗
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	206.79	5 ms	—	1.48	119.3	1	196678d3 ↗
embeddinggemma-2	L4 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	177.46	6 ms	—	0.99	49.8	6	e0700e04 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 2 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	90.07	11 ms	—	0.75	56.2	2	2f9465c4 ↗
embeddinggemma-2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	174.95	6 ms	—	0.85	54.8	5	29b19870 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 2 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	108.76	9 ms	—	0.89	50.3	2	cfe051ff ↗
embeddinggemma-2	L4 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	177.21	6 ms	—	0.99	50.5	4	d74f69cc ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
bge-small-zh-v1.5	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	1,169.81	1 ms	—	0.29	39	1	0fb03772 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-embedding-97m-multilingual-r2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	791.85	1 ms	—	0.51	37.4	1	d396c54b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-embedding-97m-multilingual-r2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	464.26	2 ms	—	0.47	45.4	1	37c260e4 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX 4090 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	258.52	4 ms	—	1.84	151.6	1	3ba3d2cb ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	232.18	5 ms	—	2.22	155	1	65594583 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	95.33	11 ms	—	1.75	70.8	1	f91802d1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	RTX 4090 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	81.19	12 ms	—	8.34	258.9	1	4f6f3fbb ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	40.94	25 ms	—	8.25	123.4	1	cd3861d6 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 3 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	571.03	7 ms	—	0.85	57	6	7ad6a821 ↗
embeddinggemma-2	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	303.35	13 ms	—	0.75	60.1	2	43c67a74 ↗
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	713.32	6 ms	—	1.33	142.8	1	c673556b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 3 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	365.65	11 ms	—	0.89	57.9	2	2846de88 ↗
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	764.95	6 ms	—	1.48	141.4	1	196678d3 ↗
embeddinggemma-2	L4 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	650.16	6 ms	—	0.99	56.9	6	e0700e04 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 2 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	302.9	13 ms	—	0.75	61.5	2	2f9465c4 ↗
embeddinggemma-2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	572.6	7 ms	—	0.85	58.4	5	29b19870 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 2 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	365.82	11 ms	—	0.89	58.1	2	cfe051ff ↗
embeddinggemma-2	L4 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	648.08	6 ms	—	0.99	57.1	4	d74f69cc ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
bge-small-zh-v1.5	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	2,301.06	2 ms	—	0.29	32.5	1	0fb03772 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-embedding-97m-multilingual-r2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	2,816.45	1 ms	—	0.51	37.9	1	d396c54b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-embedding-97m-multilingual-r2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	1,713.26	2 ms	—	0.47	49.4	1	37c260e4 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX 4090 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	921.44	4 ms	—	1.84	159.5	1	3ba3d2cb ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	308.67	13 ms	—	1.75	90.1	1	f91802d1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	RTX 4090 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	279.31	14 ms	—	8.34	270.5	1	4f6f3fbb ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	100.23	40 ms	—	8.25	146.9	1	cd3861d6 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 8 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	1,601.21	5 ms	—	2.22	191.2	1	65594583 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 3 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	1,639.08	10 ms	—	0.85	59.8	6	7ad6a821 ↗
embeddinggemma-2	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	814.64	20 ms	—	0.75	64.1	2	43c67a74 ↗
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	2,571.53	7 ms	—	1.33	162.7	1	c673556b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 3 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	1,004.11	16 ms	—	0.89	63.9	2	2846de88 ↗
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	2,938.63	6 ms	—	1.48	161.8	1	196678d3 ↗
embeddinggemma-2	L4 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	2,039.17	8 ms	—	0.99	58.4	6	e0700e04 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 2 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	809.43	20 ms	—	0.75	64	2	2f9465c4 ↗
embeddinggemma-2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	1,649.62	10 ms	—	0.85	60.7	5	29b19870 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 2 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	1,002.27	16 ms	—	0.89	63.9	2	cfe051ff ↗
embeddinggemma-2	L4 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	2,035.67	8 ms	—	0.99	58.1	4	d74f69cc ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
bge-small-zh-v1.5	T4 · 16 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	5,188.01	3 ms	—	0.29	32.5	1	0fb03772 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-embedding-97m-multilingual-r2	L4 · 24 GB	nf4	sentence-transformersPyTorch 2.12.1+cu126	8,880.86	2 ms	—	0.51	38.9	1	d396c54b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-embedding-97m-multilingual-r2	T4 · 16 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	4,083.56	4 ms	—	0.47	38.3	1	37c260e4 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX 4090 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	2,017.74	8 ms	—	1.84	133	1	3ba3d2cb ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	771.26	21 ms	—	1.75	99.2	1	f91802d1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	RTX 4090 · 24 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	507.04	32 ms	—	8.34	264.4	1	4f6f3fbb ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-4B	V100-PCIE-32GB · 32 GB	fp16	sentence-transformersPyTorch 2.12.1+cu126	228.35	70 ms	—	8.25	180.4	1	cd3861d6 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 32 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	3,763.07	9 ms	—	2.22	208.7	1	65594583 ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformers	39.4	26 ms	—	—	—	1	2c97c869 ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformers	119.85	44 ms	—	—	—	1	2c97c869 ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 6 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	38.33	31 ms	—	0.82	50.5	4	2c17d254 ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 6 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformers	38.27	32 ms	—	0.82	48.6	1	47f71b64 ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 6 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	138.64	33 ms	—	0.82	54.2	4	2c17d254 ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 6 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformers	137.82	35 ms	—	0.82	51.9	1	47f71b64 ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 12 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	38.8	27 ms	—	0.82	48.8	1	dc5960cd ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 12 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	141.65	29 ms	—	0.82	52.4	1	dc5960cd ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 12 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	553.54	30 ms	—	0.87	56.4	1	dc5960cd ↗

Embedding · Text → Vectors · embedding-sanity-v1 · Test duration 12 s · Batch 64 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	vGPU-32GB · 32 GB	bf16	sentence-transformersPyTorch 2.13.0+cu130	1,766.47	38 ms	—	1.03	65.3	1	dc5960cd ↗

Speech recognition · Audio → Text · asr_prompts · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	3,143.16	—	—	1.64	95.8	1	d77a0cfd ↗

Speech recognition · Audio → Text · asr_prompts · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	vGPU-32GB · 32 GB	fp16	faster-whisperPyTorch 2.13.0+cu130	73.68	—	—	0.38	46.8	1	7d182de2 ↗

Speech recognition · Audio → Text · asr_prompts · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
SenseVoiceSmall	RTX 3080 Ti · 12 GB	fp16	funasrPyTorch 2.12.1+cu130	359.69	—	—	1.27	101.7	1	c6b5f31d ↗

Speech recognition · Audio → Text · asr_prompts · Batch 2 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
SenseVoiceSmall	RTX 3080 Ti · 12 GB	fp16	funasrPyTorch 2.12.1+cu130	364.87	—	—	1.27	101.6	1	c6b5f31d ↗

Speech recognition · Audio → Text · asr_prompts · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	vGPU-32GB · 32 GB	fp16	faster-whisperPyTorch 2.13.0+cu130	68.2	—	—	0.38	49.2	1	7d182de2 ↗

Speech recognition · Audio → Text · asr_prompts · Batch 8 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,162.53	—	—	1.64	105.2	1	d77a0cfd ↗

Speech recognition · Audio → Text · asr_prompts · Batch 16 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,581.45	—	—	1.92	111.9	1	d77a0cfd ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 1 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	V100-PCIE-32GB · 32 GB	fp16	faster-whisperPyTorch 2.12.1+cu126	104.96	—	—	0.51	51.5	1	64035c7b ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 1 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	vGPU-32GB · 32 GB	fp16	faster-whisperPyTorch 2.13.0+cu130	137.01	—	—	0.38	54.3	1	4b83f25d ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 1 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	V100-PCIE-32GB · 32 GB	fp16	faster-whisperPyTorch 2.12.1+cu126	155.93	—	—	0.52	50.1	1	64035c7b ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 1 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	vGPU-32GB · 32 GB	fp16	faster-whisperPyTorch 2.13.0+cu130	205.27	—	—	0.38	66.3	1	4b83f25d ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	RTX 3080 Ti · 12 GB	fp16	faster-whisperPyTorch 2.12.1+cu130	65.65	—	—	0.41	134.9	1	3f8b97e6 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	faster-whisperPyTorch 2.14.1	232.81	—	—	1.42	113.5	2	ba5a45ac ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	RTX 3080 Ti · 12 GB	fp16	faster-whisperPyTorch 2.12.1+cu130	91.84	—	—	0.4	143.6	1	3f8b97e6 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	faster-whisperPyTorch 2.14.1	364.19	—	—	1.43	115.8	2	ba5a45ac ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 4 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	vGPU-32GB · 32 GB	fp16	faster-whisper	121.9	—	—	0.42	73.6	1	7aaa1f0d ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 4 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	vGPU-32GB · 32 GB	fp16	faster-whisper	168.25	—	—	0.42	77.4	1	7aaa1f0d ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 5 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	V100-PCIE-32GB · 32 GB	fp16	faster-whisperPyTorch 2.12.1+cu126	90.94	—	—	0.51	51.5	10	e4ef00c1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 5 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	V100-PCIE-32GB · 32 GB	fp16	faster-whisperPyTorch 2.12.1+cu126	132.16	—	—	0.52	51.2	10	e4ef00c1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	2,662.33	—	—	1.64	122.1	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 2 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	4,844.87	—	—	1.67	134.9	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	6,237.58	—	—	1.75	148	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 8 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	7,907.41	—	—	1.9	165.1	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 16 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	9,083.29	—	—	2.33	194.8	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 32 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	9,193.57	—	—	3.25	243.5	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 64 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,622.98	—	—	5.09	345.5	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 128 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,355.43	—	—	10.1	451	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 256 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,343	—	—	20.09	477.6	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 512 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,391.94	—	—	40.63	492.3	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 1024 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,387.34	—	—	80.89	501.3	1	3be7f67e ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 9 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
IndexTTS-2.5	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	indexttsPyTorch 2.8.0+cu128	4.24	—	—	6.48	221.7	1	a46ec2c4 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 19 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
magpie_tts_multilingual_357m	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	nemo-magpiePyTorch 2.13.0	3.57	—	—	4.01	178.8	1	666efb0f ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 43 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-TTS-12Hz-0.6B-CustomVoice	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	qwen-ttsPyTorch 2.14.1	1.6	—	—	3.34	154.4	1	ebb272f8 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 43 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-TTS-12Hz-0.6B-CustomVoice	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	qwen-ttsPyTorch 2.14.1	5.22	—	—	4.92	161	1	ebb272f8 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	RTX 3080 Ti · 12 GB	fp32	kokoroPyTorch 2.12.1+cu130	49.18	73 ms	—	1	158.9	1	3c7db092 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp32	kokoroPyTorch 2.13.0+cu130	120.3	29 ms	—	2	152.1	2	061d047a ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	V100-PCIE-32GB · 32 GB	fp32	kokoroPyTorch 2.12.1+cu126	35.95	105 ms	—	1.45	76.3	11	b40f8533 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	vGPU-32GB · 32 GB	fp32	kokoro	43.43	101 ms	—	0.99	64.1	1	d8e4b3ce ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	vGPU-32GB · 32 GB	fp32	kokoroPyTorch 2.13.0+cu130	40.36	104 ms	—	0.99	60.5	1	c0488108 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	vGPU-32GB · 32 GB	fp32	kokoroPyTorch 2.13.0+cu130	76.72	45 ms	—	1	83.5	1	d90a5c29 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 2 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	RTX 3080 Ti · 12 GB	fp32	kokoroPyTorch 2.12.1+cu130	52.8	140 ms	—	1.06	194.7	1	3c7db092 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 2 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	V100-PCIE-32GB · 32 GB	fp32	kokoroPyTorch 2.12.1+cu126	37.83	237 ms	—	1.45	82.6	11	b40f8533 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 2 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	vGPU-32GB · 32 GB	fp32	kokoroPyTorch 2.13.0+cu130	79.84	100 ms	—	1.03	100.8	1	d90a5c29 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp32	kokoroPyTorch 2.13.0+cu130	125.19	124 ms	—	2.03	169.7	2	061d047a ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	vGPU-32GB · 32 GB	fp32	kokoro	44.88	396 ms	—	1.03	69.1	1	d8e4b3ce ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	vGPU-32GB · 32 GB	fp32	kokoroPyTorch 2.13.0+cu130	41.8	426 ms	—	1.03	65.9	1	c0488108 ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX 5090 · 32 GB	NVFP4	vllm	70.82	29.01 s	633 ms	27.29	405.5	1	bad294c0 ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-UD-Q4_K_M.gguf	L4 · 24 GB	Q4_K_M	llamacpp	14.09	145.5 s	279 ms	17.05	71.9	1	4bc32a6c ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	Q4_K_XL	strata	125.06	23.75 s	1.8 s	79.71	279.5	1	18c8631f ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	151.39	15.46 s	1.1 s	71.22	292.2	1	60a45fd1 ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	134.63	19.87 s	1.3 s	71.22	280	1	f2b52c15 ↗

Text generation · 8K → 2K tokens · No cache reuse · 1 request · tok/s · 3 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	214.6	9.63 s	298 ms	86.28	303.8	1	86d36e4f ↗
Qwen3-4B-AWQ	L4 · 24 GB	AWQ	vllmvLLM 0.29.0	55.3	37.17 s	1.6 s	19.54	71.9	1	9301a9ba ↗
Qwen3-4B-AWQ	T4 · 16 GB	AWQ	vllmvLLM 0.29.0	15.39	133.36 s	28.5 s	12.74	67.9	1	fcfe2d64 ↗

Text generation · 8K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 3 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	516.94	16.14 s	872 ms	86.28	339.5	1	86d36e4f ↗
Qwen3-4B-AWQ	L4 · 24 GB	AWQ	vllmvLLM 0.29.0	109.43	78.88 s	3.1 s	19.55	71.9	1	9301a9ba ↗
Qwen3-4B-AWQ	T4 · 16 GB	AWQ	vllmvLLM 0.29.0	20.31	468.13 s	53.1 s	12.74	67.5	1	fcfe2d64 ↗

Text generation · 8K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 2 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	L4 · 24 GB	AWQ	vllmvLLM 0.29.0	57.39	35.7 s	60 ms	19.55	71.8	1	7e2cbfda ↗
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	222.74	9.19 s	28 ms	86.28	288.8	1	adbfef8a ↗

Text generation · 8K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 2 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	L4 · 24 GB	AWQ	vllmvLLM 0.29.0	175.39	46.75 s	114 ms	19.55	71.9	1	7e2cbfda ↗
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	741.85	11.05 s	35 ms	86.28	335	1	adbfef8a ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX 5090 · 32 GB	NVFP4	vllm	62.48	32.79 s	3.6 s	27.29	433.3	1	746bbc39 ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-UD-Q4_K_M.gguf	L4 · 24 GB	Q4_K_M	llamacpp	12.94	158.32 s	371 ms	17.05	71.8	1	d8a4b66b ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	114.25	20.98 s	4.8 s	71.22	316.1	1	4342000e ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	Q4_K_XL	strata	106.11	23.97 s	6.7 s	79.71	308	1	66369e97 ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	124.44	18.26 s	4.1 s	71.22	319.5	1	4a41258c ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	153.44	13.39 s	753 ms	85.25	343.4	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.78	38.8 s	3.3 s	87.37	337.3	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	55.59	36.91 s	3.2 s	85.98	341	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	208.65	19.64 s	1.1 s	85.25	360.4	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	87.2	48.71 s	4.9 s	87.37	343.4	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	91.37	46.48 s	4.6 s	85.98	342.9	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	259.19	33.21 s	1.6 s	85.25	398.9	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	142.14	64.91 s	7 s	87.37	375.3	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	149.21	62.63 s	6.5 s	85.98	372.9	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	187.69	79.85 s	8.1 s	87.37	402.8	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	196.73	76.24 s	7.8 s	85.98	398.9	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	202.31	85.21 s	8.7 s	87.37	411.8	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	211.87	81.41 s	8.3 s	85.98	408.7	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	295.37	59.5 s	2.1 s	85.25	422	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	215.05	94.52 s	9.3 s	87.37	421.4	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	224.61	90.5 s	8.9 s	85.98	414.8	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 12 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	311.33	86.14 s	2.7 s	85.25	432.4	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 16 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	319.17	112.4 s	3.3 s	85.25	438	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	161.57	12.72 s	71 ms	85.25	327.8	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	56.96	36.17 s	222 ms	87.37	322.6	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.09	35.88 s	231 ms	86.63	319	1	37f29252 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	56.4	36.51 s	244 ms	86.35	319.1	1	791b887d ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	60.1	34.34 s	217 ms	85.98	323	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	59.68	34.59 s	230 ms	86.82	323.3	1	83d37469 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	60.47	33.87 s	228 ms	86.86	320.3	1	cbf1431c ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	269.43	15.21 s	86 ms	85.25	349.1	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	104.14	39.34 s	355 ms	87.37	305.9	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	109.3	37.48 s	349 ms	85.98	310.8	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	460.41	17.83 s	106 ms	85.25	427.3	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	198.42	41.3 s	704 ms	87.37	319.3	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	197.8	41.44 s	650 ms	86.63	330.2	1	37f29252 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	196.45	41.73 s	654 ms	86.35	315.5	1	791b887d ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	209.23	39.18 s	686 ms	85.98	324.8	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	207.31	39.54 s	664 ms	86.82	320	1	83d37469 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	207.86	39.44 s	636 ms	86.86	333.6	1	cbf1431c ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	291.12	42.23 s	1 s	87.37	342.1	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	306.94	40.06 s	992 ms	85.98	336	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	333.32	43.04 s	1.1 s	87.37	349.1	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	351.24	40.83 s	1.1 s	85.98	342.6	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	644.45	25.52 s	127 ms	85.25	486.9	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	374.04	43.82 s	1.3 s	87.37	353.2	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	372.88	43.98 s	1.3 s	86.63	367.3	1	37f29252 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	370.68	44.24 s	1.3 s	86.36	348.1	1	791b887d ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	392.93	41.71 s	1.3 s	85.98	348.2	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	394.04	41.6 s	1.2 s	86.82	346.1	1	83d37469 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	394.52	41.6 s	1.3 s	86.86	361.6	1	cbf1431c ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 16 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	771.04	42.77 s	184 ms	85.25	533.9	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 17 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	812.79	43.14 s	195 ms	85.25	550.4	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 18 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	856.64	43.41 s	196 ms	85.25	568.9	1	75198c63 ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	101.26	20.26 s	2.3 s	85.25	369.1	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	44.86	45.67 s	8.6 s	87.37	360.5	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	47.15	43.45 s	8.4 s	85.98	364.4	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	123.11	33.41 s	3.6 s	85.25	404.8	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	67.59	63.21 s	12.7 s	87.37	379.1	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	70.39	60.71 s	12.3 s	85.98	381.3	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	84.95	84.07 s	14.6 s	87.37	398.5	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	88.36	80.85 s	14.2 s	85.98	403.6	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	140.19	64.26 s	4.5 s	85.25	430	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	96.77	105.62 s	16.3 s	87.37	415.8	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	100.66	101.65 s	15.8 s	85.98	419.2	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	142.08	97.79 s	5.3 s	85.25	430.4	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	150.73	122.37 s	6.3 s	85.25	446.7	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	113.51	18.06 s	135 ms	85.25	341.3	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	54.72	37.56 s	329 ms	87.37	322.4	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	54.04	38.07 s	361 ms	86.36	320.7	1	417f69be ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	54.54	37.55 s	359 ms	86.63	324.9	1	7d1a7c8e ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.45	35.8 s	326 ms	85.98	326.1	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.08	36.15 s	358 ms	86.82	326	1	275a096d ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.67	35.51 s	357 ms	86.86	325.8	1	fe8141d2 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	153.3	26.73 s	161 ms	85.25	369.5	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	99.95	40.99 s	522 ms	87.37	315.9	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	104.7	39.14 s	515 ms	85.98	321	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	281.86	29.1 s	202 ms	85.25	483.8	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	189.07	43.34 s	971 ms	87.37	337.8	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	187.12	43.81 s	879 ms	86.36	333.2	1	417f69be ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	188.48	43.48 s	878 ms	86.63	350.2	1	7d1a7c8e ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	198.72	41.24 s	954 ms	85.98	344.2	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	196.91	41.63 s	870 ms	86.83	338.7	1	275a096d ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	197.42	41.52 s	870 ms	86.86	354.3	1	fe8141d2 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	293.74	41.93 s	214 ms	85.25	459.2	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	264.7	46.44 s	1.3 s	87.37	364	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	277.65	44.28 s	1.3 s	85.98	358.8	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	340.1	42.27 s	227 ms	85.25	496.1	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	303.08	47.32 s	1.4 s	87.37	373.4	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	317.41	45.18 s	1.4 s	85.98	369.2	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	387.93	42.41 s	237 ms	85.25	534.9	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	339.61	48.27 s	1.7 s	87.37	381.2	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	336.95	48.66 s	1.6 s	86.36	379.7	1	417f69be ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	338.65	48.43 s	1.6 s	86.63	396.9	1	7d1a7c8e ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	354.95	46.19 s	1.6 s	85.98	377.1	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	355.89	46.08 s	1.6 s	86.83	375.7	1	275a096d ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	356.25	46.05 s	1.6 s	86.86	392.4	1	fe8141d2 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 9 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	433.07	42.78 s	246 ms	85.25	575.8	1	5fee1315 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	56.09	36.63 s	8 s	85.25	418.9	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	31.54	64.93 s	24.9 s	87.37	402.7	1	73fcb7c1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	31.45	65.12 s	24.9 s	86.63	415.7	1	448b0777 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	32.72	62.59 s	24.3 s	85.99	406.5	1	28f51cf1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	62.52	67.49 s	12 s	85.25	444.3	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	41.27	111.14 s	34 s	87.37	426.3	1	73fcb7c1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	42.54	106.24 s	33.5 s	85.99	428.9	1	28f51cf1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	63.59	109.31 s	13.5 s	85.25	446.8	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	66.75	144.63 s	15 s	85.25	461.6	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	50.67	220.6 s	45.8 s	86.63	469.4	1	448b0777 ↗

Text generation · 128K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.74	421.65 s	61.9 s	86.63	494.5	1	448b0777 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	71.05	28.83 s	268 ms	85.25	356.6	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	49.98	40.98 s	506 ms	86.36	322.7	1	68d1f2c6 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.97	38.72 s	435 ms	85.99	333.9	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.9	38.72 s	508 ms	86.86	338	1	7cea351c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.59	38.95 s	509 ms	86.83	328.8	1	d14962c1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	89.43	45.82 s	323 ms	85.25	399.6	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	97.25	42.14 s	646 ms	85.99	340.8	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	135.43	45.41 s	379 ms	85.25	470.9	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	138.66	44.36 s	822 ms	85.99	358.8	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	165.29	49.61 s	420 ms	85.25	524	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	171.33	47.85 s	1 s	86.36	365.5	1	68d1f2c6 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	181.16	45.25 s	1.1 s	85.99	375.3	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	179.81	45.65 s	1 s	86.86	387.5	1	7cea351c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	179.42	45.72 s	1.1 s	86.83	371.9	1	d14962c1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	287.31	57.12 s	1.9 s	86.36	425.4	1	68d1f2c6 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	301.16	54.48 s	1.9 s	86.86	443.7	1	7cea351c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	300.98	54.52 s	1.9 s	86.83	426	1	d14962c1 ↗

Models with reports but no qualified results · 136

Their original reports remain available on each model page. They did not qualify for comparison; each report states why.

Models with reports but no qualified results · 136 rows
Model	Test task
AutoGLM-Phone-9BZhipuAI/AutoGLM-Phone-9B	Text generation
ChatTTS2Noise/ChatTTS	Speech synthesis
DASD-4B-ThinkingAlibaba-Apsara/DASD-4B-Thinking	Text generation
DeepSeek-OCR-2deepseek-ai/DeepSeek-OCR-2	Text generation
DeepSeek-R1-Distill-Llama-8Bdeepseek-ai/DeepSeek-R1-Distill-Llama-8B	Text generation
DeepSeek-R1-Distill-Qwen-1.5Bdeepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B	Text generation
DeepSeek-R1-Distill-Qwen-1.5B/root/autodl-tmp/.cache/modelscope/models/deepseek-ai--DeepSeek-R1-Distill-Qwen-1.5B/snapshots/master	Text generation
DeepSeek-R1-Distill-Qwen-7Bdeepseek-ai/DeepSeek-R1-Distill-Qwen-7B	Text generation
DeepSeek-R1-Qwen3-8Bdeepseek-ai/DeepSeek-R1-0528-Qwen3-8B	Text generation
dots.ocrrednote-hilab/dots.ocr	Text generation
Gemma-3-270mgoogle/gemma-3-270m	Text generation
Hunyuan-1.8B-FP8/root/.cache/modelscope/hub/models/Tencent-Hunyuan/HY-MT1___5-1___8B-FP8	Text generation
Hunyuan-1.8B-FP8/home/tiger/.cache/modelscope/hub/models/Tencent-Hunyuan/HY-MT1___5-1___8B-FP8	Text generation
Hunyuan-7B-FP8/root/.cache/modelscope/hub/models/Tencent-Hunyuan/HY-MT1___5-7B-FP8	Text generation
Hunyuan-MT2-1.8B-FP8Tencent-Hunyuan/Hy-MT2-1.8B-FP8	Text generation
Hunyuan-MT2-7B-FP8Tencent-Hunyuan/Hy-MT2-7B-FP8	Text generation
HunyuanOCRTencent-Hunyuan/HunyuanOCR	Text generation
HY-MT1.5-1.8B-Q4_K_M/root/.cache/modelscope/hub/models/Tencent-Hunyuan/HY-MT1___5-1___8B-GGUF/HY-MT1.5-1.8B-Q4_K_M.gguf	Text generation
HY-MT1.5-1.8B-Q6_KTencent-Hunyuan/HY-MT1.5-1.8B-GGUF/HY-MT1.5-1.8B-Q6_K.gguf	Text generation
HY-MT1.5-1.8B-Q8_0Tencent-Hunyuan/HY-MT1.5-1.8B-GGUF/HY-MT1.5-1.8B-Q8_0.gguf	Text generation
HY-MT1.5-7B-Q4_K_M/root/.cache/modelscope/hub/models/Tencent-Hunyuan/HY-MT1___5-7B-GGUF/HY-MT1.5-7B-Q4_K_M.gguf	Text generation
HY-MT1.5-7B-Q6_KTencent-Hunyuan/HY-MT1.5-7B-GGUF/HY-MT1.5-7B-Q6_K.gguf	Text generation
HY-MT1.5-7B-Q8_0Tencent-Hunyuan/HY-MT1.5-7B-GGUF/HY-MT1.5-7B-Q8_0.gguf	Text generation
LightOnOCR-2-1Blightonai/LightOnOCR-2-1B	Text generation
Llama-3.1-8B-Instruct-FP8nvidia/Llama-3.1-8B-Instruct-FP8	Text generation
Llama-3.1-8B-Instruct-NVFP4nvidia/Llama-3.1-8B-Instruct-NVFP4	Text generation
MAI-UI-8BTongyi-MAI/MAI-UI-8B	Text generation
MiniCPM-o-4_5openbmb/MiniCPM-o-4_5	Text generation
MiniCPM-V-4.7-35B-A3B-NVFP4sambit9238/MiniCPM-V-4.7-35B-A3B-NVFP4	Text generation
Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512	Text generation
Ministral-3-8B-Instruct-2512mistralai/Ministral-3-8B-Instruct-2512	Text generation
Ministral-3-14B-Instruct-2512mistralai/Ministral-3-14B-Instruct-2512	Text generation
Ministral-3-14B-Instruct-2512-FP8mistralai/Ministral-3-14B-Instruct-2512-FP8	Text generation
Nanbeige4-3B-Thinking-2511Nanbeige/Nanbeige4-3B-Thinking-2511	Text generation
NVIDIA-Nemotron-3-Nano-4B-FP8nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8	Text generation
NVIDIA-Nemotron-Nano-9B-v2-FP8nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8	Text generation
NVIDIA-Nemotron-Nano-9B-v2-NVFP4nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4	Text generation
NVIDIA-Nemotron-Nano-12B-v2-VL-FP8nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8	Text generation
NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QADnvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD	Text generation
PaddleOCR-VL-1.5PaddlePaddle/PaddleOCR-VL-1.5	Text generation
Qwen2.5-0.5Bqwen2.5:0.5b	Text generation
Qwen2.5-0.5B-Instruct-AWQ/root/.cache/modelscope/hub/models/Qwen/Qwen2___5-0___5B-Instruct-AWQ	Text generation
qwen2.5-0.5b-instruct-q4_k_m.ggufQwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-1.5Bqwen2.5:1.5b	Text generation
Qwen2.5-1.5B-Instruct-AWQ/root/.cache/modelscope/hub/models/Qwen/Qwen2___5-1___5B-Instruct-AWQ	Text generation
qwen2.5-1.5b-instruct-q4_k_m.ggufQwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-3Bqwen2.5:3b	Text generation
Qwen2.5-3B-AWQ/home/u20250324/.cache/modelscope/hub/models/Qwen/Qwen2___5-3B-Instruct-AWQ	Text generation
Qwen2.5-3B-InstructQwen/Qwen2.5-3B-Instruct	Text generation
qwen2.5-3b-instruct-q4_k_m.ggufQwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-7Bqwen2.5:7b	Text generation
Qwen2.5-7B-AWQ/home/u20250324/.cache/modelscope/hub/models/Qwen/Qwen2___5-7B-Instruct-AWQ	Text generation
Qwen2.5-7B-Instruct-AWQ/root/.cache/modelscope/hub/models/Qwen/Qwen2___5-7B-Instruct-AWQ	Text generation
Qwen2.5-7B-Q4_K_MQwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-14B-AWQ/home/u20250324/.cache/modelscope/hub/models/Qwen/Qwen2___5-14B-Instruct-AWQ	Text generation
Qwen2.5-14B-Q4_K_MQwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-32B-Q4_K_MQwen/Qwen2.5-32B-Instruct-GGUF/qwen2.5-32b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-72B-AWQQwen/Qwen2.5-72B-Instruct-AWQ	Text generation
Qwen3-0.6BQwen/Qwen3-0.6B	Text generation
Qwen3-0.6B/root/autodl-tmp/.cache/modelscope/models/Qwen--Qwen3-0.6B/snapshots/master	Text generation
Qwen3-0.6B/root/.cache/modelscope/models/Qwen--Qwen3-0.6B/snapshots/master	Text generation
Qwen3-0.6B-FP8Qwen/Qwen3-0.6B-FP8	Text generation
Qwen3-0.6B-FP8/root/autodl-tmp/.cache/modelscope/models/Qwen--Qwen3-0.6B-FP8/snapshots/master	Text generation
Qwen3-0.6B-GPTQ/root/autodl-tmp/.cache/modelscope/models/Qwen--Qwen3-0.6B-GPTQ-Int8/snapshots/master	Text generation
Qwen3-0.6B-GPTQ-Int8Qwen/Qwen3-0.6B-GPTQ-Int8	Text generation
Qwen3-0.6B-Q8Qwen/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q8_0.gguf	Text generation
Qwen3-0.6B-Q8/root/.cache/modelscope/models/Qwen--Qwen3-0.6B-GGUF/snapshots/master/Qwen3-0.6B-Q8_0.gguf	Text generation
Qwen3-1.7BQwen/Qwen3-1.7B	Text generation
Qwen3-1.7B-FP8Qwen/Qwen3-1.7B-FP8	Text generation
Qwen3-1.7B-Q8Qwen/Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf	Text generation
Qwen3-4BQwen/Qwen3-4B	Text generation
Qwen3-4B-FP8Qwen/Qwen3-4B-FP8	Text generation
Qwen3-4B-Instruct-2507-FP8Qwen/Qwen3-4B-Instruct-2507-FP8	Text generation
Qwen3-4B-Q4Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf	Text generation
Qwen3-4B-Q5_K_MQwen/Qwen3-4B-GGUF/Qwen3-4B-Q5_K_M.gguf	Text generation
Qwen3-4B-Q8_0Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q8_0.gguf	Text generation
Qwen3-4B-Thinking-2507-FP8Qwen/Qwen3-4B-Thinking-2507-FP8	Text generation
Qwen3-8BQwen/Qwen3-8B	Text generation
Qwen3-8B/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-8B	Text generation
Qwen3-8B-AWQQwen/Qwen3-8B-AWQ	Text generation
Qwen3-8B-FP8Qwen/Qwen3-8B-FP8	Text generation
Qwen3-8B-NVFP4RedHatAI/Qwen3-8B-NVFP4	Text generation
Qwen3-8B-Q4Qwen/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf	Text generation
Qwen3-8B-Q5_K_MQwen/Qwen3-8B-GGUF/Qwen3-8B-Q5_K_M.gguf	Text generation
Qwen3-8B-Q8_0Qwen/Qwen3-8B-GGUF/Qwen3-8B-Q8_0.gguf	Text generation
Qwen3-14B/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-14B	Text generation
Qwen3-14B-AWQQwen/Qwen3-14B-AWQ	Text generation
Qwen3-14B-FP8Qwen/Qwen3-14B-FP8	Text generation
Qwen3-14B-NVFP4RedHatAI/Qwen3-14B-NVFP4	Text generation
Qwen3-14B-Q4Qwen/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf	Text generation
Qwen3-14B-Q5_K_MQwen/Qwen3-14B-GGUF/Qwen3-14B-Q5_K_M.gguf	Text generation
Qwen3-14B-Q8_0Qwen/Qwen3-14B-GGUF/Qwen3-14B-Q8_0.gguf	Text generation
Qwen3-30B-A3B/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-30B-A3B	Text generation
Qwen3-30B-A3B-FP8Qwen/Qwen3-30B-A3B-Instruct-2507-FP8	Text generation
Qwen3-30B-A3B-FP8/root/.cache/modelscope/hub/models/Qwen/Qwen3-30B-A3B-FP8	Text generation
Qwen3-30B-A3B-Q4_K_MQwen/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf	Text generation
Qwen3-30B-A3B-Q8_0Qwen/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q8_0.gguf	Text generation
Qwen3-30B-A3B-Thinking-FP8Qwen/Qwen3-30B-A3B-Thinking-2507-FP8	Text generation
Qwen3-32B/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-32B	Text generation
Qwen3-32B-AWQQwen/Qwen3-32B-AWQ	Text generation
Qwen3-32B-FP8Qwen/Qwen3-32B-FP8	Text generation
Qwen3-32B-Q4Qwen/Qwen3-32B-GGUF/Qwen3-32B-Q4_K_M.gguf	Text generation
Qwen3-32B-Q5_K_MQwen/Qwen3-32B-GGUF/Qwen3-32B-Q5_K_M.gguf	Text generation
Qwen3-VL-2B/root/.cache/huggingface/hub/models--Qwen--Qwen3-VL-2B-Instruct/snapshots/89644892e4d85e24eaac8bacfd4f463576704203	Text generation
Qwen3-VL-2B/home/obscura/.cache/modelscope/hub/models/Qwen/Qwen3-VL-2B-Instruct	Text generation
Qwen3-VL-2B-Instruct/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-VL-2B-Instruct	Text generation
Qwen3-VL-2B-Instruct-FP8Qwen/Qwen3-VL-2B-Instruct-FP8	Text generation
Qwen3-VL-4B-Instruct/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-VL-4B-Instruct	Text generation
Qwen3-VL-4B-Instruct-FP8Qwen/Qwen3-VL-4B-Instruct-FP8	Text generation
Qwen3-VL-8B-Instruct/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-VL-8B-Instruct	Text generation
Qwen3-VL-8B-Instruct-FP8Qwen/Qwen3-VL-8B-Instruct-FP8	Text generation
Qwen3-VL-30B-A3B-FP8Qwen/Qwen3-VL-30B-A3B-Instruct-FP8	Text generation
Qwen3-VL-30B-A3B-Instruct/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-VL-30B-A3B-Instruct	Text generation
Qwen3-VL-32B-FP8Qwen/Qwen3-VL-32B-Instruct-FP8	Text generation
Qwen3-VL-32B-Instruct/home/lyx/.cache/modelscope/hub/models/Qwen/Qwen3-VL-32B-Instruct	Text generation
qwen3:0.6bqwen3:0.6b	Text generation
Qwen3.5-0.8BQwen/Qwen3.5-0.8B	Text generation
Qwen3.5-0.8B/root/autodl-tmp/.cache/modelscope/models/Qwen--Qwen3.5-0.8B/snapshots/master	Text generation
Qwen3.5-0.8B/root/.cache/modelscope/models/Qwen--Qwen3.5-0.8B/snapshots/master	Text generation
Qwen3.5-2BQwen/Qwen3.5-2B	Text generation
Qwen3.5-2B/root/.cache/modelscope/models/Qwen--Qwen3.5-2B/snapshots/master	Text generation
Qwen3.5-4BQwen/Qwen3.5-4B	Text generation
Qwen3.5-4B/root/.cache/modelscope/models/Qwen--Qwen3.5-4B/snapshots/master	Text generation
Qwen3.5-9BQwen/Qwen3.5-9B	Text generation
Qwen3.5-27B-FP8Qwen/Qwen3.5-27B-FP8	Text generation
Qwen3.6-27B-AWQ/root/.cache/modelscope/models/tclf90--Qwen3.6-27B-AWQ/snapshots/master	Text generation
Qwen3.6-35B-A3B-AWQ-4bitcyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	Text generation
Qwen3.6-35B-A3B-FP8Qwen/Qwen3.6-35B-A3B-FP8	Text generation
Qwen3.8-27B-NVFP4hf:nvidia/Qwen3.8-27B-NVFP4	Text generation
Qwen3.8-27B-UD-Q4_K_M.ggufhf:unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf	Text generation
Step3-VL-10Bstepfun-ai/Step3-VL-10B	Text generation
testtest	Text generation
WeMM-Embedding-2Btencent/WeMM-Embedding-2B	Embedding
WeMM-Embedding-4Btencent/WeMM-Embedding-4B	Embedding
WeMM-Embedding-9Btencent/WeMM-Embedding-9B	Embedding
whisper-large-v3-turboopenai/whisper-large-v3-turbo	Speech recognition

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	214.6	9.63 s	298 ms	86.28	303.8	1	86d36e4f ↗
C4	516.94	16.14 s	872 ms	86.28	339.5	1	86d36e4f ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	222.74	9.19 s	28 ms	86.28	288.8	1	adbfef8a ↗
C4	741.85	11.05 s	35 ms	86.28	335	1	adbfef8a ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	15.39	133.36 s	28.5 s	12.74	67.9	1	fcfe2d64 ↗
C4	20.31	468.13 s	53.1 s	12.74	67.5	1	fcfe2d64 ↗

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	38.33	31 ms	—	0.82	50.5	4	2c17d254 ↗
B4	138.64	33 ms	—	0.82	54.2	4	2c17d254 ↗

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	38.27	32 ms	—	0.82	48.6	1	47f71b64 ↗
B4	137.82	35 ms	—	0.82	51.9	1	47f71b64 ↗

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	39.4	26 ms	—	—	—	1	2c97c869 ↗
B4	119.85	44 ms	—	—	—	1	2c97c869 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	49.18	73 ms	—	1	158.9	1	3c7db092 ↗
B2	52.8	140 ms	—	1.06	194.7	1	3c7db092 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	120.3	29 ms	—	2	152.1	2	061d047a ↗
B4	125.19	124 ms	—	2.03	169.7	2	061d047a ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	35.95	105 ms	—	1.45	76.3	11	b40f8533 ↗
B2	37.83	237 ms	—	1.45	82.6	11	b40f8533 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	43.43	101 ms	—	0.99	64.1	1	d8e4b3ce ↗
B4	44.88	396 ms	—	1.03	69.1	1	d8e4b3ce ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	40.36	104 ms	—	0.99	60.5	1	c0488108 ↗
B4	41.8	426 ms	—	1.03	65.9	1	c0488108 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	76.72	45 ms	—	1	83.5	1	d90a5c29 ↗
B2	79.84	100 ms	—	1.03	100.8	1	d90a5c29 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	65.65	—	—	0.41	134.9	1	3f8b97e6 ↗
B4	91.84	—	—	0.4	143.6	1	3f8b97e6 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	232.81	—	—	1.42	113.5	2	ba5a45ac ↗
B4	364.19	—	—	1.43	115.8	2	ba5a45ac ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	104.96	—	—	0.51	51.5	1	64035c7b ↗
B4	155.93	—	—	0.52	50.1	1	64035c7b ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	90.94	—	—	0.51	51.5	10	e4ef00c1 ↗
B4	132.16	—	—	0.52	51.2	10	e4ef00c1 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	73.68	—	—	0.38	46.8	1	7d182de2 ↗
B4	68.2	—	—	0.38	49.2	1	7d182de2 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	121.9	—	—	0.42	73.6	1	7aaa1f0d ↗
B4	168.25	—	—	0.42	77.4	1	7aaa1f0d ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	137.01	—	—	0.38	54.3	1	4b83f25d ↗
B4	205.27	—	—	0.38	66.3	1	4b83f25d ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	359.69	—	—	1.27	101.7	1	c6b5f31d ↗
B2	364.87	—	—	1.27	101.6	1	c6b5f31d ↗