Run an AI model on your computer

1. Qwen3-4B-AWQ

   Qwen · 1 tested configuration

   • AWQ · 8K → 2K tokens · No cache reuse

     Output throughput · includes prefill · 1 request

     15.4tok/s

     Best measured · 4 concurrent requests

     20.3tok/s

     Observed GPU occupancy

     12.7GiB

     First token

     28.5 s

     Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 concurrency levels

     Total output tokens divided by the full request window, including prefill and queueing.

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     vllm · vLLM 0.29.0

     Protocol

     wbench-fixed-http-v3

     Dataset

     migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

     Workload

     8192in-2048out

     Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     C1	15.39	133.36 s	28.5 s	12.74	67.9	1	fcfe2d64 ↗
     C4	20.31	468.13 s	53.1 s	12.74	67.5	1	fcfe2d64 ↗

2. embeddinggemma-2

   unsloth · 2 tested configurations

   • nf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

     Throughput · Batch 1

     90.1vectors/sec

     Best measured · Batch 16

     809vectors/sec

     Observed GPU occupancy

     0.7GiB

     P95 latency

     11 ms

     Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 3 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     sentence-transformers · PyTorch 2.12.1+cu126

     Protocol

     native-forward-v1

     Dataset

     embedding-mteb-tasks-v1

     Workload

     embedding

     Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	90.07	11 ms	—	0.75	56.2	2	2f9465c4 ↗
     B4	302.9	13 ms	—	0.75	61.5	2	2f9465c4 ↗
     B16	809.43	20 ms	—	0.75	64	2	2f9465c4 ↗

   • 1 other test configurations on this GPU

     • fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

       Throughput · Batch 1

       109vectors/sec

       Best measured · Batch 16

       1,002vectors/sec

       Observed GPU occupancy

       0.9GiB

       P95 latency

       9 ms

       Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       sentence-transformers · PyTorch 2.12.1+cu126

       Protocol

       native-forward-v1

       Dataset

       embedding-mteb-tasks-v1

       Workload

       embedding

       Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	108.76	9 ms	—	0.89	50.3	2	cfe051ff ↗
       B4	365.82	11 ms	—	0.89	58.1	2	cfe051ff ↗
       B16	1,002.27	16 ms	—	0.89	63.9	2	cfe051ff ↗

3. embeddinggemma-2

   google · 2 tested configurations

   • nf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

     Throughput · Batch 1

     90vectors/sec

     Best measured · Batch 16

     815vectors/sec

     Observed GPU occupancy

     0.7GiB

     P95 latency

     11 ms

     Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 3 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     sentence-transformers · PyTorch 2.12.1+cu126

     Protocol

     native-forward-v1

     Dataset

     embedding-mteb-tasks-v1

     Workload

     embedding

     Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	90.02	11 ms	—	0.75	54.6	2	43c67a74 ↗
     B4	303.35	13 ms	—	0.75	60.1	2	43c67a74 ↗
     B16	814.64	20 ms	—	0.75	64.1	2	43c67a74 ↗

   • 1 other test configurations on this GPU

     • fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

       Throughput · Batch 1

       109vectors/sec

       Best measured · Batch 16

       1,004vectors/sec

       Observed GPU occupancy

       0.9GiB

       P95 latency

       9 ms

       Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       sentence-transformers · PyTorch 2.12.1+cu126

       Protocol

       native-forward-v1

       Dataset

       embedding-mteb-tasks-v1

       Workload

       embedding

       Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	108.65	9 ms	—	0.89	49.3	2	2846de88 ↗
       B4	365.65	11 ms	—	0.89	57.9	2	2846de88 ↗
       B16	1,004.11	16 ms	—	0.89	63.9	2	2846de88 ↗

Models shown: 3 / 5

Recipe availability is checked on each report against the published recipe executor.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.