Run an AI model on your computer

1. embeddinggemma-2

   google · 5 tested configurations

   • bf16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 9 s

     Throughput · Batch 1

     251vectors/sec

     Best measured · Batch 16

     2,710vectors/sec

     Observed GPU occupancy

     1GiB

     P95 latency

     4 ms

     Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 3 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     sentence-transformers · PyTorch 2.13.0+cu130

     Protocol

     native-forward-v1

     Dataset

     embedding-mteb-tasks-v1

     Workload

     embedding

     Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	251.11	4 ms	—	1.05	99.4	1	f489c748 ↗
     B4	924.83	4 ms	—	1.05	111.5	1	f489c748 ↗
     B16	2,710.19	6 ms	—	1.05	114.9	1	f489c748 ↗

   • 4 other test configurations on this GPU

     • bf16 · Text → Vectors · embedding-sanity-v1 · Test duration 6 s

       Throughput · Batch 1

       38.3vectors/sec

       Best measured · Batch 4

       139vectors/sec

       Observed GPU occupancy

       0.8GiB

       P95 latency

       31 ms

       Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       sentence-transformers · PyTorch 2.13.0+cu130

       Protocol

       native-forward-v1

       Dataset

       embedding-sanity-v1

       Workload

       embedding

       Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	38.33	31 ms	—	0.82	50.5	4	2c17d254 ↗
       B4	138.64	33 ms	—	0.82	54.2	4	2c17d254 ↗

     • bf16 · Text → Vectors · embedding-sanity-v1 · Test duration 12 s

       Throughput · Batch 1

       38.8vectors/sec

       Best measured · Batch 64

       1,766vectors/sec

       Observed GPU occupancy

       0.8GiB

       P95 latency

       27 ms

       Use this recipe · Batch 64 ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 4 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       sentence-transformers · PyTorch 2.13.0+cu130

       Protocol

       native-forward-v1

       Dataset

       embedding-sanity-v1

       Workload

       embedding

       Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	38.8	27 ms	—	0.82	48.8	1	dc5960cd ↗
       B4	141.65	29 ms	—	0.82	52.4	1	dc5960cd ↗
       B16	553.54	30 ms	—	0.87	56.4	1	dc5960cd ↗
       B64	1,766.47	38 ms	—	1.03	65.3	1	dc5960cd ↗

     • bf16 · Text → Vectors · embedding-sanity-v1 · Test duration 6 s

       Throughput · Batch 1

       38.3vectors/sec

       Best measured · Batch 4

       138vectors/sec

       Observed GPU occupancy

       0.8GiB

       P95 latency

       32 ms

       Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       sentence-transformers

       Protocol

       native-forward-v1

       Dataset

       embedding-sanity-v1

       Workload

       embedding

       Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	38.27	32 ms	—	0.82	48.6	1	47f71b64 ↗
       B4	137.82	35 ms	—	0.82	51.9	1	47f71b64 ↗

     • bf16 · Text → Vectors · embedding-sanity-v1

       Throughput · Batch 1

       39.4vectors/sec

       Best measured · Batch 4

       120vectors/sec

       Observed GPU occupancy

       Not recorded

       P95 latency

       26 ms

       Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       sentence-transformers

       Protocol

       native-forward-v1

       Dataset

       embedding-sanity-v1

       Workload

       embedding

       Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	39.4	26 ms	—	—	—	1	2c97c869 ↗
       B4	119.85	44 ms	—	—	—	1	2c97c869 ↗

Models shown: 1 / 1

Recipe availability is checked on each report against the published recipe executor.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.