Embedding Performance for Real-Time Retrieval Systems

Hitesh Jain·February 2026

Real-time AI systems rarely fail due to raw model capability. They fail when retrieval becomes slow and unstable under sustained load.

Applications such as AI copilots, chatbots, voice agents, semantic typeahead, and search results pages require:

• Stable tail latency under concurrency
• Reliable early-rank relevance
• Infrastructure efficiency at scale

This report measures a domain-specific commerce embedding model — 768 dimensions, purpose-built for low-latency, infrastructure-efficient retrieval — across three axes:

• Real-time query embedding latency
• Retrieval quality on structured commerce catalogs
• Throughput and infrastructure efficiency

Executive Summary

• P95 query embedding latency under 20 ms at sustained load, holding under 30 ms as concurrency rises
• Strong early-rank relevance on structured commerce queries — nDCG@25 of 72.3%, with an 8.1% zero-result rate at 25
• A 768-dimensional footprint, roughly half the ANN index size and per-query ANN compute of a 1536-dimensional general-purpose embedding
• Stable scaling under sustained concurrency

1. Query Embedding Latency

The model was evaluated under moderate and high sustained concurrency.

Latency Results

Key Observations

• At 30 QPS, P95 latency holds at ~17 ms
• At 50 QPS, P95 remains below 30 ms, demonstrating stable scaling
• Tail latency degrades gracefully rather than cliff-edging as concurrency rises

Test Conditions

• Continuous 5-minute steady-state interval following warm-up
• Hardware: NVIDIA A10G (AWS g5.xlarge)
• Region: AWS us-east-1
• Client workload executed in-region
• Warm state
• No client-side batching
• Query length: 8–32 tokens

2. Retrieval Quality

Low latency alone is insufficient. Embedding models must:

• Surface at least one relevant product in early ranks (critical for chat and voice agents, and for typeahead)
• Provide broad candidate coverage for search results pages

Core Metrics

Interpretation

• Zero-result rate @25: percentage of queries with no relevant result in the top 25
• nDCG @25: ranking quality among early results
• Recall @50: coverage of relevant products in the top 50

Evaluation Setup

• ~300K structured products
• ~20K held-out queries (attribute- and use-case-specific, mixed intent breadth)
• Multi-field catalog (title, attributes, variants)
• Human and synthetic relevance labels
• Identical retrieval pipelines and ANN configurations across evaluated models

The dataset reflects real-world structured commerce search behavior typical of production systems.

3. Infrastructure Efficiency

Vector dimensionality directly impacts ANN memory footprint and compute cost.

Implications

• Lower memory pressure
• Reduced ANN compute per query
• Improved scaling efficiency under production load

In high-QPS environments, dimensionality materially affects infrastructure requirements.

Summary

A domain-specific embedding model optimized for real-time production retrieval matters most where:

• Tail latency directly impacts user experience
• Retrieval failures reduce conversion and agent effectiveness
• Infrastructure efficiency determines scaling economics

For real-time commerce and agentic systems, retrieval performance is not only a model decision — it is an infrastructure decision.

Methodology and Reproducibility

To ensure a production-relevant measurement:

• Identical ANN configuration and retrieval pipeline across evaluated models
• Same catalog dataset and held-out query distribution
• Same AWS region and in-region client execution
• Warm-state measurements only
• No client-side batching
• Sustained steady-state load (5-minute interval after warm-up)
• Each model evaluated using its recommended production serialization and tokenization strategy
• All measurements reflect realistic deployment conditions for real-time systems