Exa Boosts Document Indexing Throughput by 50% Using AWS AI Hardware
Discover how search company Exa optimized its embedding infrastructure on AWS Inferentia2 and Trainium2 chips, achieving a 47% cost-performance improvement.

Exa, an organization building a comprehensive web index for semantic search and AI grounding, has increased its embedding throughput by roughly 50% through infrastructure optimizations on AWS Neuron hardware. Operating at a scale of tens or hundreds of billions of documents makes unit economics critical, driving the company to refine its deployment on AWS Inferentia2 and AWS Trainium2 chips.
To handle both live indexing and a massive backfill of 30 billion priority documents concurrently, Exa evaluated its hardware footprint. By moving from a baseline floating-point implementation to heavily tailored execution strategies, the engineering team successfully brought down the cost and resource requirements per one million documents embedded.
Step-by-Step Optimization Journey
Starting with a standard, data-parallel FP16 model compiled for Inferentia2 xlarge instances, Exa applied a series of performance enhancements using the Neuron SDK. Initial gains came from weight layout optimization and enabling autocast, which lowered precision for matrix multiplications while preserving a 0.999 cosine similarity accuracy threshold compared to the reference model.
Next, the team adopted the Neuron Kernel Interface (NKI) library. By implementing custom weight-stationary fused SwiGLU MLP kernels, engineers eliminated high-bandwidth memory (HBM) spills and reloads that previously kept the Tensor Engine from running at full capacity. Addressing core contention and memory bottlenecks required deeper architectural changes. Developers tailored attention kernels to match the exact tensor shapes of their model, cutting unnecessary padding and folding operations into single matrix multiplications to minimize HBM round trips.
Developers can try top AI models cheaply through one API at https://apixoai.online.
Agentic Methodology and Multi-Region Flexibility
An automated agentic development workflow managed through Flyte drove the optimization process. Different AI models handled distinct tasks within a validation harness: lightweight models ran compiler parameter grid searches, reasoning agents identified bottlenecks, and code generators produced custom kernels. Every iteration was checked for correctness, performance, and workload saturation.
Beyond single-chip performance, Exa integrated these optimizations into a multi-Region Kubernetes scheduler. By shifting workloads across diverse instance types and regions based on spot and on-demand availability, the infrastructure team can balance compute demands dynamically, carrying optimizations smoothly from Inferentia2 over to Trainium2.
Source: Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips | Amazon Web Services — AWS Machine Learning. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key

