Scaling OpenAI's Habitat Storage Platform to Over 1 Billion Users
Discover how OpenAI evolved its online storage platform, Habitat, using Azure Cosmos DB, advanced multi-tenancy, and distributed caching to serve over 1 billion users globally.
OpenAI products require fast and reliable data access, a responsibility handled by the company's online storage platform, known as Habitat. Operating across 40 geographic regions for over 1 billion users, Habitat had to overcome significant scaling hurdles while maintaining reliability, performance, and developer velocity. This journey involved transitioning Habitat from a simple client library and database into a globally distributed platform managing over 500 petabytes of data, backed by Azure Cosmos DB.
The Multi-Tenancy Evolution
As organizations expand, infrastructure teams typically navigate a transition from single-tenant architectures to multi-tenant models. Single-tenancy offers clear ownership boundaries and limits the blast radius of failures, but it quickly becomes expensive and forces product teams to handle capacity planning, often leading to over-provisioning. Conversely, multi-tenancy places resource control under a centralized database team but introduces challenges like noisy-neighbor issues.
Habitat's underlying storage architecture on Azure Cosmos DB underwent a circular journey. Initially, the platform relied on a single shared default container. This design exposed logical partition key selection directly to clients, causing frequent collisions on popular keys like user_id or workspace_id. Furthermore, physical partitions enforced limits that allowed noisy neighbors and weak partitions to impact unrelated workloads.
To mitigate these issues, OpenAI briefly shifted to single-tenancy by provisioning separate Azure Cosmos DB containers for critical use cases. While this isolated workloads and limited the blast radius, it created new operational bottlenecks. Product teams frequently forgot to pre-scale their containers prior to launches, leading to last-minute firefighting and a lack of stable aggregate growth.
Ultimately, OpenAI returned to a default multi-tenant approach in close collaboration with the Azure team. They introduced hot key throttling and Request tiering (QoS) to protect interactive requests from offline bulk operations. Additionally, an improved serverless mode built by Azure allowed physical partitions to scale up to 10k RU/s and introduced request-unit-based splits alongside storage-based splits, eliminating the need for manual container scaling.
Deep Co-Engineering at Scale
Operating at OpenAI's scale means managed cloud services require deep integration rather than hands-off maintenance. With the largest Azure Cosmos DB accounts storing tens of petabytes and serving more than 20 million QPS, OpenAI and Azure established joint ownership over failure modes and operational tests. Regional outages prompted both engineering teams to improve control-plane and gateway scalability, implement connection pooling, and establish direct communication channels between engineers.
Global Performance and Caching Strategies
Once reliability was established, reducing latency for a globally distributed user base became the next priority. Because compute and storage capacities are often located in different regions, cross-region requests are common. OpenAI deployed a layered read-performance strategy:
- Regional placement: Storing data near users when local data center capacity allows, primarily driven by data residency requirements.
- Broad replication: Replicating latency-critical datasets, such as authentication metadata, across multiple secondary regions or all 40 Azure regions.
- Integrated caching: Utilizing a distributed lookaside caching layer powered by Valkey servers close to user requests. To maintain high hit rates without serving stale data, Habitat uses Change Data Capture (CDC) via Azure Cosmos DB's change feed to propagate invalidations globally within seconds.
- Rust migration: Rewriting the Habitat service in Rust to reduce proxy overhead.
Together, these optimizations reduced the overall global read p50 latency from approximately 120 milliseconds down to roughly 10 milliseconds.
What it means for developers
Underneath the complex plumbing of multi-tenant databases, change data feeds, and distributed caching, Habitat's core mission is to abstract infrastructure details away from product developers. By handling sharding, replication, and scaling internally, developers can build features faster without needing deep knowledge of the underlying database. For developers looking to experiment with top AI models without managing complex infrastructure themselves, Apixo offers cheap pay-per-token API access to leading models like Claude, GPT, Gemini, Grok, and DeepSeek through a single key at https://apixoai.online.
Source: Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II) — OpenAI News. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
