OpenAI has revealed the intricate details of its custom-built online data storage platform, Habitat, designed to ensure rapid and reliable data access across all its products. The platform is engineered to handle queries from over a billion users across 40 global regions, responding in milliseconds.
The rapidly growing demand and novel use cases for AI services present a significant challenge for OpenAI’s infrastructure. To maintain and improve reliability, performance, and product development velocity, the company has had to fundamentally rethink its approach to designing and operating its online data storage platform.
This two-part series delves into how Habitat evolved from a simple client library and database into a globally distributed platform storing over 500 petabytes of data. While Part I explored Habitat’s journey from a Python client to a robust service, Part II focuses on scaling the underlying storage component – Azure Cosmos DB – and achieving a high-performance, globally reliable multi-tenant model with co-located containers.
The Online Data Storage Platform: Habitat
Habitat serves as OpenAI’s core online storage solution, built to provide swift and dependable access to essential information. Its primary functions include managing:
- Queries
- Responses
- Change Data Capture (CDC)
Reliable Multi-Tenancy at Scale
As organizations grow, infrastructure teams often transition from single-tenant to multi-tenant systems. While single-tenancy offers reduced risks and clearer responsibility delineation, it can become prohibitively expensive. Multi-tenancy, conversely, centralizes resource management but can introduce “noisy neighbor” issues, where one tenant excessively consumes shared resources.
OpenAI’s path involved moving from a single multi-tenant Azure Cosmos DB container, to dedicating individual containers (a single-tenant approach), and ultimately returning to a default multi-tenant model. This transition was facilitated by leveraging Azure Cosmos DB’s built-in mechanisms.
Phase I: Leveraging Azure Cosmos DB’s Base Capabilities
Initially, OpenAI allowed any client workload, relying on the default limit of 10,000 Request Units per second (RU/s) per logical partition key. However, this approach led to several critical issues:
- Logical Partition Key Limitations: Frequent collisions on shared keys (e.g., `user_id`) across different use cases capped overall throughput.
- “Noisy Neighbors”: Physical partitions, where actual limits are enforced, could become overloaded by a single logical key, negatively impacting others.
- Weakening Partitions: As data volume grew, the throughput limit per physical partition decreased, leading to performance degradation even under stable load.
Phase II: Workload Isolation and Single-Tenancy
The use of dedicated containers for critical use cases partially addressed these problems. Each container possessed its own physical partitions, mitigating the “noisy neighbor” effect. However, this necessitated significant capacity planning for every new container deployment.
This approach also created challenges for stable aggregated growth. New, rapidly adopted use cases could unpredictably surge in demand on their dedicated containers, requiring urgent scaling. The constant need for capacity planning became a complex burden for a large-scale organization.
Phase III: Default Multi-Tenancy with Data Migration Capabilities
To overcome operational hurdles, OpenAI reverted to a multi-tenant solution, collaborating closely with the Azure team on key enhancements:
- Throttling Most Active Keys: This prevents “noisy neighbors” from excessively consuming resources.
- Quality of Service (QoS): Prioritizes requests from interactive users over background tasks (e.g., bulk data loading).
- Serverless Mode: The enhanced Azure Cosmos DB serverless mode allows physical partitions to scale up to 10,000 RU/s, eliminating the need for manual container monitoring and scaling.
These adjustments enabled a return to a single, shared container model, drastically simplifying capacity planning for development teams while maintaining workload isolation. In recent months, OpenAI has reported no incidents related to resource contention or “noisy neighbors,” even as product development accelerates.
Boosting Performance Globally
With reliability secured, the next priority was enhancing performance for global users. The geographical distribution of compute and storage inevitably led to cross-region requests, resulting in substantial latency.
To optimize read performance (which accounts for over 90% of Habitat’s queries), OpenAI implemented a three-pronged strategy:
- Data Proximity to Users: Where feasible, data is stored in the same regions as users to minimize latency, constrained by the available physical capacity of cloud data centers.
- Broad Replication of Smallest & Most Latency-Sensitive Datasets: Critical data, such as authentication metadata, is replicated across numerous regions to reduce round-trip times.
- Integrated Caching: The utilization of Valkey, a distributed cache, stores frequently accessed data, significantly reducing latency, particularly across regions. An innovative cache invalidation approach, leveraging Azure Cosmos DB’s Change Data Capture (CDC), ensures data freshness.
Rewriting parts of the Habitat service from Python to Rust also substantially reduced overhead, further boosting performance. These optimizations have led to a dramatic reduction in p50 read latency, from approximately 120ms down to 10ms.
Scaling Change Data Capture
Robust Change Data Capture (CDC) is crucial not only for Habitat’s caching mechanisms but also for powering secondary indexes, data migration pipelines, analytics, and more. OpenAI’s CDC platform abstracts the complexity of handling millions of changes per second, offering developers a streamlined interface.
Looking Ahead
To further scale Habitat and conversation storage for the next billion users, OpenAI is developing its own exabyte-scale online storage system. This system is optimized for massive data volumes and combines a RocksDB Cloud core with Raft replication. It implements the same document model as Azure Cosmos DB but features disaggregated storage to reduce reliance on flash memory.
source: OpenAI
