Confluent, an IBM company, has announced a significant advancement in data architecture with its new Streamhouse offering. This novel architecture aims to provide AI agents, applications, and real-time analytics with up-to-the-minute business context. With the general availability of Lightning Tables on AWS, Confluent has now completed its five-component Data Streaming Platform, integrating capabilities for data streaming, connectivity, processing, governance, and querying into a unified platform for building Streamhouses.
Why Businesses Need a Streamhouse
Traditional data lakes, warehouses, and even lakehouses were primarily designed for reporting or batch data processing. As AI agents take on increasingly complex tasks within organizations, they require access to timely information from disparate systems that update at varying frequencies. When these systems hold conflicting views of the current business state, AI agents can operate on outdated or contradictory data, leading to errors in subsequent workflows.
Streamhouse addresses this architectural challenge by continuously collecting, processing, governing, and serving up-to-date business context to agents, applications, and analytics in real time.
An Airline Industry Example
Consider an airline managing a disrupted flight. Today, various systems might hold conflicting information: the operations department tracks aircraft location, the scheduling system monitors departure times, and the booking system shows seat availability. Due to differences in system update cadences, an autonomous rebooking agent might construct a itinerary that appears valid in each system individually but is unworkable in practice (e.g., a seat is available, but the aircraft won’t arrive in time for a passenger’s connection).
Customer service teams and notification systems could also rely on the same inconsistent data, exacerbating the disruption. With Streamhouse, every change updates a single, unified view of the airline’s operations. When an aircraft’s location changes, a flight is delayed, or a seat is booked, the originating system pushes this update into a continuous event stream. Streamhouse unifies these updates into a current business context and makes it accessible to rebooking agents, customer service teams, and applications. Instead of piecing together fragmented information, they can collectively verify aircraft location, departure time, and seat availability before offering a passenger a new itinerary.
Enterprises are already seeing tangible results. Accenture, for instance, is leveraging a Confluent-powered Streamhouse architecture to unify data across its global business.
“Operating across more than 120 countries, our global enterprise has been constrained by legacy batch and point-to-point architectures, where data extraction duplication increased operational costs and analytical insights lagged. Transitioning to a Confluent-powered Streamhouse architecture decouples data producers from consumers, transforming high-volume event streams into governed, reusable, real-time data products that flow seamlessly into operational and analytical systems. With Confluent’s infrastructure automation and seamless enterprise connectivity, our developers work in a standardized environment, unhindered by building custom integrations or micromanaging data movement. This continuous data foundation has reduced time to insights from months to days and significantly decreased reporting times, allowing our teams to focus on building high-value AI for our 800,000 employees,” said Thomas Mahaffey, Chief Data Strategist SAI, Accenture.
What Sets Streamhouse Apart
Streamhouse is built upon three core architectural principles: timeliness, business criticality, and decentralization. It supports a continuously updated business context, is designed with production service levels in mind, and operates on data where it resides without requiring prior consolidation. It functions alongside existing data stores and lakes, adding a real-time operational layer for decision-making and action.
“As an AI-driven commercial platform that powers personalized shopper experiences and operational/financial benefits for merchants, Extend requires exceptional adaptability to rapidly onboard merchants, adjust pricing in real-time, and instantly detect fraud. We initially struggled with traditional message queues and static data lakes that lacked event replayability, durability, and strict schema enforcement at scale. Adopting a 100% serverless Streamhouse architecture on Confluent Cloud transforms Kafka into our operational source of truth, allowing downstream microservices to asynchronously consume and query data in real time without breaking contracts. This fundamentally decoupled streaming foundation directly provides the agility and responsiveness needed to scale our operations, adapt to real-time market signals, and power real-time AI agent processes,” said Jonathan Cropp, Director of AI & Architecture, Extend.
Confluent Data Streaming Platform Now Offers a Complete Streamhouse
The Confluent Data Streaming Platform (DSP) provides a complete, unified foundation for building a Streamhouse. At its core is Apache Kafka®, which ingests and transmits business events. Kafka alone requires teams to build everything else needed to transform those events into current business context. DSP brings this together in a single platform, now adding Query as the fifth component alongside Stream, Connect, Process, and Govern. Key capabilities and recent additions to each platform component include:
- Stream, built on Apache Kafka and the Kora engine, provides a durable, scalable foundation for capturing and streaming business events between systems with sub-millisecond latency (per Confluent internal testing; actual latency depends on configuration and load).
- Connect, with over 150 fully managed and custom connectors, ingests data from enterprise systems into Kafka and fans it out. Hybrid connector management extends teams’ ability to govern these connections.
- Process leverages Apache Flink to join, enrich, and derive business context near the source as data changes. Additional batch execution capabilities help bootstrap streaming workloads based on existing data.
- Govern helps teams discover, classify, and protect data through lineage, cataloging, quality, encryption, schema, and access control features.
- Query provides real-time business context for analytics, agents, and applications.
Query enables the use of streaming data for analytics, agents, and applications through task-specific interfaces:
- Confluent Tableflow makes streaming data available to data lakes and warehouses as open table formats, ready for analytics.
- The Real-Time Context Engine provides current business context to AI agents via the Model Context Protocol (MCP).
- Lightning Tables offers applications low-latency access via SQL and APIs to the current business state, allowing them to react to changing conditions.
Lightning Tables: Fast Access to Current Business State
Lightning Tables, generally available on AWS within Confluent Cloud, is a read-service layer that reliably stores and materializes Apache Kafka data into tables, keeping them up-to-date as changes occur. This makes the current business state accessible for low-latency queries. It features:
- Data freshness and query performance: Between approximately 100 milliseconds from data creation to availability, with 10-100 millisecond query latency and over 1,000 queries per second throughput (per Confluent internal testing).
- Lower cost and complexity: Continuous serving via a fully managed service layer, without separate ETL pipelines, databases, or caches.
- Multiple query interfaces: Queries via SQL and direct API.
- Unified governance: Application of schema, lineage, and access policies from data ingestion to serving.
- Schema evolution: Management of schema changes to prevent downstream application failures.
- Feature store: Computation of features in Flink and storage for machine learning use cases, including real-time personalization.
“The next era of enterprise AI will be won by agents that know what’s happening in the business at the moment an action is taken,” said Sean Glas, General Manager, Data, Confluent, an IBM company. “That’s precisely what Streamhouse can provide. With Lightning Tables, our Data Streaming Platform provides every agent and application with a governed, real-time view of the business. This means companies can react to fraud signals sooner, serve customers at the moment of need, and operate based on what is true now, not what was true yesterday.”
About Confluent
Confluent, an IBM company, is the data streaming platform. Confluent Cloud is the foundational platform for data in motion—designed as an intelligent, connective tissue that delivers a continuous, real-time flow of data from many sources across the organization. With Confluent, organizations can meet the new business imperative of delivering rich customer experiences and transform into sophisticated, software-driven operations in real time.
About IBM
IBM is a leading provider of global hybrid cloud and AI, and business consulting services. It helps clients in over 175 countries harness insights from their data, optimize business processes, reduce costs and gain a competitive advantage in their industries. Thousands of organizations in more than 175 countries—including more than 70% of the Fortune 100—run on IBM’s hybrid cloud platform and Red Hat OpenShift to create the digital foundations for their businesses. IBM’s groundbreaking innovations in AI, quantum computing, industry cloud solutions and consulting deliver open and flexible options to its clients. All of this is underpinned by IBM’s long-standing commitment to trust, transparency, responsibility, inclusivity and service.
source: IBM
