Databricks has launched Zerobus Ingest, a serverless service that streams data directly to the lakehouse. The solution is designed to eliminate the complexity of traditional message buses, such as Kafka, and drastically reduce costs.
Organizations seeking to scale real-time operational intelligence often get bogged down in expensive streaming architectures. Managing message buses, schema registries, and connector frameworks imposes a significant “complexity tax,” diverting valuable engineering resources from strategic projects. Databricks aims to eliminate this bottleneck with Zerobus Ingest, part of Lakeflow Connect.
Zerobus Ingest is a fully managed service that streams data directly into governed Delta tables. By removing intermediate layers, it delivers a simplified, high-performance architecture. The system supports thousands of concurrent connections and achieves a throughput of over 10GB per second in less than 5 seconds.
Single-sink architecture replaces complex setup
Traditional message buses such as Kafka are designed as multi-sink architectures: universal hubs that route data to dozens of independent consumers. However, that flexibility comes at a high cost when the lakehouse is the only destination. Zerobus Ingest takes a fundamentally different approach with a single-sink architecture, optimized for a single task: pushing data directly to the lakehouse.
This architectural choice eliminates complexity and reduces costs. No more brokers that need to scale as data volumes grow, no partitions to tune for optimal performance, no consumer groups to monitor and debug. Cluster upgrades are also no longer necessary, and specialized Kafka expertise is no longer required.
With Zerobus Ingest, a single managed Databricks endpoint is sufficient. Engineers create a table in Unity Catalog and start writing data via the API or SDK. The serverless architecture automatically scales up to gigabytes per second of ingestion without configuration changes. According to Databricks, this simplifies the traditional streaming architecture from five managed systems to two components.
Interfaces for different use cases
Developers can integrate via gRPC and REST APIs or use language-specific SDKs. Zerobus Ingest offers a wide range of push-based interfaces for industry-specific integrations. The gRPC API is recommended for high-performance applications that require the lowest latency and highest throughput. The REST API in beta is designed for webhooks, serverless functions, and languages where gRPC support is limited.
Production-ready libraries for Python, Java, Rust, Go, and TypeScript simplify authentication and batching logic via gRPC. There is also Open Telemetry support in beta, which allows operational logs, metrics, and traces to be brought into the lakehouse for long-term historical analysis with just a configuration change.
Because every write is governed by Unity Catalog, users get automatic lineage tracking and fine-grained access control from the moment data is created. This provides unified governance for streaming data alongside the rest of the lakehouse.
Tip: Databricks unifies data engineering work through LakeFlow