Observability has been a prevalent trend throughout the last decade’s worth of cloud computing development, if not longer. With specialist vendors hosting major technology conferences dedicated to nothing but observability, software developers and their operations counterparts have been well-sold on the need to look inside our virtualised spaces so that they can maintain order and control. But is it time for observability to evolve and grow – could observability warehouses be the next big thing?
Why bother with observability should be our first question, clearly.
Why observe?
Technologists at all levels agree that we need observability in all systems, but especially in the cloud, where we require real-time insight into system configuration, behaviour, performance and state so that software teams can detect anomalies, diagnose failures and understand performance across the distributed infrastructure upon which they run their applications and data services. We know that observability is able to help reduce downtime, accelerate incident response and ensure reliability at scale.
The observability industry generates enormous volumes of telemetry data as logs, metrics and traces flow through modern systems. As organisations grow, so does their data volume. Over the past decade, telemetry data (the automated, real-time measurements collected from systems to monitor performance and health) has accelerated dramatically… and many think that the next decade will bring even greater scale.
Ingestion-based pricing models
For Eric Tschetter in his role as chief architect at Imply, we need to talk more about this subject. Imply is known for its real-time analytic platforms powered by Apache Druid, a high-performance, real-time analytics database designed for fast, sub-second data queries. Tschetter reminds us that recent discussions across the observability industry, including the debate around ingestion-based pricing models, but this really just highlights how difficult it has become for organisations to manage the cost of modern telemetry.
“Today’s telemetry volumes are already putting traditional observability economics under pressure. According to Imply’s Breaking Point for Observability Leaders Report, nearly 80% of teams now filter or archive telemetry data to control costs. The common solution is to fix the pricing model for data collection, but that only solves the immediate issue. Many vendors focus on adjusting ingestion-based pricing models to address this challenge. But pricing changes alone do not solve the long-term issue if the underlying architecture can’t economically scale with the telemetry growth,” explained Tschetter.
But there are challenges here i.e. as telemetry volumes continue to expand, many tightly coupled observability architectures are beginning to show strain.
Monolithic observability architectures challenges
Here’s how Imply views the challenge ahead. Many traditional observability stacks tightly couple collection, storage, compute and visualisation into a single system. This model worked well when data volumes were smaller, but modern distributed systems generate far more data than these architectures were designed to handle.
As telemetry volumes grow, Tschetter and team say that organisations often respond by pruning the data they retain or moving older telemetry to lower-cost storage. These approaches help control costs, but they can make it harder to access the full data set when incidents require deeper analysis. These trade-offs reflect a deeper architectural constraint. When storage, compute, and analysis are tightly bundled together, scaling one layer often requires scaling them all.
Lessons from BI’s decoupling evolution
“Observability today resembles business intelligence from 30 years ago. Early BI systems from vendors like SAP BusinessObjects and IBM Cognos bundled data pipelines, storage, compute and dashboards into proprietary stacks. As data volumes and use cases expanded, these architectures became increasingly restrictive,” said Tschetter. “Over time, the industry shifted to a more modular model where storage, compute, and visualisation operate independently. This allowed teams to scale systems easily without moving data or rewriting workflows.”
Could it be that observability is now reaching a similar architectural evolution? It feels like AI is accelerating this shift i.e. every AI-driven system becomes a producer and consumer of telemetry. These workloads increase telemetry volumes while also demanding fast query access and long-term retention.
Existing observability architectures often struggle to support these requirements economically. As AI adoption expands, the pressure on traditional systems will continue to grow.
Into the observability warehouse
Tschetter suggests that most observability stacks lack a dedicated data layer designed to scale with telemetry growth and connect with their existing user workflows. Instead, telemetry is often loaded into a singular system that supports one workflow, but not others, ultimately requiring yet another set of tools to try to move the data into other systems to support other workflows.
“An observability warehouse introduces a purpose-built data layer optimised for logs, metrics, and traces. By separating storage from compute, organisations can retain far larger volumes of telemetry while scaling query capacity independently as data volumes grow. This approach allows observability data to remain accessible across tools and workflows without requiring duplication or costly rehydration from cold storage, concluded Tschetter.
The company notes that Imply Lumi introduces an observability warehouse designed to support high-volume telemetry while preserving fast query performance. By providing a scalable data foundation beneath existing observability tools, it enables organisations to retain more telemetry and evolve their observability architecture as data volumes continue to grow.