AWS has added a new log analysis engine to Amazon OpenSearch Service. The new technology is designed to help organizations process the rapidly growing volume of log data more efficiently. According to AWS, processing speed doubles, storage costs drop significantly, and no additional license or premium service is required.
The new engine is primarily aimed at observability and monitoring environments, where the volume of log data is rapidly increasing due to cloud-native applications and AI workloads. According to AWS, the volume of log data at many organizations is growing by 30 to 40 percent annually. As a result, DevOps teams are increasingly having to invest in additional storage or supplementary analytics tools, or are forced to delete log data sooner than they would like.
New storage architecture
To accommodate this growth, AWS has modified the storage and processing architecture of Amazon OpenSearch Service. Whereas log data is traditionally stored using inverted indexes, the new engine uses the column-oriented Apache Parquet format. This enables more compact storage and significantly reduces the amount of storage space required.
AWS uses multiple open-source components to process search queries. Apache Calcite automatically determines which query engine is best suited for a given task. Analytical operations on columnar data are handled by Apache DataFusion, while traditional search queries continue to run on Apache Lucene.
In addition, the engine supports both SQL and Piped Processing Language (PPL) with the same full-text search capabilities. This allows users to combine complex analyses and targeted searches within a single query.
Better performance at no extra cost
According to internal AWS benchmarks, the new engine can process approximately 1.78 million documents per second. The tests analyzed 24.4 billion documents and 9.5 terabytes of JSON data. AWS states that this throughput is approximately twice as high as that of an implementation based solely on Apache Lucene.
The price-performance ratio is also said to improve. AWS reports an average of approximately 70 percent lower storage costs and a fourfold improvement in price-performance for analytical queries on large volumes of log data. These claims are based on internal measurements; independent benchmarks have not yet been published.
Available within OpenSearch Service
The new log analysis engine is available immediately in all AWS regions where OpenSearch Optimized Instances are supported. Users select the engine when creating a new OpenSearch domain by choosing “observability” as the primary use case.
AWS does not charge any additional licensing fees for the new functionality. Customers pay only the standard rates for compute capacity and storage.