With Elasticsearch, Elastic has brought a product to market that puts search at the center. What once started as a new, faster way of searching has grown into a highly scalable, powerful, and efficient way of providing insight into data. Now that the world is moving towards more and more AI, Elastic sees an opportunity to take on a bigger role. With the Elastic Search AI platform, Elastic shows that a good search engine leads to better AI results.
Elastic has experienced significant growth in recent years. Organizations were already using Elastic for web, log, application, and document search, but in recent years there has been a major shift toward making security data transparent and searchable. This is partly for better observability, but also to improve overall cybersecurity. If we are to believe Elastic, the success does not stop there. Elastic predicts that the success of many AI strategies will soon depend on good search.
Elastic’s growth through acquisitions and open source
In recent years, we have seen Elasticsearch experience rapid growth. This is partly thanks to the open source community, which also makes valuable contributions to all Elastic projects. The most important components of Elastic are currently the ELK stack: Elasticsearch, Logstash, and Kibana. For many users, Kibana is the solution that appeals most to the imagination, because it allows you to easily visualize data indexed in Elasticsearch with dashboards, graphs, and diagrams. However, you can also perform analyses on your deployments, zones, and indexes. This allows you to see whether your Elastic environment is performing well or whether you may need to add more CPU, memory, or nodes. Kibana is an important addition to the Elastic ELK stack that is used by virtually all Elastic customers.
Logstash is another important component, making it possible to easily index and enrich logs from all kinds of applications. Its use has increased significantly in recent years, especially in view of better observability and higher demands on overall cybersecurity.
There are also other modules, for example for machine learning. This is a special custom-developed node that is optimized to perform machine learning tasks as efficiently as possible. The Elasticsearch database is the breeding ground for training the machine learning model. Previously, Elastic also ran a separate Search module that originated from Elastic’s acquisition of Swiftype. Swiftype was best known for its SaaS solution for site search and workplace search. Website owners could use it to easily make their website searchable, and workplace search made a lot of company data in SaaS environments searchable. Swiftype’s technology has now been integrated into Elastic’s stack, making this functionality available in the core. Organizations can now combine the functionality that was previously in Swiftype with data in Elastic. As a result, the Enterprise Search module will eventually be discontinued.
Elastic owes a lot to open source and gives back to open source
In recent years, there has been some controversy surrounding Elastic’s license. It was initially open source (Apache 2.0), but after discussions with Amazon Web Services, the company decided to switch to a different, less open license model. Elastic felt that AWS was benefiting too much from Elastic’s developments and name, while giving little in return. That discussion is now over, and Elastic and AWS are friends again. Elastic is also available again under the AGPL license.
That’s a good thing, because Elastic owes a lot to the open source community. It also gives a lot back to the open source world, but it also confirms that they really belong there. Elastic earns its money by offering a hosted Elastic environment that it manages from one of the three major hyperscalers.
Elastic has also recently started offering a serverless version of its hosted environment, which takes all the hassle out of things for customers. Customers don’t have to worry about Elastic versions, patches, nodes, or configurations at all. This serverless version is also extremely scalable, making it very suitable for AI workloads.
Organizations cannot do without search in their AI strategy
Since the introduction of ChatGPT, AI has gained momentum. Organizations want to reap the benefits of AI and generative AI in the short term. However, after thousands of experiments with large LLMs, it has become clear that this does not happen automatically. An important part of effectively deploying AI in business environments is applying RAG and, of course, prompt engineering.
What you are actually doing with RAG is telling an LLM: “use this data to generate your answer.” The data provided by RAG must therefore consist of the highest quality factual business data available.
RAG
Organizations are experimenting with RAG and linking all kinds of databases with business data to models in the hope of achieving better AI applications. Some things sound logical but often go wrong. For example, handing over all personnel data (HR) to an AI model via RAG is not wise. Unless you use role-based access, you don’t want all employees to be able to view confidential HR files, the salaries of colleagues and the CEO, or other sensitive information.
Or what about organizations that use RAG to supply the entire product database with hundreds of thousands or millions of products? This forces the AI model to plow through huge amounts of superfluous data, especially when the model has to generate something for one specific product. This causes significant delays but also much higher costs. The heavier the task for an AI model, the higher the costs.
RAG via the Search AI platform
At Elastic, they are positive about RAG, but not about how organizations are currently applying it. The massive delivery of data via RAG is inefficient and creates complexity. Everyone now knows that the data must be of high quality, but that also applies to its relevance. User rights must not be overlooked either. The issue of compliance and governance is certainly just as important in Europe. Yet we often see and hear about experiments where these kinds of issues are ignored. To use AI effectively, clear rules are needed, so-called guard rails within which an AI model is allowed to operate.
Elastic claims to have another solution, namely a way to feed an AI model more efficiently with high-quality, relevant data. Elastic has developed the Search AI platform for this purpose. What Elastic does is not deliver all data via RAG, but make it searchable via Elasticsearch. In this way, every AI task or scenario first performs a search via Elasticsearch. The relevant output is then delivered to the AI model via RAG.
By searching first, Elastic can deliver an optimized dataset via RAG
Elastic can deliver data via RAG that is concise, relevant, and, where necessary, limited to user rights. This is done simply by applying search. It is still high-quality business data, but the initial exclusions have already been made. Based on user rights, only data that the user has access to is delivered. The dataset is also optimized and more efficient because the AI model is no longer provided with hundreds of thousands of irrelevant documents. This allows it to produce output faster and at a lower cost.
Search as a valuable addition to your AI strategy
Elastic makes a good case for adding search to your AI strategy. It makes sense from a cost perspective alone, but it also ensures governance and even the privacy of customer data. Elastic has taken a step in the right direction with the Search AI platform.
For some readers, this may immediately raise questions. How do you get the necessary business data into Elastic? It’s easier than you might think, because Elastic now has Swiftype’s workplace search component at the core of its stack. This also includes the necessary Connectors managed by Elastic. This allows you to easily connect data sources in the cloud to Elastic. Examples include Box, Confluence, Dropbox, Salesforce, ServiceNow, Sharepoint, Slack, Microsoft SQL Server, Microsoft Teams, MySQl, Oracle, and Zoom. Organizations can also easily build their own connectors and use them to import data into Elastic.
Internal organization-wide search engine
With the Search AI platform, Elastic not only offers a search function to enrich AI models. It also offers an internal search engine for employees to search through all company data. AI comes into play here too, because by using a connector, you can also link an LLM to it. This allows employees to ask questions, to which the AI model generates an answer based on the available company data. Document level security (DLS) is also important here. Organizations must be able to properly manage access to data at the user level.
The Elastic Search AI platform is therefore unique in the market and offers a solution that many other parties have not yet thought of. Although organization-wide search engines are not new, we suspect that the Elastic Search AI platform will become part of the total AI stack in large organizations. The search will be used as described above to achieve a better RAG dataset. The application logic for the AI scenario, the search query, and the interaction with the LLM will then be located elsewhere.
Is Elastic the missing piece of the puzzle in the AI strategy?
Elastic can certainly solve certain problems in organizations’ AI strategies. Whether it is suitable for everyone and whether there are better alternatives is difficult to say at this point. Developments in AI are happening so fast. What we write today may already be outdated tomorrow.
What we can say with certainty is that having a powerful internal search engine within an organization is becoming increasingly important. Many different SaaS platforms are now focusing heavily on this, as they hope to centralize all business data in this way. This will enable them to take the lead in the AI strategy of many organizations.
The question is whether you, as an organization, should want that, or whether you would be better off choosing Elastic in such a case. Then you can at least be sure that you are entrusting your business data to a player that specializes in search. Storing data centrally is one thing, making it easily searchable is another. With an online platform where the expertise lies elsewhere, that remains to be seen. With Elastic’s basic connectors, you can already make a lot of business communications and business data searchable while retaining user rights. Especially when you consider the investments Elastic has made in semantic search. We are confident that a search query within Elastic will offer more than any other platform.
The next step could be to use Elastic Search for an AI scenario via the Search AI platform. Elastic then links data to an AI model via RAG. Organizations that also have a lot of unstructured data and need vector search are also in the right place with Elastic. In recent years, Elastic has made great strides in the field of vector search, which is essential for unstructured data. However, that is a separate story that we may explore in more depth in the future.
All in all, with its Search AI platform, Elastic has put an essential component in the market for the AI strategy of many organizations. It shows that the power of search can lead to better AI results.