We need smarter and better chatbots. As we humans now ‘talk’ to our computers via an increasing number of chatbot services, the need for conversational style ‘communication’ in natural language will increasingly become a key interaction channel between humans and machines. If we accept this proposition to be true, then how do we build the newer smarter software machines in the realm of Large Language Models (LLMs) that work with the new breed of generative Artificial intelligence (AI) today and tomorrow?
Vignesh Subramanian, VP of product development at Infor thinks we are about to witness a paradigm shift in the world of chatbots in relation to how we use LLMs today. Why is this so? Because he says, existing chatbots and digital assistants are pre-trained to answer a finite number of questions.
“For example, a self-service chatbot in a telecom provider’s website could perform a limited set of tasks such as providing remaining usage quota, changing your address, providing outstanding balance on your invoice or enabling global roaming on your SIM,” states Subramanian. “It could not perform slightly more intelligent tasks, such as reasoning why your current bill is much higher than your last month’s bill or confirming that bank account details are changed for the next invoice.”
Composite tasks are more complex
The chatbot finds these more complex queries tough because they are more analytics and they require a comparison between values. They also present an obstacle because the request is a ‘composite task’ involving bank account details from the user’s profile and a cross-checking process with the billing system.
But, an LLM based chatbot can handle such tasks.
“What we need to remember is that LLMs are not search engines, rather they produce their output based on the data they are trained on,” Subramanian reminds us. “Regardless, the most popular use case for LLMs is building chatbots that end up treating the LLM like an enterprise search tool. All popular LLMs have an astounding amount of prefabricated general knowledge. However they are yet to be exposed to a business’s enterprise data with its many systems of records. Employee self-service use cases will require policy documents and user documentation, prompts from support analysts would need to go to incidents from a ticketing system and knowledge base articles, business users’ questions would need to go to data stored in enterprise software systems, queries based on market data would need to use LLM’s general domain knowledge and so on.”
What all that means is that, when a prompt comes in, the biggest challenge is to resolve the data sources and to determine the correct sequence of retrieving the response. Because the input for the second source may require additional data from the response of first source. When the final response is formed, it needs to be a cohesive answer produced from careful orchestration.
Human-like chat is tablestakes
As we can see, contrary to popular perception, providing a human-like conversation experience is in fact the least of the challenges here, that’s pretty much tablestakes i.e. the core basic need (or bet) thay anyone has to lay down to enter this game.
“As a developer building chatbots, the key is to create agents that can service complex requests. An ‘agent’ is essentially an application that creates cascading business logic leveraging a series of LLM prompts. Agents are responsible for decision-making throughout the session such as selecting the right data source, fetching the answer, interrogating further and moderating the final response,” clarified Infor’s Subramanian.
He further notes that while doing so, agents also maintain the memory chain of conversation, so that a follow-up question by user can still be answered within the same context. Agents also handle the technical plumbing behind the user interaction – such as identifying the user (who are they), their roles and privileges (do they have the authorisation to the requested data), managing the session (in case of multiple questions from multiple users simultaneously, ensuring that the right answer is delivered to the right question) and so on.
Opensource frameworks such as Langchain are emerging as a popular choice to build such LLM powered chatbots.
Inside a chatbot brain
“Further to all of this, developers must ensure data privacy (your data is not used to train the generic LLM), topicality and moderation (any question from user unrelated to the business context must be squashed immediately) and thwarting prompt-injections (fooling the LLM to generate a malicious response with a deceiving prompt) are inbuilt into the architecture as table stakes,” concluded Subramanian, suggesting that these guidelines can help build a safe, user-friendly, universal chatbot for enterprises.
The takeaway here is perhaps the need to ‘get inside’ a chatbot brain and think about a) the requirements and requests we are making of these software machines while also b) appreciating the limitations on the data knowledge and systems of record we have armed them with while also c) considering the cross-referencing composite complexity of the questions we pose in the first place.
Free image: Wikimedia Commons