Last week, Mozilla announced a new AI tool for its own Mozilla Developer Network (MDN). AI Help is intended to “streamline interactions” with the Web developer-focused Web site. Searching through the huge array of documentation should thus become a lot easier. Unfortunately, according to users, AI Help turns out to be more of an AI liar.
In recent months, more and more is available to programmers and other professionals regarding AI assistance. Security personnel can get more insights via Charlotte AI from CrowdStrike, while developers can be more productive with various GitHub Copilot features. Now Mozilla offers AI Help within MDN, which, like GitHub and many other tools, runs on OpenAI technology. Specifically, the tool in question is GPT3.5. In addition to generative AI, the tool uses “embeddings for similarity search.” Specifically, documents are linked on keywords and other similarities within a Supabase relational database.
The Register notes, however, that this tool has gone down the wrong way with GitHub users. While an AI assistant actually sounds like a good mainstay for novice developers, the tool “lies” at every turn. With a chatbot, we call this a “hallucination,” or making up facts. Ultimately, GPT 3.5 is a large language model, not a “knowledge model,” as one GitHub contributor rightly points out.
The annoying thing is that AI Help still sounds convincing while lying. This is a frequent problem that can also be seen with ChatGPT. Ultimately, these errors can only be fished out if you continue reading independently of the AI Assistant or already know.
All this would be a more minor problem if AI Help were completely free. It is if you only need to ask 5 questions daily and create an MDN Plus account. However, unlimited access just costs a minimum of 5 euros per month, and surely you should expect such a feature to come out only when it is working properly.
Too much responsibility
It is striking that the tool is inspired by an AI tool from Supabase. The so-called “ClippyGPT” was only available for a week to gather feedback. This was marketed as a “ChatGPT for Supabase Docs” and was introduced with some irony. Again, this was extensive, cluttered documentation that AI could potentially handle. However, the short duration of the testing phase was possibly good advice for Mozilla not to come up with a similar tool right away.
Thus, it seems that the tool has taken up too much responsibility for itself. After all, generative AI is certainly suitable for searching a database quickly, especially if they are not well-organized. However, it then functions like a librarian instead of also explaining the content. In that regard, Mozilla might choose to relegate AI Help to that functionality. Now it can provide rather novice Web developers with incorrect information that is of no use to anyone.