Shortly after Meta shook up the AI world, Mistral comes out with a new competitor. Mistral Large 2, smaller than the 405 billion parameters of Meta’s Llama 3.1, is said to be “large enough” according to its creators. Is that true?
First, a number of similarities between Large 2 and Llama 3.1 stand out. For example, both models include a context window of 128K tokens. Also, the LLMs’ benchmark scores are often very close. They excel in different fields, which allows both AI players to fill out a use case.
Tip: Llama 3.1 is the largest model: turning point in open source AI?
Mistral Large 2 contains 123 billion parameters, more than three times fewer than Llama 3.1. Still, benchmarks show that competitive performance is possible with a smaller model.
Good at coding
Mistral already has some experience with specialized coding LLMs. Codestral 22B and Codestral Mamba already provided superior AI programmers versus Meta’s Code Llama. Now, Mistral Large 2 surpasses these specialists in code generation and mathematical problems. In HumanEval, a well-known AI benchmark, it only has to beat OpenAI’s GPT-4o. Large 2 excels particularly in Java and scores just slightly lower than GPT-4o with C++ and TypeScript. Its predecessor Large 1 can’t really compete: its average score of 58.8 percent falls far short of Large 2’s 74.4 percent.
Language node
Llama 3.1 needs 405 billion parameters to surpass Large 2, the Multilingual MMLU benchmark shows. The variant with 70 billion parameters always loses out to Mistral Large 2, which stays within a few percent in multilingual scores.
Not open-source
Although researchers are allowed to use Mistral Large 2, it is not actually open-source. Organizations must turn to Mistral for professional use or to build applications on top of it. In that, the setup differs from Meta, which also makes Llama 3.1 available for commercial use.