Update, 10/09, 09:15: The benchmark tests of Reflection 70B appear to have been tampered with. Independent AI tester Artificial Analysis posted its results this past weekend, claiming performance was the same as Llama 3 70B but “significantly lower than Meta’s Llama 3.1 70B.” According to the developer of the new AI model, Matt Shumer, this is due to uploading to Hugging Face, where performance is reduced, but even with the private API, Artificial Analysis could not achieve the results that Shumer achieved. So, nothing lost for OpenAI after all?
Original, 6/09, 11u43: Reflection 70B is the new leading open-source model. The model specializes in the technique of “reflection tuning,” what does that mean?
Reflection 70B is built on the foundations of Llama 3.1-70B Instruct. The LLM thus constitutes a product from the open-source community. Within this community, competition for commercial models is growing stronger, as Reflection 70B is said to score well on benchmarks. According to developer Matt Schumer, Reflection 70B is “the world’s best open-source AI model.”
Reflection tuning
The name of the company reflects which technique the LLM can apply. Reflection tuning allows LLMs to extract errors from their own reasoning. The errors are extracted even before the model has written down the full answer.
On the demo site, the capability is shown in a prompt asking how often the letter r appears in “strawberry. That is a reference to OpenAI’s GPT models, which are currently available and consistently answer this question with two. OpenAI has a model in the works that no longer makes these errors. The company has codenamed this model “Strawberry.” The project has long been heralded and, with Reflection, actually outdated before launch.
Sufficient computing power was found
A day after launching Reflection 70B, Meta had already contacted Schumer. This gives the model more computing power and capacity to handle the demand for the model. The new name of the model is Reflection-Llama-3.1-70B, and it can be found on Hugging Face.
Also read: Aleph Alpha’s open-source LLMs fully comply with the AI Act