There’s often talk of a breakthrough in the GenAI field. However, a new move by Aleph Alpha is particularly useful, especially for multilingual LLM application.
A new LLM architecture from Germany’s Aleph Alpha eliminates so-called tokenizers, which break up text and code for AI training into smaller tokens. Those applying fine-tuning on LLMs cannot deviate too much from the tokenization used previously, as it dramatically reduces the efficiency of the process. For example, GenAI models are regularly trained on the entire public Internet, where there is a strong bias toward the English language. Less represented languages are also more difficult to tokenize as a result. This then costs much more energy than necessary, according to Aleph Alpha.
AMD and Schwarz Digits/StackIT of Lidl owner Schwarz Group are also on board in Aleph Alpha’s restructuring of the generally accepted GenAI architecture. This shows that the initiative has an emphatically European character, especially since Finland’s SiloAI as part of AMD contributed to the development of this “turnkey solution” to take LLMs to a new level.
Sustainable and sovereign
The technical term for the new way of working is Hierarchical Autoregressive Transformers (HAT). Instead of splitting a word, LLMs are provided with processing at the level of whole words or single bytes. The precise explanation of this, like Aleph Alpha’s models themselves, is freely available. A somewhat more accessible story can be read on the German AI builder’s website.
Founder and CEO of Aleph Alpha Jonas Andrulis sees a great opportunity for sovereign AI models for every culture, industry and country. This extends beyond just Western languages: other alphabets can just as easily be used for fine-tuning without tokenization without becoming extremely inefficient.
It’s unsurprising for Aleph Alpha to come up with a breakthrough like this. ““I founded Aleph Alpha with the mission to empower the sovereignty of countries and companies around the world in the era of AI”, Andrulis explains. “For our customers, this means open-source integration, maximal transparency and trustworthiness for a new era of human-machine collaboration, future-proof transformative AI solutions and free choice of execution environment.”
In other words, we should not expect a move similar to OpenAI’s toward a closed solution, perhaps listening to the name ChatHAT based on the new breakthrough. Other AI players are free to adopt HAT for a more flexible suite of models for any application.
AMD factor
Another interesting feat of the Aleph Alpha innovation is that Nvidia is nowhere to be seen, aside from benchmarks in which its hardware is outperformed by the competition. Indeed, combined with an optimized ROCm stack, performance on AMD Instinct MI300 chips is significantly better than on an Nvidia H100.
LLM benchmarks for the efficiency of AI training were conducted in both English and Finnish. Why Finnish in particular? First, there is a logical connection to Silo AI, the Finnish AI player acquired by AMD in 2024. Finnish is also a particularly tough challenge: like Hungarian, it is an Uralic language, structured entirely differently and with origins different from Romance (including French and Spanish) and Germanic languages (English, German, Dutch, etc.). Without Aleph Alpha’s architectural breakthrough, such a language group would be extremely inefficient to tokenize.
“This collaboration brings more than AI – it delivers resiliency and innovation to the European AI ecosystem,” says Keith Strier, SVP, Global AI Markets, AMD. “We are thrilled to collaborate with Aleph Alpha and Schwarz Digits to boost Europe’s native AI capabilities and create a new AI trifecta for governments: a hyper-transparent GenAI platform, developed and trained within Europe, delivering exceptional efficiency on our AMD AI infrastructure.”