With the acquisition of Taalas, AMD has bought itself a “pioneer in specialized AI inference silicon”. Just weeks ago, details emerged of an internal Google “Frozen v2” chip that hardwires parts of the Gemini model architecture. As OpenAI and Anthropic, too, are ramping up to build their own processors that will presumably also be “LLM-shaped”, the question arises: what are the benefits? And, more importantly, what other constraints remain?
We’ve seen accelerating sophistication when it comes to running large language models for some time. Deals with both AWS and AMD have highlighted the abilities of Cerebras’ enormous chips in particular parts of AI inferencing, the workload that ingests tokens and generates them in response. The former, prefill, is heavily compute-restricted, whereas decode is limited in its output tokens by memory bandwidth. It therefore makes sense to split AI inferencing. More generally, though, the developments we listed above are more ambitious. They aim to optimize inferencing with what The Register labels “MSICs”, or Model-Specific Integrated Circuits. It makes a lot of sense, and very little, at the same time.
Why etching models makes sense now
Like we said, inferencing is bottlenecked in different areas. But the underlying cause for these bottlenecks to exist in the first place is a design mismatch. GPUs, TPUs, Trainium, even the supposed AI-focused NPUs inside some PCs, they all possess some degree of optimization for the AI calculation workhorse that is matrix multiplication. They perform these calculations in parallel, with tens of thousands of cores able to chip in. Several limitations exist that constrain the size of individual processors, the speed of copper connections between them, and much more. We needn’t delve into those too much here, but suffice it to say that you wouldn’t spec out your hardware the same way if you just wanted to run LLMs on it.
This is where something like the Taalas HC1 Technology Demonstrator comes in. Built on TSMC’s somewhat older 6 nanometer node, with just a quarter of the transistor count of an Nvidia B200, it is capable of running Llama 3.1 8B at 16,960 tokens per second per user. That is, quite frankly, a staggering number. That is 48 times the speed of the aforementioned B200 from Nvidia at 353 t/s. Cerebras, identifying itself as producer of “the world’s fastest AI”, is 8.5x slower at 1981 t/s. It needs 75 times more transistors than Taalas to reach that speed, on a smaller TSMC node no less.
Then again, there is that “Technology Demonstrator” part. Llama 3.1 at 8 billion parameters was a comparatively weak model relative to the AI frontier… back in 2024. The point, of course, is that you could convert any small model into silicon like this, with similar astounding speed gains. This is where AMD’s acquisition, the terms of which were not disclosed, comes in. Despite focusing its MI400 Instinct GPUs on AI inferencing more than training, there’s good reason to hedge on other chip designs. Nvidia has done the same with its 20 billion dollar acqui-hire of Groq a few months ago, and Google’s aforementioned Frozen v2 points to the same diversification away from just TPUs and GCP-hosted GPUs.
The general consensus has clearly shifted to the etching of models into silicon, albeit in an experimental way, while diversifying away from one monolithic parallelized processor is already providing real-world gains on any LLM. Claude Fable 5, the current state-of-the-art LLM available to just about any paying customer, runs on several kinds of silicon (specifically GPUs, TPUs and Trainium). The logical next question is: what would the world look like if not Llama 3.1 8B, but Fable 5 at however many trillion parameters, could run at warp speed?
Admittedly, there is an inherent lag in model-specific chips. LLM generations last about as long as those of fireflies. Designing, testing, manufacturing and scaling up a chip still takes many months, often years. We are assuming that the use of mature chip nodes, reproducible designs and co-engineering can reduce this lag, but this is a real problem for model-specific chips becoming an everyday reality. Nevertheless, we’ll put this to one side for the moment.
Speed is not (just) the answer
It will be an exciting day when OpenAI finally makes its promised 750 tokens/second version of GPT-5.6 Sol available. It’s running on Cerebras’ gigantic wafer-scale engine, and already available to select preview and priority customers. The offering is listed as being “up to” 10x faster than other instances, mostly running on Nvidia GPUs. Under certain conditions, having such a capable AI model available at these high speeds will unlock brand new use cases. Examples include the fastest possible complex analyses of financial trades, high-speed data extraction and AI-to-AI communication. The last one needs a bit more explanation: right now, as AI expert Andrew Ng already highlighted two years ago, AI consumption is primarily human-directed. At higher speeds, nearer to computational timescales (i.e. microseconds, nanoseconds, not 200 milliseconds), inferencing can smartly adjust workloads, intelligently route networks, achieve compliance, filter noise, et cetera, all at runtime.
Right now, this is impractical even with tiny AI models. These smaller models are still simply too slow to start, as they load into the memory they then hog, and remain somewhat untrustworthy options compared to deterministic solutions and bigger, slower models. The algorithms currently running behind the scenes in everyday use cases are not generative AI-based or indeed consisting of billions of parameters for the most part. They are already optimized to be relentlessly small.
There are still many, many more limits beyond just LLM inferencing speed however. The easy one to wrap one’s head around is that of light: anything you aren’t running directly on your own machine has a natural speed limit, which becomes relevant for cloud workloads. This is generally fine for users engaging with LLMs, as token generation and reasoning effort may consume many minutes. When these token generation times approach zero, other bottlenecks emerge. Those go beyond light, especially for agentic workflows.
Think of any time you’ve faced noticeable latency on a computer. That happens often, and is often acceptable. If you’ve ever taken note of a long-running agentic workflow, these constraints (a page loading, an API or MCP call) will quite regularly not be the biggest limitation. But with LLMs running at thousands of tokens per second or more, any 500-millisecond wait is positively huge. If you were to plot out the token generation of these long-running, tool-calling workloads, they would currently show a consistent peak, a temporary pause, and then a return to near full utilization. With MSICs, you’d see them ramp up quickly, then see a giant trough by comparison, followed by another very brief spike. This won’t always be the case, but AI systems will run headlong into legacy speed barriers constantly.
Etching the world?
You can’t etch into silicon every future bottleneck to make it go away. Apart from the speed of light, which isn’t even the current constraint for the copper wiring in and around AI accelerators, AI’s engagement with current IT infrastructure will remain a persistent problem. We’ve seen CPUs step up their AI game, memory pools expanding to meet rising parameter counts, bandwidth limits removed by new networking standards, and many advances in and around LLMs themselves. But to turn any given AI model into silicon, even the most advanced ones, is to present yet another major hurdle to overcome.
There’s some talk as well of AI companies turning to these etched solutions because of a supposed lack of conviction internally. After all, if insiders assume that LLMs are approaching their technical maximum, the first thing they’d do is optimize the running of them to lower operating costs. However, legislation and an emerging desire in the AI industry to slow down model advancement are additional arguments for sticking with whatever the current state of the art in AI is for longer. We will continue to see better LLMs that perform tasks more consistently, and more use cases for them too.
Conclusion
A shift towards faster models and comparatively slower infrastructure brings with it different considerations. As much as tool-calling has revolutionized AI usefulness over the past year, it will become more sensible to have the LLM do as much work as it can without API/MCP calls or tool use. Perhaps this will give someone the bright idea to further optimize the tools themselves, or even bake them into silicon as well…
There’s a bigger point to make here. As much as the adoption of AI is lagging behind the marketing campaigns and software integrations, we’ve not tapped the full potential of even yesterday’s AI models, let alone those of today. Just think of the useful applications the IT industry would have come up with had they been told that there would just be, say, GPT-4 or o1 or indeed Claude Fable. Give these LLMs time and you can work around their limitations, optimize the world around them and build the silicon to run them as fast as nature will allow.
Right now, it appears we’re getting a glimpse of that. Expect LLMs to advance more slowly, but for the infrastructure running them to leap ahead by comparison. This gives time for the AI buildout to align properly with demand, and for chip designs to optimize for a known workload. The silicon will become more expendable as its architectures age out quicker than current GPU generations do. However, the desire to make full use of the hardware will power advances in other areas, and point the finger at other bottlenecks. Put in two words, that bottleneck is legacy infrastructure. That includes both software and hardware still in the mix as cloud workloads, tools and data sources traverse them. Solving those non-AI speed limits is the long-term challenge if LLMs are to run, and not just walk.