We know that technology moves fast and that Artificial Intelligence (AI) tooling has gone mainstream over the past year in particular. It has created a significant shift for developers who are trying to ‘drink from the firehose’ of available technologies to implement at the coalface (well, the command line) of code creation. Instead of staying in their lane, software engineers are now being asked to reach outside their comfort zone to become AI and Machine Learning (ML) experts. While Large Language Models (LLMs) and generative AI have received the lion’s share of attention, recent years have also seen a quiet revolution in a somewhat different field. Edge AI, the combination of Artificial Intelligence with embedded and Internet of Things (IoT) devices, has become a near-ubiquitous technology in a short period of time.
Today, almost every mobile device uses a special blend of edge AI to power digital assistants and improve core technologies in this space such as sensor configuration and camera imaging – and there are myriad applications here touching everything from heavy industry to healthcare.
This change has been enabled by an unlikely alliance between AI/ML engineers and embedded software engineers, along with the designers of embedded silicon. But in order to build together, engineers from widely separated domains have had to learn to speak each other’s language.
Practical application leap
“There’s another challenge, in that much of the bleeding edge of AI and ML has been more academic and less practical in nature. Now that it’s spilling over into mainstream use cases, that personality clash can make it even tougher to onboard developers learning these skills for the first time,” said Daniel Situnayake in his capacity as head of machine learning at Edge Impulse, an edge AI platform for enterprise teams building products that incorporate both AI and embedded technologies.
Culture and expectations change over time. Until recently, AI specialisation had been more prevalent for engineering projects. But now developers have to learn about AI, DevOps, infrastructure and other arenas that may be completely foreign to them.
This is a big factor in the rise of platform engineering and the mission many technology vendors voice as they work to make it easier to stretch a developer’s skills across these various disciplines. But we need to think deeper still i.e. building for embedded systems brings its own special problems.
Why AI in IoT is hard
The limitations of embedded ML [that which is typically carried out on IoT devices, also known as the computing ‘edge’] are particularly challenging. There’s a big difference between computation done at the edge and computation done in a centralised way [inside a corporate datacentre] that most developers are used to operating. Situnayake suggests that the constraints of building for the edge are often unfamiliar to AI and ML engineers, who are used to having high compute resources to work with and a lot of processing power, in addition to being able to serve models using the same frameworks and tooling they used to create them.
“You have the exact opposite with the edge: scarce compute resources, low bandwidth and processing power, minimal storage space to utilise, plus the added problem that special software tooling is required to run models on resource-constrained devices that is often very challenging to use,” said Situnayake.
Embedded engineers that specialise in working with edge devices are similarly ill-equipped to build with AI and ML. While they often have experience with digital signal processing, most embedded engineers aren’t experienced with the tools and workflows required to build ML-ready datasets and train and evaluate effective models.
BLERP: Bandwidth, Latency, Economics, Reliability, Privacy
Overcoming these obstacles is a worthy endeavour. Edge AI is a technology with the potential to mitigate some of the biggest downsides of legacy AI, enabling new use cases in the process. Its key benefits are nicely captured by the mnemonic BLERP: Bandwidth, Latency, Economics, Reliability and Privacy.
“Essentially, by running AI algorithms locally on-device instead of remote servers, systems are able to make smart real-time decisions without the complexity, cost and privacy implications of streaming data to the cloud,” said Situnayake. “In addition, they can be faster, more responsive and support use cases where super-fast decisions are required.”
Companies are applying these benefits to create new products. For example, Know Labs is using edge AI to interpret the output of radio frequency sensors on a healthcare device that can read a patient’s glucose levels through the skin. Running models on the edge allows this device to work without a network connection, guaranteeing reliability and privacy. Izoelektro’s RAM-1 uses edge AI to monitor remote power distribution infrastructure, watching for equipment failure in places with limited connectivity. It also saves engineers a trip into the wilderness.
Situnayake reminds us that many developers find these kinds of projects exciting and they welcome the opportunity to learn new skills – but without the right tools this process can feel overwhelming and unproductive, plus it may require prohibitive amounts of training.
Successful Edge AI implementation
The Edge Impulse team recommends a process that centralises around streamlined workflows to make it easier for any developer on an edge project to build outside of their comfort zone. Then they can quickly gain confidence in applying AI and ML to edge devices and embedded ecosystems, allowing them to focus on applying their domain expertise rather than spending their limited time wrestling with unfamiliar tools and concepts.
“The key to building those workflows is to understand the different types of developers involved, how they think and what kind of tools they need in order to make the transition. In the O’Reilly book I co-authored, ‘AI at the Edge’ we recommend that the ideal team makeup should include embedded engineers, ML engineers who are experts on machine learning, domain experts and experts in AI ethics and fairness – in addition to the product management, engineering and technical services roles required to build any modern product,” said Situnayake.
Having the right team in place is the first step. The second is to implement the tools and workflows to help them get to success fast. Quick start guides, Software Development Kits (SDKs) and test development environments built to account for both the edge and AI/ML side of things will help a developer feel at home from the beginning as they gradually learn more to fill in their skills gaps. Ideally, tools should be designed to play to a user’s strengths, supporting them in doing new tasks, but ‘getting out of the way’ when they prefer to use an existing workflow that they are already productive with.
Sharpening the Edge edge
“As a developer, the luxury of being in your own little pocket is starting to disappear again, especially for the edge. With new tools focused on good DX, it can be easier to confront the unfamiliar and feel confident learning skills that are quite new to us,” concluded Situnayake.
As a relatively new field, developers are still getting familiar with best practices for edge AI. Successful projects require input from numerous people, including domain experts, ML engineers, embedded engineers etc. Coordinating work and knowledge transfer between these groups can be challenging and effective tooling can often help bridge those gaps – while ensuring the latest best practices are followed.