14 min Analytics

The state of automation in 2024

The state of automation in 2024

Automation is everywhere, obviously. The machine intelligence that started out in the 1960s in the experimental software engineering labs that grew alongside what became the PC revolution has been through various stages. After a relatively quiet period at the end of the last century when artificial intelligence (AI) was largely confined to the movies and rarely discussed alongside machine learning (ML), AI finally started to emerge as a specialist autonomous software systems accelerator with Robotic Process Automation (RPA) gaining the limelight. Now in the new era of generative AI (with predictive and reactive AI still very much with us) and the proliferation of Large Language Model (LLM) technologies, AI has broken into our lives at iPhone-level penetration with bots (both chatty and the less vocal kind as well) working away to make our lives better and lower the barrier to entry for humans to perform all manner of tasks. Where then do we stand with the state of automation today?

“While today there is so much focus on AI, many are missing out on another great efficiency opportunity: automation in a practical business sense,” argues Jakob Freund, co-founder & CEO, Camunda. “Ironically, one of the reasons that businesses haven’t maximised the automation opportunity is because one type of automation caught on so quickly that it created misconceptions about the entire category.”

Freund is referring to the above-mentioned practice of RPA, which allowed businesses to quickly and simply automate many of the tasks that had previously been manual or even paper-based. Businesses were applying RPA bots wherever they could and they did have some great results.

“It’s when some companies eventually came to have this mass of RPA bots that they started bumping up against the limits of the technology – and, unfortunately, rather than seeing that those limits were imposed by RPA itself, many came to see it as a limit on what was possible through enterprise automation at large,” suggested Freund. “This put a glass ceiling on automation and really slowed the pace of adoption of other automation tools. How to break through that ceiling, lies in looking at why RPA stalls out – why its effectiveness doesn’t scale. Simply put, businesses max out all that they can with RPA and suddenly they have a lot of tasks separately automated but with no connectivity between them. There’s no rhyme or reason – just a bunch of robots doing this one thing they are supposed to do. We call this the spaghetti bot problem.”

He proposes that the way to fix this issue is to accept that pieces of automation can still happen individually. They just can’t be siloed if a firm wants to maximise their impact. For Freund, there is an opportunity to lift automation to the next level, by not just automating single tasks, but by doing so with a tool for process orchestration to be able to automate processes holistically.

Orchestration-driven automation evolution

In his role as vice president for global solutions consulting partners at Appian, Sathya Srinivasan proposes that the evolution of automation has seen us progress from process-centric and task-centric models to more sophisticated orchestration-driven automation that is embedded with AI. 

“Assessing the state of automation in 2024 therefore presents a paradox: while the concept of automation is straightforward, achieving effective and rapid implementation remains a challenge. This complexity is partly due to the myriad of automation solutions in the market, the evolving nature of automation technology and the confusion of introducing new buzz terms associated with automation, namely AI itself and generative AI,” advised Srinivasan.

He insists modern process automation technologies should be API-first, AI-assisted, process-driven and cloud-native. This shift ensures better integration, scalability and efficiency, making automation more adaptable and responsive to business needs. However, this is not always the case.

“Most organisations are still stuck with legacy (sometimes even green screen) technologies where integration is challenging, if not impossible. RPA tools still have a role to play in such use cases to drive integration to these legacy systems and enhance automation. Whilst RPA was the shining new automation tool a few years ago and it has been deployed by many companies now, it is often used as a tactical fix to an enterprise automation challenge,” said Srinivasan. “The future of automation will likely be autonomous and intelligent. It will feature embedded intelligence, allowing systems to learn and adapt alongside autonomous orchestration. It should also enable automated processes to identify tasks for automation without human involvement and opportunities for AI to assist humans to be efficient in completing their tasks by providing insights, predictive analysis and recommended actions from existing data.”

Composable architectures

For this future automation vision to flourish, Appian’s Srinivasan says that it needs to be built on a composable architecture, providing flexibility to create and modify processes easily with an adaptive user experience, ensuring that automation solutions remain user-friendly and user-centric. He says that as businesses embrace these advancements, the role of AI-driven autonomous orchestration will become increasingly pivotal. This evolution will redefine business operations, fostering greater efficiency, agility and innovation and ultimately pave the way for a new era of intelligent automation.

Largely concurring with Srinivasan and drawing upon his own firm’s well-documented work in RPA platform development is John Kelleher, vice president for UK & Ireland at UiPath. Explaining that business leaders require enterprise automation for end-to-end processes such as order-to-cash for the CFO, procure-to-pay for the COO and hire-to-retire for the CHRO, he underlines the fact that these leaders want solutions that help them understand, automate and operate these processes across their entire organisation. 

“Today we can say that ‘traditional’ automation is evolving into process orchestration, an emerging category at the convergence of several established categories like automation, IPaaS, LCAP, BPM, IDP and workflow automation,” steered Kelleher. “End-to-end process automation involves capabilities from each of these categories. Documents and communications within organisations are integral to every enterprise workflow. Insights from these workflows need to be leveraged more easily by automation developers and business professionals alike through the design surfaces they are already operating in.”

But, he thinks, we are only at the beginning of a powerful shift where context is applied to business and process-specific data to allow automation to take actions that are real, relevant and impactful. The UiPath VP says that specialised AI, which focuses on specific tasks and is trained on task-specific data, offers distinct advantages in enterprise AI compared to large foundational models trained on general knowledge. 

“Specialised AI is trained using an organisation’s data and optimised for its specific needs, resulting in more accurate and tailored solutions,” added Kelleher. “For example, in the healthcare industry, an LLM that’s securely trained specifically on deidentified medical data, can unlock patient data and other critical information inside the unstructured text of hundreds of pages of medical records and provide a clinician-level summarisation of a medical record that’s human-managed and organised in easy-to-understand segments with traceable citations. This means professionals in healthcare organisations can find the information they need within voluminous medical records within minutes versus hours or days.”

Leaping limitations

The industry appears to widely agree with the suggestion that RPA remains a key component in the automation toolkit, largely because it plays an important part in reducing human effort in narrow and repetitive tasks. But despite significant investment and evolution in the capability since the technology’s inception, it still has limitations, limitations which AI is now overcoming suggests Steve Ponting, UK&I and South Africa director for SoftwareAG.

“As a result, we are seeing automation move from back-office into front-line customer-facing processes such as the use of chatbots in customer services and personalised product recommendations.,” said Ponting. “This is where process intelligence can help businesses fully maximise the impact of automation. Process intelligence provides insight to business and technical leaders so they understand how processes operate and the level of variation within them enables businesses to choose which processes to automate. Beyond providing a full picture of the processes that are in play, businesses can also simulate how automation will positively impact the performance of that process’s execution and help in the selection of the appropriate automation tooling. AI can be leveraged to generate new optimised processes or make recommendations for improvement based on mined process insights.” 

With all of this in mind, the SoftwareAG director says that to ensure the inclusion of the focused automations has not had a detrimental effect on the end-to-end process, process intelligence could be used to provide a holistic view and monitor the performance. This prevents bottlenecks or human effort constraints being pushed downstream.

Automation is nothing new

“Automation – in the widest sense – is not new,” notes Crystal Morin, cybersecurity strategist at Sysdig. “Processes across manufacturing, data analysis, coding and commercial piloting have been automated for a long time. We even have unmanned aerial vehicles (UAVs) and self-guided missiles. Automation removes time-consuming and tedious processes for humans, or at least reduces them. What has changed is that automation is available to the general public for the most mundane tasks — it allows a model to do repetitive or simple tasks and frees up time for more interesting, more creative work for the human brain.”

She says that despite the many automation advances around us, it’s important to remember that attackers are benefitting from automation too. Cyberattacks now clock in under 10 minutes and they’re getting faster. Why? Attackers have automated their efforts to the point where they no longer feel the need to hide. If they can download malware or exfiltrate data fast enough, they don’t care that you’ve been alerted to their activity. 

“If you want to keep pace with attackers, you need to do two things: automate your security processes and secure your automation,” said Morin, bringing her clearly security-centric viewpoint into the auto-acceleration discussion. “According to several recent surveys and studies, a majority of companies are using or plan to begin using AI tools in 2024. Those companies need an actionable plan for securing their AI adoption. In other words, they need to keep their automation safe. On the other hand, securing your infrastructure is only half the battle — outpacing attackers means finding opportunities to speed up your security processes. Automating some of your detection, investigation and response efforts gives you back meaningful time and every second counts when a threat actor is in your environment.”

No set & forget automation

As an experienced software engineering specialist with a trained view on system strength and solidity, Morin says that there will be a consequence if we ‘set and forget’ automation. Our automation code and models (and our automated security processes) must be regularly audited to ensure the output quality remains high before perhaps data poisoning goes too far. She further suggests that there should be a concern, too, with an over-reliance on automation. Automation should be used to improve innovation and free up time — think about it in the context of your work-life balance. Automate the mundane tasks that add hours to your workday but don’t require critical or creative thought. 

“Why memorise times tables if you can automate that work and work on mapping out a cancer genome instead? Commercial flights are automated, however, landing is not. Pilots do not nap when the plane flies, they are constantly auditing the autopilot telemetry mid-flight,” highlights Morin, perhaps comfortingly. “All of the pessimism and hesitancy leading people away from embracing automation will eventually give them FOMO. Embracing automation for the time-consuming yet necessary tasks frees up your availability for more creative and innovative work projects, it also gives back time for hobbies or your children’s extracurricular activities. Automation won’t take your job, but it will allow you to expend more effort on your more critical tasks.”

The rise of AI and advanced automation technologies has made it easier, cheaper and faster to automate complex workflows. But we should point out, it has also simultaneously created a critical need for better risk management. As we delegate more tasks to autonomous systems, we lose visibility and control over how and why decisions are made, especially in the business-critical processes that encode a company’s unique logic and competitive advantages.

“While plenty of people are worried about AI as an existential threat, not enough are focused on more practical concerns, like what happens when it generates incorrect data, makes suboptimal decisions, or fails in unexpected ways,” said Jeremiah Lowin, CEO of workflow automation company Prefect. “It’s unrealistic to expect AI to be flawless; like any technology, it will have its limitations and failure modes. The key is learning how to embrace these technologies in the real world, with all their imperfections. That’s why it’s imperative that we develop methodologies, software and best practices that help us automate with confidence, even in the face of uncertainty.”

Stability of standardised systems

To build truly resilient and trustworthy automation, Lowin thinks that we need to start by designing standardised systems of accountability and efficacy. After all, we can’t manage what we can’t measure. This means investing in monitoring, transparency and governance frameworks that can give us insight into these increasingly autonomous systems. It means designing for failure from the start, not as an afterthought – and it means recognising that the path to trusted automation lies not in blind faith in the perfection of our tools, but in the hard work of building systems that can adapt and recover when things inevitably go wrong. 

“Only by embracing this mindset can we harness the full potential of AI and automation while still maintaining the confidence and control we need to depend on them in our most important workflows,” suggested Lowin. “AI is not magic and it’s not a black box. It’s a powerful tool that requires careful engineering, rigorous testing and constant vigilance to ensure it behaves as intended. By treating AI and automation with the same discipline we apply to traditional software systems, we can unlock their transformative potential while mitigating their risks. However, traditional approaches are not sufficient for the unique challenges posed by these newer, more autonomous technologies.”

Clearly a person who has spent a lot of time thinking about risk management, Lowin insists we need fundamentally new paradigms for resilience engineering, starting with clear metrics for success and failure, to truly embrace AI and automation with confidence. It’s a proposition that resonates with the thinking of Karel Callens, CEO & founder, Luzmo, an AI-centric customer analytics platform company. He suggests that as we move on now from traditional RPA, we can elevate ourselves past its specificity and lack of connectivity. 

Unshackled robots

“In the early years of automation, you could program a bot to perform one task in isolation,” said Callens. “The limitation of this was that at a certain point the stacked automations became unwieldy to manage and if a component in the chain was broken it could ripple outwards through systems and cause unforeseen issues. This is why tools like observability became so important to overview these interconnected systems. In the post-AI era, the shackles are off the robots. Modern AI agents are multimodal, able to adapt to different tasks in a variety of formats.”

He cites his own team’s work to develop tools that reduce dashboard build times by scanning simple, hand-drawn drafts. Users can draw a picture of a desired dashboard and convert it to a working dashboard in minutes using a phone camera. Traditional tools would require a precise draft it wouldn’t be worth it because it would be faster to build it in-app. Now, AI can interface and automate the process of taking a draft and then users can fine-tune specifics. These bots aren’t designed around one use case or specific tech stack, they are tech-agnostic and live inside existing tools.

The future for automation appears to be modular, composable and environment-agnostic in an essentially cross-platform, cross-cloud, cross-AI engine and cross-functional enough to span multi-model multi-modal AI deployments that work in an essentially more networked, connected and orchestrated way than the toolsets of the past. AI is no longer just in the tech lab, in the movies or in our aspirations for the platforms of tomorrow, it’s in our pockets and on our desktops. Intelligence is everywhere, so let’s be smart about how we harness and use it as the next tier of development happens in this space.