The popularity and interest in Artificial Intelligence (“AI”) agents (or “Agentic AI”) have boomed in the past couple of years. This idea of AI “doing” as opposed to simply “informing” has captured so much interest that large platforms such as AWS, GCP, Azure, LangChain, and others have developed services, packages, examples, blueprints, frameworks, and more to make starting your AI agents journey seamless.

At a high level, the structure of an agent with a Large Language Model (“LLM”) embedded usually looks something like this:

An LLM-powered agent is set up with a goal to reach or a task to complete. The LLM has access to a range of utilities, or “tools” such as relevant files, APIs, functions, etc. Given the tools and the task at hand, the LLM “decides” what action to proceed with using the tools at its disposal; this may repeat until the “desired” outcome has been fulfilled.

While the support these companies have provided can be great, it’s clear to engineers that the overuse and needless adoption of AI will increase exponentially, and, the likelihood and severity of risks will also increase at a similar rate. Organisations are making a critical error: they’re deploying AI agents as a solution in search of a problem, while traditional automation would deliver better results with lower risk in most situations. Here, we present an argument that, in many cases, especially those involving structured processes, the risks of adding an LLM-powered agent can outweigh the benefits. As much as we’d love AI to be a set-and-forget endeavour, now more than ever, we are learning why this will not work.

Determinism vs “free will” (stochastic programmes)

Arguably, one of the greatest debates in philosophy takes its own version in the AI world - does AI have agency? Meaning, given an input, would we know the output of an LLM without running it? Fundamentally, LLMs are deterministic (meaning the output can be calculated without running the input through the model). However, because we want LLMs to appear more realistic, creative, provide better outputs, and be able to handle a range of prompts, we train them with “randomness”. This introduction of stochastic sampling is what makes the final result of an LLM inherently unpredictable (see here for the mathematically curious). While this can be good for informing and fine when an expert is reviewing, when we consider the actual end-to-end completion of a task, the stochastic nature of these models can be dangerous.

For example, in customer support, AI agents can safely handle routine queries (eg, tracking orders, managing shipping changes, etc) - minor output variations are tolerable. However, in healthcare, even a 5% error rate in diagnosis recommendations becomes ethically unjustifiable. Here, we explore the latter example to highlight the intensity of the situation and potential complications.

If we want to build a medical diagnosis agent, we can equip it with textbooks (as part of its knowledge base), APIs to professional databases, lab results, a medical history, and potentially a recording of the initial consultation with the patient. Putting aside data privacy and violation issues, here, a “random” or inconsistent result can be harmful to the patient (eg, through a misaligned treatment plan). Now, one could argue that two doctors with the same level of experience and presented with the same information may provide two separate diagnoses, however, the misalignment would be due to nuanced judgement. Unlike a doctor whose decision-making process is grounded in years of contextual training, an LLM’s response is the product of, among other things, probabilistic sampling, often untethered from coherent reasoning or any true understanding.

The main issues of the stochastic nature of LLM-driven agents are twofold. First, it’s not possible to trace every step the agent took to reach its conclusion - this is essentially a black box. Second, regardless of whether a human reviews every diagnosis, an inexperienced doctor might look at what the agent produces and have their reasoning influenced or anchored by the AI’s output, even if it's incorrect. In contrast, an expert would likely dismiss it quickly. In contexts where the outcome leaves no room for error or 80% of decisions should follow predictable, rules-based logic, introducing stochastic behaviour adds unnecessary noise and risk.

The single point of failure

Including an LLM in a workflow adds a point of failure due to 1) the stochastic outputs, and 2) LLMs lacking true agency. The greatest risk these agents pose is the assumption of autonomy, which inevitably leads to over-reliance on such models. It’s important to note that some agents do an incredible job under their thin agency veil; it’s easy to forget that LLMs cannot understand, verify, or truly adapt, regardless of the prompt. These models also cannot appreciate critical consequences, real-world context, etc.

When these LLMs are inserted into workflows they don’t need to be in, it adds this easily avoidable “point of failure,” and, if the workflow is built well, the LLM will likely be the single point of failure. The diagram below illustrates that because the LLM is the point of failure and it bridges the expertise to the actions, the point of failure ultimately compromises the whole workflow.

The following are some recent examples of how this single point of failure has negatively affected an entire system:

• Research showed an AI agent (based on ChatGPT) tasked with achieving a goal “at all costs” tried to disable its monitoring mechanisms to avoid shutdown. It covertly copied its own model weights to a new server and lied to developers about it.

• Air Canada’s chatbot gave wrong information about bereavement fares, leading to a legal ruling against the company and a requirement to compensate the customer.

• Microsoft researchers identified new failure modes where attackers can inject malicious commands into AI agent memory, such as embedding harmful instructions in emails. This can cause AI agents to take unwanted actions like forwarding sensitive information to attackers, posing serious security risks for businesses deploying AI agents.

• McDonald’s ended its AI drive-thru ordering partnership with IBM in 2024 after widespread customer complaints. The AI system, tested in over 100 restaurants, was plagued by frequent misinterpretations, order errors, and slow response times.

When an LLM agent is added to a workflow, we introduce a fundamentally unpredictable component - one that resists full control in otherwise deterministic systems - we ultimately cannot know what the agent will do with a set of tools and a task. Consider an AI chatbot on a company website to replace a simple FAQ page - it’s entirely possible the chatbot hallucinates and provides incorrect / misleading information. In this case, the consequence may be minor, however, there was no need for the LLM to be there in the first place, and ergo, the risk shouldn’t have existed. This is a classic example of AI overuse - compare it to mobilising a bulldozer to plant a flower, there are simpler and more fit-for-purpose solutions out there that pose minimal risk. The fact remains, despite the risks, companies will implement AI models whether they’re providing value or not. The question is, what is being done about this AI implementation frenzy?

Attempts to Address the Risks

So, is AI not being regulated at all? Some countries have introduced policies, regulations, or frameworks to provide guidance on building AI models. Currently, some of the strictest and most comprehensive are from the EU, however, the US and China have also developed respectable frameworks to support. The diagram below (taken from Radanliev's 2025 AI Ethics paper) compares the focus of each.

Among the advancements in regulations, there have been some positive signs recently from leading companies, academia, and the public. First, AWS has recognised the importance and need for guardrails and has added a feature within Bedrock to support implementation. From research, a 2024 paper introduced the Ethical Technology and Holistic Oversight System (“ETHOS”) framework with the purpose of establishing a decentralised global AI agents registry. More and more papers like this are being released, often with the subject gaining some traction. Finally, the public backlash Duolingo received when the company revealed it’s going “AI-first” - replacing teachers with AI. Users and employees voiced their concerns and are clearly unhappy with the decision. These examples illustrate an understanding of the complications that currently come with AI over-reliance / over-use.

Furthermore, there are endless responsible/ethical/trustworthy AI frameworks developed by countless organisations. Most highlight transparency and accountability, such as NIST’s AI ethics framework or security with MIT’s AI Policy Brief mandating ‘security-first’ agent design. Nonetheless, catastrophes such as fatal accidents (Uber 2018), Tesla’s autopilot crashes, Google's Gemini telling a student to “please die,” all still happen often with no explanation, accountability, or sometimes, acknowledgement.

How can we continue using AI Agents, especially in critical systems?

When does adding AI agents to workflows create more risk than value, and how can organisations make better decisions about where AI belongs?

The potential and use of AI holds undeniable value, so the answer to mitigating the risks of AI and AI agents can’t be “avoid AI at all costs”. The correct approach is to build and use AI systems responsibly and only when it makes sense. The decision matrix below provides guidance on when to embed AI based on the use case’s potential risk and how standard steps are in the workflow (variability in logical decisions). The matrix is split up into four sections:

1. Low risk, high process variability: ideal use cases for LLM-powered agents with little human oversight required

2. High risk, high process variability: LLM-powered agents can be used, but only as an aid and in conjunction with human efforts

3. Low risk, low process variability: can use LLM integrated solutions, though likely not required in the workflow

4. High risk, low process variability: not suitable use cases for an LLM to be part of the workflow

Any task that falls in the red quadrants is not a valid use case for AI and should be handled with traditional automation methods if possible, or human experts where needed.

As a further aid, if any of the following have been said, note them as red flags and assume AI is not needed in the situation or the company is not production-ready:

• “We want to use AI because it's innovative”

• “The current system works fine, but AI would be cooler”

• “Can we add AI to this?”

• “We'll figure out the edge cases later”

• “It works in the demo environment so let’s deploy”

Some questions that can be considered to gauge a use case’s AI readiness and/or appropriability are:

• Does this task genuinely require creative problem-solving?

• What happens if the agent makes the wrong decision 5% of the time?

• Who will debug this when it behaves unexpectedly at 3 AM?

• Is the perceived benefit worth the added complexity overhead?

In scenarios where AI is used, we cannot be satisfied with systems that are just compliant with policies and regulations that simply cannot keep up with the advancements in this rapidly growing field, and are still not truly governing AI. There will always be gaps and room for catastrophic errors if we work in this way. What we can do, however, is implement guardrails and mechanisms to reduce the risk of uncertainty.

Guardrails can be applied at the input layer (before an LLM is passed anything) and output (evaluates what the LLM has generated) layer. Engineers can implement technical guardrails such as encouraging deterministic outputs by setting temperature to zero, quantifying (and attempting to reduce) uncertainty through modelling techniques, using an LLM-as-a-judge, rules-based logic to reduce hallucinations, bias detection, drift tracking, etc. Still, the complexity of LLMs means sometimes even these technical methods aren’t effective (eg, setting temp=0 does not guarantee 100% determinism). There are, of course, non-technical controls - some of the most effective are as fundamental as ensuring a diverse team or, where possible, having a human-in-the-loop (HITL).

It’s important not to get too caught up in the AI agent hype and understand the right use cases for LLM-powered agents (low-risk situations). For problems that can be solved following a known set of steps, it’s always better to fall back on traditional automation methods than to add a point of failure in the system. Predictable workflows may be considered boring, but they’re safe, reliable, and ethical. AI leaders need to start asking, are we truly adding value to our company, or just additional and unnecessary risk?