Designing escalation paths for AI agents.

Escalation paths for AI agents are the designed routes by which an agent stops, hands a case to a person with everything that person needs, and picks the work back up once they decide. Get them right and the agent can be trusted with real work in production; leave them until later and every hard case is handled however the model decides on its own.

Every agent project reaches the same meeting. Someone asks what happens when the agent is not sure, and the answer is a shrug and a promise to deal with it later. Later arrives in production, in front of a customer.

What is an escalation path for an AI agent?

An escalation path is the complete route a case takes when an AI agent should not finish it alone. A working path has five parts, and a gap in any one of them shows up quickly in production.

• Trigger. The condition that stops the agent, such as a missing field, an exceeded limit or a customer asking for a person.
• Handoff. The evidence the agent passes on: what it found, what it did and what it suggests.
• Route. Who receives the case, and what happens when they are unavailable.
• Waiting behaviour. What the agent does, and tells the customer, until someone decides.
• Decision record. How the decision returns to the system, so the case resumes and the outcome can be measured.

Most teams build the first part and improvise the rest. That is where human in the loop designs quietly fail: the agent knows when to stop, then drops the case into a shared inbox with a one-line note.

Autonomous or just unsupervised

An agent with no way to stop and ask is not autonomous. It is unsupervised, which is a different word with a different risk profile. Autonomy implies judgement, and judgement includes knowing what you are not qualified to decide.

The failures that end agent projects are usually confident mistakes made at scale, repeated until someone notices the consequences. One refund approved in error is a support ticket. Four hundred is an incident.

Assume the agent will sometimes be wrong, and design for who finds out and how quickly.

That is what AI agent oversight means in practice. Nobody reviews everything; the system makes sure the right cases reach the right person with enough context to decide well.

How much autonomy should an AI agent have?

An AI agent should have as much autonomy as its track record in a specific category of work supports, and no more. Autonomy is set per type of action rather than per agent: the same agent might route tickets alone while every refund it proposes waits for approval.

Two questions decide where a category starts: how reversible is the action, and what does a mistake cost? A mis-routed ticket is fixed with a click. A payment sent to the wrong account can be hard or impossible to recover.

Most new categories should start at suggest only or act after approval, even when the model looks capable in testing. Test sets are built from the cases people thought of in advance, and production brings the ones they did not.

Act then report only works if someone actually reads the reports while the action can still be undone. If nobody owns that review, the category is really running at act alone, whatever the design document says.

When should an AI agent hand off to a human?

An AI agent should hand off to a human when it lacks the facts to decide, when the decision falls outside its limits, or when the case is one a person must handle however capable the agent is. Most of the triggers below can be written as ordinary rules.

Triggers that are facts about the data

Missing input is the most common trigger and the easiest to detect: a required field was not found, a document was unreadable, a connected system returned nothing. It is a fact about the data rather than a judgement about the answer, so it can be checked with certainty.

Disagreement between sources means two things that should match do not, such as an invoice total and its purchase order, or the address on file and the address supplied. Never resolve a contradiction silently. The validation checks in our guide to extraction that survives a bad scan work well as escalation triggers, because each failed check is a specific, explainable reason to stop.

Triggers that come from policy

A policy threshold breach means the action exceeds a limit somebody set: a refund value, a discount, a customer tier. That is policy rather than intelligence, and it belongs in code where it can be audited.

A customer asking for a person should be honoured every time, even if the honest answer is that someone will reply later. Detect it generously, because a customer who repeats a question or says the answer did not help is often asking too. An agent that argues its way out of a human handoff damages trust quickly.

Sensitive categories go to a person by policy, however well the agent might cope: formal complaints, legal threats, medical topics, and signs a customer may be vulnerable, such as bereavement or financial hardship. Set the detection bar deliberately low. A false alarm costs a few minutes of review; a missed one can cost far more.

Triggers from the agent's own behaviour

Repeated failures or loops are the trigger teams forget. An agent retrying the same step, or passing a task back and forth with another agent, is making no progress. A hard cap on steps and retries per case, enforced in code, turns a runaway into an escalation.

Novelty is a case unlike anything the system was tested against: a new document layout, a request nobody anticipated, a product that did not exist at launch. It is harder to detect well, and it is where a signal from the model, combined with a similarity check, genuinely helps.

Notice how few of these triggers depend on the model's opinion. The escalation logic should be the most boring, most inspectable part of the system, because it is the part you will be asked to explain.

Rules, judgement and confidence scores

The most common first design is a single confidence threshold: if the model is less sure than some cut-off, send the case to a person. It is simple to build, and it escalates the wrong things.

Why is model confidence a poor escalation trigger?

Model confidence is a poor sole trigger because the number is not a reliable measure of whether a decision is right. A confidence rating the model gives itself is generated text like the rest of its answer. Token probabilities, where a provider exposes them, describe how likely the model found each word it produced, which says more about wording than about correctness.

The dangerous cases are the confident wrong ones, and a threshold waves those straight through. Calibration can also shift when the model, the prompt or the inputs change, so a cut-off tuned in testing can quietly stop meaning what it did. Use confidence as one signal alongside these:

• Validation checks. Totals that must add up, fields that must agree, formats that must match. These catch confident errors directly.
• Agreement checks. Run a classification or extraction twice, or by a second method, and escalate when the results differ. Agreement does not prove correctness and every run adds cost, so reserve this for expensive decisions.
• Retrieval signals. If the agent answers from your documents, escalate when nothing relevant was found.
• Similarity to known cases. Compare a new case with tested examples, for instance using embeddings (numerical representations of meaning), to flag novelty.
• Category history. A category with a high overturn rate should escalate more, whatever the model says about one case.

Rules in code, judgement in the model

Anything that is policy lives in code; anything that needs reading or interpretation can use the model. Thresholds, permissions, sensitive category lists, retry limits and out-of-hours behaviour are policy. They belong in configuration a person can read, change through review and test deterministically, as described in testing non-deterministic systems.

The model earns its place on interpretation: recognising that a polite message is a formal complaint, summarising a long thread for the reviewer, proposing a likely cause for a mismatch.

One rule matters more than the rest. The model may raise an escalation, but it must never be able to cancel one that a rule raised. Enforcing limits outside the model also means a customer who writes instructions into a message cannot talk the agent past them.

What should an escalation handoff include?

An escalation handoff should include the reason for stopping, the evidence, what the agent already did, what it suggests and what is on hold. A note saying the agent could not process this forces the reviewer to reconstruct everything the agent already knew, which is slower than if the agent had never touched the case.

The handoff payload

What arrives with the case matters more than the fact of the escalation. A payload for a mismatched invoice might look like this.

escalation:
  case:        INV-4471
  reason:      totals_disagree
  rule:        invoice_total_matches_po
  found:       invoice 4,850.00 / po 4,580.00
  attached:    [invoice, extracted_fields, purchase_order]
  agent_did:   extracted fields, matched supplier, fetched po
  suggestion:  digits transposed, likely on the invoice
  on_hold:     payment not scheduled
  route:       finance-review

That is the difference between an interruption and a piece of work. The reviewer sees the conflict, the two numbers and a hypothesis to accept or reject in seconds. The agent did the reading; the person does the deciding.

The on_hold field tells the reviewer what is waiting on their decision, so they can judge urgency without guessing. For customer conversations, add a short summary and the customer's own words on the issue, so nobody has to ask the customer to start again.

The agent stops rather than acting on its best guess and flagging it afterwards. A suggestion presented as a decision already made tends to get approved without a real look.

What the reviewer's screen should show

A reviewer's screen for escalated cases works best when it shows:

• The reason for escalation in plain words, rather than a rule code.
• The conflicting evidence side by side, with the relevant part of the source highlighted.
• The agent's suggestion, clearly labelled as a suggestion, with a sentence of reasoning.
• What the agent has already done and what is on hold.
• The options: approve, amend, reject or reassign.
• A short list of reason codes, plus an optional note.

Do not preselect the agent's suggestion as the default. One-click approval of a pre-filled decision is how review turns into rubber-stamping.

Who should receive an escalated case?

An escalated case should go to the person or team who owns that kind of decision, chosen by the escalation reason rather than dropped into a shared inbox. Routing by reason means the case lands with someone who can act on it without forwarding it.

Every queue needs a named owner and a backup. The table is an illustration rather than a template, because the map should follow how your organisation already assigns these decisions. This is often where an agent project becomes a workflow automation project too, since queues, assignments and notifications are ordinary workflow.

Out of hours and unclaimed cases

Decide out-of-hours behaviour for each category before launch. The sensible options are to hold the case until the next working day, send it to an on-call person, or have the agent take a defined safe default. The choice depends on what waiting costs: a supplier invoice can usually wait, while a customer locked out of their account may not be able to.

Agree response windows with the team that owns each queue, and only promise customers what that team can reliably meet. A case left unclaimed beyond the agreed window should move up to a supervisor queue, and should never fall back to the agent's suggestion.

What should the agent do while it waits?

While it waits, the agent should pause the case in a safe state, take no action that is hard to undo, and tell the customer honestly what is happening. An undefined wait is where agents do their most surprising things.

• Save the state. Record what the agent read, what it did and where it stopped, so it resumes exactly there while unblocked cases carry on.
• Take the safe default. Choose the action that does no harm whichever way the decision goes: the payment stays unscheduled, the order stays on hold. Avoid partial actions, such as updating one of two linked records, which leave the reviewer a mess to untangle.
• Tell the customer the truth. Say a person is reviewing the case, give a time only if it is reliable, and never imply a decision has been made when it has not.
• Stop working the problem. Retrying other routes to the same outcome in the background creates conflicting actions and adds model cost for no benefit.

Recording the decision and learning from it

The decision has to come back into the system, recorded against the case, rather than being lost in an inbox. Record who decided, when, what they decided, whether they accepted, amended or rejected the suggestion, and a reason code. The agent then resumes with that decision as an input.

Use a short fixed list of reason codes, such as wrong supplier matched or policy exception granted, because codes can be counted and trended. Offer a free-text note as well as the code, never instead of it.

Recorded decisions are among the most valuable data an agent project produces: real cases where the system needed help, with the right answer attached. They feed three things.

• Test cases. Overturned and amended cases join the evaluation set, so the same mistake is caught if it returns after a prompt or model change.
• Rule changes. A cluster of escalations with one reason code often points to a misplaced threshold or a changed input source.
• Examples. Where the design uses examples in prompts or a retrieval store, reviewed decisions are a good source once someone has checked them.

Which metrics show an escalation path is working?

The metrics that show an escalation path is working are the escalation rate, the overturn rate, time to decision and repeat escalations, read together and split by category.

A rate near zero is rarely good news. It usually means the triggers are too loose and the agent is deciding things it should not. A steadily climbing rate is worth investigating before it is worth fixing: it often means the inputs changed, and the agent noticing is the system working as designed.

The overturn rate asks how many reviewed cases people decided differently from what the agent proposed. High overturns mean the agent's judgement is not yet trustworthy in that category. A very low overturn rate only counts if reviewers are genuinely reviewing, so consistently near-instant approvals on complex cases deserve a closer look.

Escalation metrics only see escalated cases. To find what the triggers miss, have someone who knows the work check a random sample of cases the agent completed alone.

When can an AI agent be given more autonomy?

An AI agent can be given more autonomy in a category when that category's evidence supports it: a sustained low overturn rate across enough reviewed cases, a clean audit sample, and stable inputs over the same period. What counts as enough depends on volume and the cost of a mistake, and it should be agreed with the people who own the risk before the numbers come in.

Widen one category and one step at a time. Small refunds might move to act then report while larger ones stay at act after approval. Keep the ability to move a category back down, and treat a change of model, prompt or major input source as a reason to re-check the evidence. Designed this way, escalation paths stop being a safety net and become the mechanism by which the system earns more responsibility, which is the only sound way to give it any.

Where escalation paths go wrong

Escalation paths tend to fail in a handful of predictable ways. Most of them stay invisible unless someone goes looking.

Alert fatigue

When too many cases escalate, reviewers stop reading them properly. The queue becomes background noise and important cases wait behind trivial ones. The fix is rarely more reviewers: look at escalations by reason, fix the inputs or thresholds behind the biggest groups, and give reviewers a clear order of priority.

Rubber-stamping

Reviewers who agree with the agent almost every time, and quickly, may be checking nothing. The overturn rate then looks excellent and means little. Keep suggestions from being preselected, have a second person independently check a sample of approved decisions, and make it normal to disagree with the agent.

Escalation loops

A loop happens when a decision comes back, the agent resumes and the same trigger fires again. The usual cause is a decision recorded where the trigger cannot see it, so an approved exception looks like a fresh breach. Cap how many times a case can escalate, send anything over the cap to a named person, and track repeat escalations. Every round also costs model calls, which is why loop limits belong in any plan for keeping model costs down.

Silent fallbacks

The most dangerous failure looks like success. The review queue is unavailable, a routing rule has no match, or a timeout expires, and the code quietly lets the agent proceed. Nothing alerts, and the escalation rate even improves. Every failure inside the escalation path should fail closed (hold the case and raise an alert rather than carry on), and that behaviour should be tested like any other feature.

How to design an escalation path

A workable design process runs in this order.

1. List the agent's actions in business terms, such as issue a refund or pay an invoice, rather than by tool name.
2. Rate each action for reversibility and cost of error to set its starting level of autonomy.
3. Write the triggers in code: missing input, disagreement, thresholds, sensitive categories, customer requests and loop limits, then novelty detection where it is worth the effort.
4. Design the payload and the reviewer's screen together, and test them with real reviewers on real examples.
5. Map routing and ownership, with a backup for each queue and out-of-hours behaviour for each category.
6. Define waiting behaviour: the safe default, what the customer is told and what happens if nobody decides.
7. Build the decision record with reason codes, and check the agent resumes cleanly from every possible decision.
8. Set up metrics and an audit sample before launch, and agree what evidence would justify more autonomy.
9. Test the failure paths in a test environment by taking the queue offline and forcing a loop, and confirm every case is held and someone is alerted.

Steps four to six are the most tempting to cut, because the demo works without them. They decide whether reviewers find the agent useful or quietly work around it. They are also part of the distance between a pilot and a production system, the subject of why AI pilots stall before production.

Key takeaways

• An escalation path is a full route: trigger, handoff, routing, waiting behaviour and a recorded decision.
• Set autonomy per category of action, based on how reversible it is and what a mistake costs.
• Most good triggers are rules in code, covering missing input, disagreement, thresholds, sensitive topics and loop limits.
• Treat model confidence as one signal among several, never the sole reason to escalate or to proceed.
• A useful handoff carries the reason, the evidence, what the agent did and what is on hold.
• Every failure inside the escalation path should hold the case and raise an alert.
• Widen autonomy one category at a time, when overturn rates and audit samples support it.

Where to start with escalation design

If you already have an agent running, start with an audit. Have someone who knows the work check a sample of cases it completed without escalating. What they find shows which triggers are missing.

If you are planning an agent, write the escalation design before the prompts: list the actions, rate them, write the triggers and sketch the reviewer's screen. Building autonomy up from a careful start is far easier than clawing it back after an incident.

Escalation design is a core part of how we build custom AI agents. If you want to talk through where your agent should stop and ask, schedule a call.

Frequently asked questions

What does human in the loop mean for AI agents?

Human in the loop means a person reviews or approves certain decisions before an AI agent acts on them. In a well-designed system the person does not check everything. Rules decide which cases need review, the agent prepares each one with its evidence and a suggestion, and the decision is recorded so the agent can resume and the system can be measured. Which cases need a person usually changes as the agent builds a track record.

What is the difference between human in the loop and human on the loop?

Human in the loop means a person must approve a decision before the agent acts. Human on the loop means the agent acts on its own while a person monitors its work and can step in or reverse an action. Most production agents use both, category by category: approval for costly or irreversible actions, and monitoring with sampled reviews for routine, reversible ones.

How do you hand a chatbot conversation to a human without losing context?

Pass a structured handoff along with the conversation. It should include a short summary, the customer's own words on the issue, what the assistant already checked or changed, the reason for the handoff and any account details the person needs. The person picking it up should never need to ask the customer to repeat themselves, and the customer should be told honestly that a person is taking over.

What percentage of cases should an AI agent escalate?

There is no correct universal figure. The right escalation rate depends on the category of work, the quality of the inputs, the cost of a mistake and how long the agent has been running, and new categories should escalate more. What matters is the trend and what reviewers do with escalated cases. A falling overturn rate and clean audits of unescalated work suggest the rate can safely come down.

Can an AI agent decide for itself when to escalate?

Partly. A model is good at spotting things rules miss, such as a message that is really a formal complaint or a request unlike anything seen before, so it should be able to raise an escalation. It should not be able to cancel one. Policy triggers like value limits, sensitive categories and retry caps belong in code outside the model, so neither the model nor a cleverly worded message can get past them.

Is human oversight of AI agents required by law?

Sometimes, depending on where you operate, your sector and the decision being made. Data protection law in the UK and EU, for example, gives people rights around significant decisions made solely by automated means, which can include a right to have a person review the decision. The EU AI Act also sets human oversight requirements for systems it classes as high risk. Take legal advice for your specific use case.

Do escalation paths make AI agents slower or more expensive?

Escalated cases take longer because they wait for a person, and reviewer time has a cost, while cases that do not escalate are unaffected. Against that, a well-designed path prevents the expensive outcomes: errors repeated at scale, runaway loops that burn model calls, and incidents that halt a project. The aim is to escalate only what needs a person and to make each review quick.

How do you stop an AI agent from escalating too many cases?

Start by splitting escalations by reason and fixing the largest groups at their source. Poor input, such as unreadable scans or a changed data feed, is a frequent cause and has nothing to do with the agent's judgement. Next, check whether thresholds are set tighter than the policy actually requires. Only then give the agent more autonomy, and only in categories where reviewers rarely overturn its suggestions.

Schedule a call