Keeping AI Agents Under Control: Principles for Decision-Makers

TL;DR. AI agents produce more than any person can read, yet responsibility stays with people. They become controllable not through trust but through a procedure: assess the risk first, set autonomy accordingly, let agents work in a fenced and observable space, catch errors through independent checks, use proven knowledge, and turn every error and every anomaly into a rule. Plus an example from our own development work.

When did you last sign off on something you had not read in full? Probably today. Every manager does it daily: the tax adviser's statement, the engineering firm's expert report, the supplier's software. We don't trust blindly when we do this; we trust the procedure behind the result and the people who stand behind it.

AI agents confront us with the same question, only on a different scale. Agents write, calculate, plan and code at a pace where nobody can read every line any more. When several agents work together, they pass intermediate results along, and a small mistake at the start can show up at the end in many copies. The worry about losing control is therefore justified. The answer, however, is not to give up on AI, because the competition won't, but to organise control differently.

In our experience a handful of principles carry the weight. None of them is new; most will be familiar to anyone who has dealt with suppliers, auditors or safety-critical products. What is new is that we now apply them to machines that act on their own.

Our agent board in the playground: real jobs in fixed stages, here a certificate-management project and a blog job. On the right, the flow of a running project, from the client through requirements, estimate and concept to implementation; the red arrow at S3 marks a loop-back.

Responsibility stays with people, control moves into the process

An AI cannot be held liable; a company can. No technology changes that, and the EU legislator is currently drawing this line more sharply rather than less. Anyone who carries responsibility without being able to check every detail needs exactly what a car maker needs from its suppliers: clear agreements on what is delivered, traceable checks, and evidence to rely on. People no longer control every result; they control the procedure that produces and checks results, and they decide at the points where it matters.

Assess the risk first, then set the length of the leash

Not every task deserves the same level of control. A spelling fix needs a quick glance, a binding customer quote needs a careful look at every line, and a safety-relevant function needs the full apparatus of testing, evidence and independent confirmation. It pays to ask two questions separately: how risky is the AI system itself, and how risky is what it produces? A harmless tool can create a dangerous result. The answer determines how much freedom an agent gets: whether it simply carries on, carries on and reports, or stops until a human has agreed. Skip this classification and you end up either checking everything, which nobody can sustain, or checking nothing, which will come back to bite you.

Intent first, execution second

Many agent errors are really errors in the brief. Whatever is not clearly described, the agent fills in with its own assumptions, and with great conviction. That is why the work starts with a brief that says what is to be achieved, how you will recognise that it has been achieved, and what is explicitly out of scope. This intent becomes the yardstick for every later check. Without it there is nothing to check against except gut feeling.

Freedom within a fence

Agents should be allowed to work on their own; otherwise you waste what they are good at. But they work in a fenced-off space where a mistake cannot do real damage: no access to production systems, no credentials, and a result that is always a proposal and never a done deal. What matters is where the boundaries sit. An instruction in the brief is a request, a technical block is a boundary, and whatever is non-negotiable belongs in the second category.

The fence at work: the workspace was not allowed to reach the platform hosting the model the agent needed. The agent did not work around the block; it put the question to the client, with the situation, the consequences and how critical it is. (Screenshot in German, as the project ran in German.)

See what is happening

You can only steer what you can see. Every step an agent takes, every decision, every minute and every euro should be visible, ideally live on an overview that a non-technical person can understand. This visibility is not only control; it is also the basis for all learning, because without a record there is nothing to learn from later. And you need an emergency stop that actually works.

Every tool call of an agent shows up live on the timeline; pause, stop and abort are always one click away.

Check by detection, not by trust

Trusting an agent is not a strategy. It becomes controllable because the checks behind it catch its errors with high probability. A three-part setup has proven itself: fixed, incorruptible rules and tests that always judge the same way; a second instance, which may well be another agent, that checks the result against the original brief and has to back up every statement; and a human who decides at a few clearly defined points. The second instance may raise doubts and escalate, but it may not decide.

Every stage is checked before the next one starts: green means passed, red means failed. Here the implementation (S5) failed its checks three times, and nothing moves on until someone decides.

That such checks are not a brake is suggested by data from Databricks: according to its "State of AI Agents" report (January 2026), based on usage data from more than 20,000 customer organisations, companies using AI governance tools get over twelve times more AI projects into production, and companies using evaluation tools nearly six times more [2]. That is a correlation from vendor data, not proof of cause and effect, but it points in the same direction as the MIT finding.

Don't reinvent proven knowledge

Industry standards, testing regulations and rulebooks are the distilled experience of decades of accidents, recalls and audits. They apply even when a result comes from an AI, and they often already answer the question of how deeply something has to be checked. The same holds on a small scale for in-house knowledge: project rules, checklists and what long-serving colleagues know belong not only in their heads but in the chain of checks, where every agent is shown them before it starts work.

Learn from errors and anomalies

Most people underestimate this principle, and it may be the most important one. A system of people and agents does not become controllable by never making mistakes, but by never making the same mistake twice.

How decisive this is shows up in a widely cited MIT study (Project NANDA, 2025). Despite 30 to 40 billion dollars of investment, 95 per cent of organisations in this analysis get no measurable return from generative AI, and only 5 per cent of custom solutions make it into production. The authors see the main cause neither in the models nor in regulation, but in learning: most systems do not retain feedback, adapt to context or improve over time. One interviewee from a law firm put it plainly: the tool repeats the same mistakes and needs the full context again in every session [1]. The figures are based on interviews and surveys, and the authors themselves call them directional rather than exact. But the direction is clear, and it applies to the interplay of people and agents just as much as to the tool on its own.

That starts with looking, and not only at what goes wrong but also at what stands out. A run that takes twice as long as usual. An agent that solves a task differently from what was asked. A check that has not fired for weeks, which is more suspicious than reassuring. Anomalies like these are often the first signs of a problem, long before an error becomes visible.

Then what has been learned must be recorded in a way that has an effect. An anecdote in one employee's memory is of little use next time, especially if that employee is on holiday. A lesson in a fixed format, on the other hand, with the context, the problem, the cause and the rule that follows from it, can be found again, reviewed and automatically put back in front of people and agents on the next matching job. Once a lesson is important enough, it moves up: from a note to a rule, and from a rule to an automated check that catches the error on its own in future. Every rejection at a sign-off point is a gift here, because the human's comment on why something does not fit is the most valuable instruction an agent can get.

After three failed attempts the system stops. A human decides: approve, reject, or repeat with a hint. The comment becomes the input of the next attempt.

And finally, a human has to decide which lessons apply. Not every observation is a rule, and a knowledge store into which everything flows unchecked quickly turns into noise.

People have to stay in practice

The best control is useless if sign-off turns into rubber-stamping. The more reliable a system appears, the greater the temptation simply to believe it, and the harder it becomes to step in when it counts. Specialists have known this problem for decades from the automation of aircraft and power plants. Regular, real decisions at the sign-off points are therefore not bureaucratic overhead but training for the day it matters. That includes training the people who make these decisions, so that they know what an agent can do and where it typically goes wrong.

What this looks like at SINTARIS

The SINTARIS team follows exactly these principles in its own development work. Our agents work in a sealed-off workspace, our "playground", with no access to production systems. Every job runs through fixed stages, from analysis through implementation, testing and review to sign-off, and for each stage it is defined whether the agent carries on, reports, or waits for a human. Every step, every minute and every cent is visible on an overview, and the result is always a proposal that a human accepts or rejects.

A running certificate-management project: S0 to S4 passed, loop-backs (red) visible, sub-agents in the implementation. The implementation failed, and the project is waiting for the client ("waiting for you: S5").

Two experiences from the early days show why this pays off. In the first real pilot job, an agent was supposed to add a missing component. It found that the actual cause lay elsewhere, solved the problem a different way and documented why. Formally it had not done what it was asked; in substance it had probably solved the problem better. Whether that is what we wanted is not something a machine can decide, and that is exactly why sign-off lay with a human. The second experience concerned ourselves: the complete recording uncovered two bugs in our own control system that would otherwise probably have gone unnoticed for days. Even the observer needs an observer.

Learning is a fixed part of the workflow for us. Even while setting up the workspace we ran into hurdles that no manual had foreseen, and each one was recorded as a lesson in the fixed format. Every stage of a job ends with a section on what can be learned from it, and every job with a short retrospective comparing the estimate with the actual effort. A human reviews these proposals and accepts or discards them. Our operations log now contains well over a hundred such lessons, and step by step they flow into the rules and checks that our agents are shown before every job.

None of these principles requires expensive specialist software or a large research budget. What they require is the decision about who signs off on what at which point, and the discipline to turn every error into a rule. So, to close, a question you can answer without any AI at all: when did your company last turn a mistake into a rule that still applies today? If the answer takes a while, that is the best place to start, with agents or without.

Sources

1. MIT Media Lab, Project NANDA (Challapally, A., Pease, C., Raskar, R., Chari, P.): The GenAI Divide: State of AI in Business 2025, July 2025: https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf
2. Databricks: 2026 State of AI Agents: https://www.databricks.com/resources/ebook/state-of-ai-agents · summary: Enterprise AI agent trends: Top use cases, governance + evaluations and more, 27 Jan 2026: https://www.databricks.com/blog/enterprise-ai-agent-trends-top-use-cases-governance-evaluations-and-more