Tag: AI Governance

Human in the Loop – Leadership When Intelligence Is Everywhere

7 min.

Summary

What does “human in the loop” mean when an AI system weighs more information and more lines of reasoning than a person can follow? I put this question to futurist Kevin Benedict on October 29. His honest answer: it is one of the big unsolved questions. In his new book, however, he shows where responsibility moves. Not away from people, but upstream, ahead of the actual decision.

How Did This Post Come About?

Kevin Benedict is a futurist at Tata Consultancy Services and has been writing about technology, the future, and leadership for many years. His book “The Future of Human Leadership: When Intelligence Is Everywhere” came out in September. On October 29, I attended his masterclass “Human Leadership in a Machine-Intelligent World”, which was about exactly this book.

During the Q&A, I asked him a question that has been on my mind: “What do we do about all this regulated stuff where we say human in the loop is a must? I sometimes find it complicated to decide whether the AI result is actually better or worse than what we had before. Especially with reasoning models, AI can think much broader and consider multiple lines of reasoning that a human may no longer even be able to follow. Wouldn’t we limit ourselves if we relied only on human judgment? What’s your standpoint on this?”

The Paradox of Control

Benedict called this a major challenge with no easy answer. I am paraphrasing his reply here, not quoting it. AI can take far more information into account than a single person, who relies mainly on training, experience, practice, and tradition. Yet it is this person who is supposed to review and decide. In regulated settings, the human remains necessary because responsibility and accountability cannot be delegated to a machine.

His example: someone takes over a role. That person had no part in developing the algorithm or the decision tree. They are still accountable for its decisions. Benedict did not offer a ready-made solution. He sees this as one of the big open leadership and governance questions of the AI era.

He described the problem more broadly in a conversation with Ronald Worman. As Worman puts it in his blog post about it, the speed at which consequences unfold is starting to outrun the speed at which we understand them.

Regulators know the problem. For high-risk systems, Article 14 of the EU AI Act requires human oversight, applicable from December 2027 or August 2028, depending on the type of system. The people in charge must be enabled to understand the system’s capabilities and limitations. They must stay aware of the tendency to rely too heavily on its output, known as automation bias. And they must be able to override results and stop the system.

That is a demanding standard. The better a model reasons, the harder it becomes to check a single result. Someone who only signs off is formally in the loop. In practice, they are not.

The Decision Moves Upstream

In his book, Benedict goes a step further than in the Q&A. On LinkedIn, he quotes this line from it: “Autonomy does not eliminate the human decision. It relocates it.”

Where it moves to, he explains in a short synopsis of the book on his blog. Human oversight, he argues, cannot be meaningful if people are brought in only after the important choices have been made. As autonomy grows, leadership often moves “above the loop.” From there, it settles the following before a system starts operating (his list, condensed into questions):

  1. Purpose. What problem should the system solve, and whom does it serve?
  2. Permissions and boundaries. What may the system decide on its own, and what not?
  3. Escalation and stopping. When must a human take over, and who may halt the system?
  4. Verification. How do we check whether the results are right?
  5. Repair and appeal. How are errors corrected, and how can those affected object?

He states there: “Operational autonomy can be delegated; moral responsibility cannot simply disappear.”

In the masterclass, paraphrasing from my notes, he used an image from aviation for this shift. The Wright brothers still steered their aircraft directly with their own bodies and cables. In a modern fighter jet like the F-35, software processes so much sensor data that the pilot no longer controls every movement. The pilot monitors the system and leads the mission. Benedict already sketched this idea in a 2021 article on complexity and thinking.

Accountability Needs Information, Time, and Authority

One sentence from Benedict’s backstory to the book speaks directly to my question. A person who is accountable needs “the information, time, and power to question a recommendation, stop a harmful process, or change the direction.”

His example of the newly filled role shows where this breaks down. Anyone who does not know the decision logic carries accountability without information.

Then there is time. Benedict distinguishes four clocks: digital time, at which systems operate; human time, needed to learn, build trust, and recover; historically imagined time, the version of the past people bring into every change; and future time, the expectations and promises that shape what people are willing to do today. He illustrates this: a system can be installed over a weekend, while employees need months to trust it.

And there is energy. Benedict speaks of Transformational Energy Units, meaning the limited capacity people have for adapting. Several changes at once draw on the same reserve. The “units” are not meant as a precise measurement, he writes, but as a way to make the burden discussable. My conclusion: exhausted people review less carefully. That, too, belongs in an honest answer to human in the loop.

What Does This Mean for Projects?

In my GPM Focus Workshop on leadership in the age of AI, held in Frankfurt in March 2026 (GPM is the German Association for Project Management), a participant put it this way: “Accountable is always the human. Responsible can be human or AI – depending on the task and its complexity.” I quoted it in my post Project Manager in 2036. Benedict’s book adds what this requires. Four consequences follow for projects.

  1. Decision rights belong in the project charter. For every AI-supported process, it should be clear who recommends, who decides, who may override, and who stops it. This is an extended responsibility matrix, not an IT question.
  2. Handovers include the decision logic. When a role changes hands, handing over tasks is not enough. The new person needs the system’s purpose, assumptions, thresholds, and known limits.
  3. The depth of review depends on what is at stake. In the masterclass, Benedict recommended, in essence, differentiating response times by risk, reversibility, and impact. Small, reversible decisions can move fast. Far-reaching ones need time and several perspectives.
  4. Change is planned by load. Not only per project, but cumulatively per role and period. Otherwise the fifth rollout of the quarter hits people who no longer have the energy to review anything.

What Does This Mean for the Coming Years?

From here on, this is my assessment, not an established development.

Human in the loop will probably split into two tasks. Reviewing individual results remains where a person can still judge them. Alongside it, the work above the loop grows: defining purpose, boundaries, stop criteria, and repair paths before the system runs. This second task is closer to project management than to technology.

For qualification, this means it would have to assess exactly this kind of design work. That brings us back to the question from my post AI and the Future of PM Certification. Judgment shows more clearly in a conversation than in a document.

My original question remains open, and Benedict did not claim otherwise. When a model follows lines of reasoning that no one can retrace, a signature under the result does not help. What helps is deciding in advance what the system is for and when it must stop.

Human in the loop is not a seat in the process. It is a responsibility that begins before the first result.

Who in our projects is allowed to stop an AI system – and how does that person know when it is time?

Feel free to subscribe to my free newsletter.

Sources