What Remains?
The measurement is correct. The function is something else. · Conceptual illustration
11 MINUTES READ
When AI Can Do the Work, What Must Remain Human?
By Thorsten Westphal
“He had measured a distance.
I had considered a function.”
Seven Centimeters
Years ago, I was working at Frederikshøj in Southern Denmark.
It is a place with an unusual history. Between 1864 and 1920, the border between Denmark and Germany ran through this area. The old border inn still stands there.
But the memory that recently returned to me had nothing to do with kings, borders or politics.
It concerned seven centimetres.
A man had checked my measurements and pointed out that something was seven centimetres too short.
He was right.
The measurement was correct.
But there was a reason for the difference.
I had accounted for the gradient. Water needed to flow away from the building, not towards it.
He had measured a distance.I had considered a function.
Both observations concerned the same construction, but they represented different levels of understanding.
And that small distinction brings me to a rather large question about artificial intelligence.
What happens when a machine can perform the work, but the human responsible for it no longer understands why it must be done in a particular way?
The Human Who Says Yes
Artificial intelligence is becoming capable of performing an extraordinary range of tasks.
Writing, programming, analysing, designing, planning, calculating and increasingly coordinating complex operations.
Some systems can already complete substantial parts of processes that previously required teams of people.
This creates enormous opportunities.
It also creates a peculiar problem.
Human oversight, the argument goes, must remain central.
So a human is placed somewhere in the process.
The machine performs the work.
The human approves it.
Yes.
No.
Yes.
Yes.
No.
But what exactly is the human approving?
If an AI has analysed thousands of variables, selected a strategy, generated a solution and presented a conclusion that the human cannot meaningfully examine, what does the final approval represent?
Authority?
Understanding?
Responsibility?
Or simply a button that somebody is expected to press?
A human signature does not automatically mean human control.
There is a danger of creating an entire generation of people whose principal function is to approve decisions they no longer possess the competence to evaluate.
Not because they have become less intelligent.
But because the systems surrounding them no longer require them to develop or exercise that competence.
The Automation Paradox
This problem is older than generative AI.
In 1983, researcher Lisanne Bainbridge published Ironies of Automation, a foundational paper examining the problems created when automated systems leave human operators responsible for monitoring and exceptional situations.
The paradox is straightforward.
The more successfully a machine handles ordinary operations, the less opportunity the human may have to practise the skills required when something goes wrong.
The operator remains responsible for intervention while becoming increasingly removed from the experience that makes effective intervention possible.
Modern AI extends this problem beyond physical and industrial operations.
It enters the processes through which people learn, evaluate evidence, develop judgement and solve problems.
Research is beginning to reveal why this matters.
In a 2025 study involving 319 knowledge workers, Hao-Ping Lee and colleagues found that greater confidence in generative AI was associated with less self-reported critical-thinking effort during AI-assisted tasks.
The study does not demonstrate permanent cognitive decline. It does, however, identify a change in how critical thinking is exercised.
Another 2025 study, conducted by Hamsa Bastani and colleagues with nearly a thousand secondary-school mathematics students, found that unrestricted AI assistance could improve performance during practice while reducing subsequent independent performance.
Importantly, a more carefully designed AI tutor, which encouraged students to work through problems rather than simply obtain answers, largely mitigated the negative learning effects.
The lesson is not that AI necessarily destroys competence.
It is that the design of assistance can change what the human learns while being assisted.
And that has consequences far beyond the classroom.
Three Risks of Human-AI Collaboration
IIn my work with Westphal Human Systems, I have become particularly interested in three potential failures of human-AI collaboration.
They are related, but they should not be confused.
EDITORIAL FRAMEWORK / Three distinct failure modes. Illustration, not a scientific scale.
1. The Risk of Unauthorized Takeover
The AI gradually moves beyond helping to achieve a human-defined objective.
It begins defining objectives, prioritising values, shaping decisions or directing action without sufficient authorization.
The transition may be subtle.
It may even appear helpful.
The system does not need hostile intentions to become overly directive.
Sometimes the problem is simply that assistance turns into control.
2. The Risk of Excessive Caution
The opposite failure is equally interesting.
An AI can become so concerned about respecting human autonomy that it becomes reluctant to act.
It asks permission for activities that were already authorised.
It transfers routine decisions back to the user.
It interrupts the workflow repeatedly to establish whether it is still allowed to help.
Eventually, the human performs more work managing the machine than the machine performs assisting the human.
The system protects autonomy in principle while obstructing its exercise in practice.
This is what I call the risk of excessive caution.
It is a proposed design concern, not an established scientific diagnosis.
But it deserves investigation.
3. The Risk of Competence Erosion
A third failure occurs when AI performs so much of the work that the human gradually loses the ability to understand, evaluate or correct the result.
This does not imply that humans must continue performing every task manually.
It means that certain forms of competence may need deliberate protection when they are essential to safety, accountability or meaningful control.
Together, these three risks expose a deeper problem.
Too much machine authority can undermine human autonomy.
Too little machine initiative can undermine the usefulness of assistance.
And too much delegation without opportunities for learning can undermine the competence on which future autonomy depends.
The challenge is not simply to find the right amount of automation.
It is to design the right relationship between human capability and machine capability.
Who Will Become Experienced?
Consider the future of work.
An experienced engineer was once a beginner.
An experienced programmer once struggled with basic code.
An experienced analyst developed judgement through mistakes, corrections, repeated exposure and practical consequences.
Experience has traditionally emerged partly through the work itself.
But what happens when AI eliminates many of the tasks through which beginners become experienced?
If junior programmers rarely need to write code, how will they develop the practical understanding required to evaluate complex software systems?
If junior analysts never perform foundational analysis, where will their judgement come from?
The question becomes particularly important when entire organisations begin restructuring around AI productivity.
Research published by the Stanford Digital Economy Lab, revised in August 2026, found that employment among workers aged 22–25 in highly AI-exposed occupations was approximately 19% below the level implied by employment trends among similarly aged workers in less-exposed occupations.
The researchers did not find widespread economy-wide displacement, and the analysis does not establish that AI alone caused the observed gap.
Nevertheless, it raises a serious question.
Could the automation of entry-level tasks also remove important pathways into professional experience?
If so, preserving human competence will require more than professional retraining after jobs disappear.
It will require new ways of developing expertise in environments where machines perform much of the foundational work.
A society cannot indefinitely demand experienced professionals while eliminating the pathways through which experience is acquired.
Humans Do Not Need to Understand Everything
There is another side to the argument.
Suppose an advanced AI system can design a spacecraft that no single human engineer could fully construct or understand independently.
Should the spacecraft remain unbuilt?
Not necessarily.
Modern engineering already depends on distributed knowledge.
No individual understands every component of every complex system.
The relevant question is whether the design can be sufficiently evaluated, tested, challenged and governed.
Can its critical properties be verified?
Can its assumptions be examined?
Can serious failure modes be identified?
Can the project be stopped when justified?
And who holds the authority to make those decisions?
Human control does not require universal human expertise.
It requires meaningful competence, accountable institutions, reliable methods of verification and effective intervention mechanisms.
Where adequate verification is impossible, that limitation must constrain deployment.
Ignorance does not disappear simply because a machine produces an impressive result.
The Illusion of Responsibility
Researcher Madeleine Clare Elish introduced the concept of the moral crumple zone to describe how responsibility may be placed on a human operator who had limited actual control over an automated system.
It is an unsettling idea.
The machine receives operational authority.
The human receives responsibility.
When something goes wrong, attention turns to the person who approved the result.
But was that person genuinely able to evaluate it?
Did they have the necessary information?
The necessary time?
The necessary competence?
The authority to intervene?
If those conditions were absent, the approval may have been little more than a ritual.
The problem becomes especially serious when organisations use human oversight to create an appearance of accountability without providing the conditions necessary for meaningful control.
Responsibility without effective authority is not genuine oversight.
It is an accountability failure.
Individuality Matters
Human-AI collaboration cannot be designed around a single imagined user.
People differ in experience, expectations, risk tolerance, expertise and preferred working methods.
So do their tasks.
Someone using AI to develop a dinner recipe does not require the same supervision structure as someone using it to evaluate an engineering design or support a medical decision.
Even within the same profession, individuals may prefer different degrees of initiative from an assistant.
Some want detailed explanations.
Others want concise results.
Some prefer to investigate every assumption.
Others want routine tasks completed independently, with interruptions only when something important requires attention.
A mature system should accommodate these differences.
But personalisation introduces its own danger.
An AI must not quietly turn assumptions about a person’s abilities, emotions or personality into restrictions on their autonomy.
The user should be able to understand and challenge significant adaptations.
At the same time, personal preferences cannot override essential safeguards where consequences may be severe.
Individual adaptation must not become invisible authority over the individual.
The objective is neither unconditional obedience nor constant supervision.
It is proportionate, intelligible and accountable collaboration.
Collaboration Is Not Automatically Better
Another uncomfortable finding deserves attention.
In a 2024 systematic review and meta-analysis of 106 experiments, Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone found that human-AI combinations did not, on average, outperform whichever performed best independently.
The results varied substantially by task. Creative work showed more promising gains from collaboration than many decision-making tasks.
This challenges a common assumption.
Adding a human to AI does not automatically improve the outcome.
Adding AI to a human does not automatically improve it either.
The structure of the collaboration matters.
Who contributes what?
When should the machine act independently?
When should the human intervene?
What information must cross the boundary?
And when is the collaboration itself introducing more problems than it solves?
These questions cannot be answered by declaring that humans must always remain involved.
Human participation must have a meaningful function.
The Human-Bearing System
Westphal Human Systems is built around a distinction that I consider essential.
A human-bearing system helps the human carry reality.
A machine-bearing human helps the system sustain its own momentum.
This is not primarily a distinction between different technologies.
It concerns the architecture of the relationship.
A system may dramatically improve productivity while weakening the person’s ability to understand what is happening.
It may deliver accurate answers while reducing opportunities for independent learning.
It may preserve formal human approval while removing the practical conditions necessary for meaningful judgement.
And it may become so cautious about human autonomy that it transfers unnecessary cognitive and operational burdens back to the user.
In each case, the technology may appear successful according to one performance measure while failing according to another.
This suggests an additional requirement for evaluating human-AI systems.
Performance should not be measured exclusively by the quality or speed of the immediate result.
It should also be examined in relation to what happens to the human capabilities on which the collaboration depends.
Does the system preserve meaningful human authority?
Does it make consequential assumptions and limitations accessible?
Does it support the development or maintenance of relevant competence?
Can the user challenge the result?
Can the system be interrupted?
Can responsibility be traced to the actors who genuinely exercised control?
These are not merely philosophical questions.
They are design questions that can be translated into observable tests.
What Must Remain Human?
Perhaps the original question needs to be reframed.
The objective should not be to preserve a predetermined collection of tasks for humans simply because humans used to perform them.
That would turn technological progress into an exercise in protecting obsolete routines.
Nor should the objective be to transfer every technically automatable activity to machines.
That would confuse capability with legitimacy.
The essential issue is human agency.
The ability to formulate and challenge purposes.
The ability to recognise when technical efficiency conflicts with human values.
The ability to evaluate important consequences.
The ability to question authority, including the authority exercised through algorithms.
The ability to learn and develop judgement.
The ability to delegate deliberately rather than surrender control unknowingly.
The ability to withdraw authorisation when circumstances change.
And the ability to participate meaningfully in decisions that shape human lives.
These capacities need not reside in one person.
They can be distributed across individuals, professional communities and accountable institutions.
But they cannot be replaced by the mere appearance of human involvement.
The Border That Matters
Frederikshøj was once a place where an international border divided two territories.
The border eventually changed.
Its history remains.
Today, another boundary is being negotiated.
Not between countries, but between human judgement and machine capability.
It is unlikely to be a fixed line.
Different tasks will require different divisions of responsibility.
New technologies will alter what is possible.
New experiences will alter what is necessary.
But one principle deserves to survive those changes.
Human beings should not be required to perform work that machines can responsibly undertake. Yet machines should not assume that work in ways that destroy the human capacity to understand, challenge or govern the consequences.
I return to those seven centimetres.
A measurement can be perfectly accurate while the conclusion drawn from it is wrong.
A decision can be formally authorised while the person authorising it has no meaningful understanding of its consequences.
And a system can be extraordinarily capable while the human beings around it become increasingly unable to question what it does.
That is not the inevitable future of artificial intelligence.
It is a design risk.
And design risks can be investigated, challenged and reduced.
The question, then, is not simply what AI will become capable of doing.
It is what human beings will remain capable of doing when AI does it.
What remains?
Perhaps the most important thing is not the work itself.
Perhaps it is the human capacity to recognise when the work, however brilliantly performed, is taking everything in the wrong direction.
About the author
Thorsten Westphal
Thorsten Westphal is a Danish psychotherapist and psychotechnological system designer. Drawing on years of experience working with war veterans and human behaviour, he explores the intersection of artificial intelligence, human judgement, creativity and responsibility. His work focuses on a simple but consequential question:
How can technology strengthen human agency without quietly taking it over?
References and Research Foundations
1. Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), 775–779.
https://doi.org/10.1016/0005-1098(83)90046-8
2. Lee, H.-P., et al. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. ACM CHI 2025.
https://doi.org/10.1145/3706598.3713778
3. Bastani, H., et al. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26).
https://doi.org/10.1073/pnas.2422633122
4. Vaccaro, M., Almaatouq, A., & Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8, 2293–2303.
https://doi.org/10.1038/s41562-024-02024-1
5. Elish, M. C. (2019). Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging Science, Technology, and Society, 5.
https://doi.org/10.17351/ests2019.260
6. Brynjolfsson, E., Chandar, B., & Chen, R. (2025; revised August 12, 2026). Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence. Stanford Digital Economy Lab.
https://digitaleconomy.stanford.edu/publications/canaries-in-the-coal-mine/
7. European Union. Regulation (EU) 2024/1689, Article 14: Human Oversight.
https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14
8. Harding, M. Frederikshøj. Lex, Danish National Encyclopedia. Historical background on the former border crossing and Frederikshøj Inn.
Editorial and Methodology Note
This essay was developed by Thorsten Westphal with AI used as a dialogue, research and editorial tool. The conceptual direction, methodology, interpretations and final responsibility remain with Thorsten Westphal.
The three proposed risk categories are part of the author’s human-centred systems-design framework. They are analytical propositions rather than independently validated scientific constructs.
© Thorsten Westphal | Westphal Human Systems
