What Are You Looking For When You Review AI’s Work?
Conceptual illustration
7 MINUTES READ
You have reviewed the output.
But what, exactly, did you check?
By Kåre Bjørn Jensen
“A general feeling that something ‘looks good’ leaves considerable room between them.”
It looks right. But is that enough?
Imagine asking an AI assistant to summarize a report for an upcoming meeting. Within seconds, you have a clear introduction, a few well-organised findings and a confident conclusion.
You read it through. Nothing immediately looks wrong. You make a small adjustment to the wording and send it to your colleagues.
You have reviewed the output. But what, exactly, did you check?
Perhaps you checked whether it was readable. Perhaps you recognized the topic and found the argument plausible. Did you also check whether the summary preserved the report’s qualifications, distinguished findings from assumptions, or omitted something that might change the decision?
These are different assessments. A general feeling that something “looks good” leaves considerable room between them.
As AI becomes part of everyday work, we need a more concrete understanding of what human review involves.
Give judgment something specific to do
In my work as an educator and consultant, I teach people to use tools ranging from Excel and Power BI to generative AI. Across these tools, a useful distinction is whether a result looks reasonable and whether we have grounds for relying on it.
Consider a hypothetical spreadsheet task.
You ask an AI assistant to calculate the average time your organization takes to resolve customer inquiries. It suggests a formula. The formula runs, produces a plausible number and generates no error message.
That establishes something useful: Excel can evaluate the expression.
It leaves other questions unanswered.
Does “time to resolve” mean calendar days or working days? Should unresolved enquiries be excluded? Are reopened cases counted from their original date? Does the calculation include only the relevant reporting period?
A formula can operate exactly as written while answering a different question from the one the organization intended.
The reviewer therefore needs more than a working spreadsheet. They need a definition of the task and a way to test whether the result meets it.
The same applies to an AI-generated summary, presentation or recommendation. Before evaluating the answer, we need to decide what would make it suitable for its purpose.
What the research tells us
A 2025 study by researchers at Microsoft Research and Carnegie Mellon University surveyed 319 knowledge workers and collected 936 examples of generative AI use. Greater confidence in AI’s ability to perform a particular task was associated with less self-reported critical thinking. Greater confidence in one’s own ability to perform or evaluate the task was associated with more.
The limitations matter. This was a survey of reported experiences, rather than an experiment demonstrating that AI causes a decline in thinking ability. Confidence in one’s own skills is also different from independently measured competence.
For this article, the useful implication is a question: when we feel confident enough to accept an answer, what supports that confidence?
It might be relevant expertise. It might be a successful test. It might simply be that the answer resembles the kind of answer we expected.
Those grounds deserve different weight.
Decide what would count as a good answer
A practical starting point is to define a few acceptance criteria before asking AI to produce the work.
For a report summary, you might require that it preserve significant uncertainties, include findings that challenge the main conclusion and allow important claims to be traced to the original document.
For a spreadsheet calculation, you might specify the business definition, the treatment of missing values and a small set of examples with known results.
For a customer email, accuracy is only part of the assessment. The message must also fit the relationship and avoid commitments the sender has no authority to make.
These criteria give the review a purpose. They also make disagreement more useful. Instead of saying that an answer feels wrong, you can identify the requirement it has failed to meet.
Some criteria will emerge only after you see the first draft. That is a normal part of developing an idea. The important thing is to retain the ability to question the draft’s assumptions, including those you initially accepted.
Look for evidence that could change your mind
Once a result appears convincing, a useful next step is to identify a check that could reveal a meaningful weakness.
Return to the spreadsheet example. Rather than inspecting the formula and deciding that it looks sophisticated enough, create a few simple cases whose answers you already know. Include an unresolved enquiry, a case spanning a weekend and a reopened case.
For a summary, choose a consequential claim and compare it with the relevant passage in the original report. Check the surrounding qualifications as well as the sentence that seems to support it.
For a recommendation, ask which assumption carries the most weight. Then investigate whether that assumption holds in your situation.
AI can help propose these checks. It can suggest edge cases, identify possible omissions and offer alternative interpretations. Those suggestions can improve the review, but an explanation generated by the same tool does not independently establish that its original answer is correct.
Research also suggests that encouraging active engagement can involve a trade-off. In a 2021 experiment with 199 participants, Buçinca and colleagues found that interventions designed to make people think more deliberately reduced overreliance on incorrect AI advice compared with simpler explanation interfaces. The most effective designs also received less favourable subjective ratings. The study concerned a specific decision-support task, so it does not establish how every generative AI workflow should be designed.
My practical interpretation is that a review process should make room for useful effort. A check may interrupt the flow precisely because it requires us to reconsider something we were ready to accept.
Make the review proportionate and possible
There is an obvious objection: if every AI-generated sentence requires extensive investigation, much of the benefit disappears.
That objection is reasonable. Different tasks deserve different levels of scrutiny.
A draft invitation for an internal workshop may need a quick check of the date, location and tone. A calculation used to allocate resources deserves closer examination. A recommendation affecting someone’s employment requires relevant expertise and an appropriate decision process.
A useful starting question is
If this is wrong, what happens next?
Consider who could be affected, whether the error would be visible and how easily it could be corrected. Use those answers to decide where to spend attention.
Organizations also need to make that attention possible. Asking employees to review AI output means giving them access to the source material, time to examine it and a route to someone with relevant expertise. They need permission to delay or reject an answer when the evidence is insufficient.
Someone unfamiliar with a subject may be able to judge whether an explanation is clear while remaining unable to assess its completeness. Recognizing that limit is itself a valuable judgment. Sometimes the appropriate decision is to seek help.
Be able to explain why you accepted it
Accepting AI-generated work without changing it can be entirely reasonable. Extensive editing, by itself, demonstrates very little about the quality of the review.
What matters is whether the decision to use the result has a defensible basis.
For consequential work, try completing this sentence:
I am comfortable using this because…
A useful answer might refer to a checked source, a test with known results, a clearly defined requirement or a qualified colleague’s assessment. It should also leave room to say what remains uncertain.
You do not need absolute certainty to act. You do need to understand enough about the result, its purpose and its limits to decide whether the remaining uncertainty is acceptable.
The next time an AI answer looks good, pause before approving it.
What were you looking for and what would have made you decide otherwise?
About the author
Kåre Bjørn Jensen is a Danish independent educator and consultant and the owner of Intellia. With more than 20 years of teaching experience, he works with practical AI adoption, data tools and learning. His focus is on helping people make informed decisions about technology in their everyday work.
References and Research Foundations
1. Lee, H.-P., et al. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. CHI 2025.
Read the paper
2. Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proceedings of the ACM on Human-Computer Interaction, 5, CSCW1, Article 188.
Read the paper
