How to measure AI productivity without hiding review work

A colleague reports that an AI tool cut a weekly report from three hours to forty minutes. That number usually describes one stage: how long it took to get a draft.

It rarely includes the manager who now reads the report more carefully, the analyst who re-checks every figure against the source, the second round of edits, or the client correction that arrives a week later. Those costs are real. They often land on different people, at different times, in different systems from the one where the draft was produced.

If you have already chosen a first task, our guide on how to start using AI at work explains how to run a small first-week pilot. This article goes one level deeper: how to design a measurement that will survive scrutiny when someone asks whether the tool is worth paying for.

Why drafting time is the wrong denominator

Productivity is output divided by input. For knowledge work, the honest version is accepted outputs per hour of total human time: everyone's time, from the person who writes the prompt to the person who signs off the result.

An "accepted output" is one that meets the standard the work had to meet before AI arrived. A faster draft that a reviewer sends back twice has not been completed yet.

The clearest demonstration of why this matters comes from METR, a non-profit research group. In a randomized controlled trial run in early 2025, 16 experienced open-source developers completed 246 real tasks in mature projects they knew well. Each task was randomly assigned to allow or disallow AI tools. The outcome was total implementation time, explicitly including time spent before and after pull-request review.[1]

Before starting, the developers forecast that AI would cut their time by 24%. Afterwards, they estimated it had cut their time by 20%. The measured result: allowing AI increased completion time by 19%.[1] METR's summary draws the lesson directly:

“we now have strong evidence that anecdotal reports/estimates of speed-up can be very inaccurate.”

— METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025).[2]

Screen recordings showed where the time went. With AI allowed, developers spent less time actively coding and searching for information, and more time prompting, waiting on and reviewing AI output, and idle. They accepted fewer than 44% of AI generations, and on the 44 AI-allowed issues with valid labelled screen recordings, about 9% of their time went to reviewing and cleaning up AI output.[1]

That result has firm limits. It covers one group of developers, early-2025 tools and large codebases with very high review standards. METR itself now describes the findings as out of date: a follow-up study begun in August 2025 suggested developers were likely sped up by later tools, but METR judged its own new data an unreliable signal because developers increasingly declined to work without AI.[3] The point for measurement is not that AI slows people down. It is that perceived and measured time can point in opposite directions, so you have to measure.

The six parts of an AI-assisted task

To see review work, break each task into stages and time all of them:

1. Setup: gathering context, writing the brief or prompt, choosing inputs.
2. Drafting: generating the first version, including waiting for the tool.
3. Review: reading the draft against the source material and the brief.
4. Rework: editing, regenerating, or rewriting sections that missed.
5. Verification: fact-checking, testing, or a second person's approval.
6. Downstream correction: errors that escape and are fixed later by someone else, such as a reopened ticket, a rolled-back change or a client query.

Time the same six stages for the baseline process too. Manual work also has review and correction; the comparison is only fair if both sides are counted the same way.

AI Chronicle

Put quality and errors inside the metric

Time without quality rewards fast, wrong work. The strongest studies avoid this by building quality into what they count.

Count resolved work, not attempts. Brynjolfsson, Li and Raymond studied a generative AI assistant rolled out in stages to 5,172 customer-support agents at a software firm. Their headline measure was issues resolved per hour, which combines handle time with the share of conversations actually resolved; the firm also tracked customer satisfaction. Access raised resolutions per hour by 15% on average and by 30% for less skilled and less experienced agents. The most experienced and highest-skilled agents saw small gains in speed and small declines in quality.[4]

The authors stress these are medium-run effects at a single firm, with a tool built on GPT-3.[4] (An earlier 2023 working-paper version reported 14% and 34% for a slightly different sample of 5,179 agents.[5]) For measurement, two lessons travel: choose a unit that only counts work that succeeded, and report results by experience level, because an average can hide a loss among your best people.

Score correctness separately from polish. In the Harvard Business School and Boston Consulting Group experiment by Dell'Acqua and colleagues, 758 consultants were randomly assigned to work with or without GPT-4. On 18 tasks inside the AI's "frontier", AI users completed 12.2% more tasks and worked 25.1% faster, with significantly higher quality.[6]

On a business-case task deliberately designed to sit outside that frontier, the picture reversed. About 84.5% of the control group reached the correct recommendation, against 60% and 70.6% in the two AI groups: on average, 19 percentage points lower.[6] Yet graders rated the AI users' recommendations as more coherent and persuasive, whether or not they were right. The authors conclude:

“AI improves presentation and argumentation even when the underlying analysis is flawed.”

— Dell'Acqua and colleagues, Navigating the Jagged Technological Frontier, Organization Science (2026).[6]

If your quality check is a manager skimming for clarity, it can reward exactly the outputs that most need correcting. Check correctness against a source or an answer key, separately from how well the work reads. And segment by task type: the same workflow can contain tasks on both sides of the frontier. The study used one outside-frontier task and the GPT-4 of April 2023, so the location of that frontier will have moved.[6]

Notice when a study leaves review out. Noy and Zhang's experiment, published in Science, gave 453 college-educated professionals short writing tasks and randomly gave half access to ChatGPT. Average time fell by 40% and quality, rated by blinded professionals in the same occupations, rose by 18%.[7]

The tasks took 20 to 30 minutes, were self-contained, and did not require precise factual accuracy; the researchers did not think it feasible to hire fact-checkers.[8] In the March 2023 working-paper version, 68% of treated participants reported submitting ChatGPT's initial output without editing it.[9] In other words, much of the review and verification your workplace would require was absent from the setting. Noy made the same point when the paper was published, noting that real-world speed benefits may be smaller because "you need to spend time fact-checking and writing the prompts."[8] That is not a flaw in the study. It is a reminder to add those stages back when you measure your own work.

Measure the system, not just the individual

Faster individuals do not guarantee a faster team. Work that skips careful review moves the cost downstream.

Google Cloud's 2024 DORA report on software delivery, based on a survey of nearly 3,000 technology professionals, estimated that a 25% increase in AI adoption was associated with a 2.1% increase in individual productivity, 7.5% better documentation quality, 3.4% better code quality and 3.1% faster code review.[10][11] The same increase was associated with an estimated 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability. The report also found a 2.6% decrease in time spent on work respondents considered valuable, and 39.2% of respondents reported little or no trust in AI-generated code.[10]

The authors hypothesise that AI lets developers produce more code at once, so changes grow larger, and larger changes have historically been slower and more prone to failure. Their summary:

“improving the development process does not automatically improve software delivery”

— DORA, Accelerate State of DevOps Report 2024.[10]

These are survey-based associations. The report states plainly that its authors "didn't do longitudinal studies or a proper experiment."[10] The method still applies outside software: pair every individual speed metric with a downstream one, such as error escape rate, reopened work, or the size of each batch handed to reviewers. For how these findings fit the wider debate about jobs and wages, see our review of what the evidence says about AI and jobs.

A measurement design you can run

The studies above share a structure. Here it is adapted for an ordinary team.

1. Define the unit and the bar before you start. Name the output (a report, a resolved ticket, a merged change) and write the acceptance criteria. Agree the decision rule in advance, for example: adopt if total time per accepted output falls and the escaped-error rate does not rise.

2. Take a baseline. Measure the current process across the same six stages for several cycles of the work. Without it, every later number is a guess.

3. Prefer system timestamps to memory. Ticket opened and closed, draft submitted and approved, review comments and revisions. Where you need a time log, record it per stage as the work happens. Do not rely on end-of-week estimates.

4. Count reviewers' time. Reviewers, approvers and fact-checkers are where hidden costs usually sit. Record who touched each output and for how long.

5. Measure quality with a blinded check. Have a reviewer who does not know which outputs used AI score a sample against the rubric. Score correctness separately from writing quality.

6. Track rework and escapes. Revision rounds, outputs returned, and errors found after release within a fixed window.

7. Use a comparison group. Randomly assign tasks or people to AI-allowed and AI-not-allowed, or stagger the rollout by team so later teams act as the comparison, as in the customer-support study.[4] Watch for selection effects: METR's 2026 update found that 30% to 50% of surveyed developers were choosing not to submit some tasks because they did not want to do them without AI, and that time records became unreliable for developers running several AI agents at once.[3]

8. Segment before you average. Report by task type and by experience level. One blended number can hide a gain in one place and a loss in another.

If you are evaluating an AI agent rather than an assistant, the same arithmetic applies to every failed run. Our analysis of what AI agent demos leave out covers cost per completed task.

A worked example (hypothetical): weekly client status reports

This scenario is invented to illustrate the method. No team was studied, and every number below is a placeholder for the calculation.

An operations team of twelve analysts writes weekly status reports for clients. Each report summarises project data and flags risks. A manager reviews every report before it is sent.

The plan. For four weeks, the team logs time on every report across the six stages, and the manager logs review time. Then, for six weeks, each report is randomly assigned to "AI allowed" or "AI not allowed" when it is created. Assigning reports rather than analysts lets everyone serve as their own comparison. A colleague from another team checks a random sample of 30 reports from each group against the project data without knowing which used AI, scoring factual errors and writing quality separately. Client corrections are tracked for ten working days after each report is sent. The decision rule, agreed in advance: adopt if total minutes per accepted report fall by at least 10% and the escaped-error rate does not rise.

An illustrative result. Suppose a baseline report takes 90 minutes to draft, 20 minutes of manager review, 10 minutes of rework and 5 minutes of later corrections: 125 minutes. Suppose an AI-allowed report takes 10 minutes of setup, 25 of drafting, 45 of review because the manager now checks every figure, 25 of rework, 15 of verification and 10 of later corrections: 130 minutes.

Drafting time fell by 72%. Total time per report rose by 4%. A drafting-time metric would have reported a large win. The full measurement says the team should revise the workflow, perhaps by giving the tool the project data directly, or by limiting it to the narrative section, and test again. If reports with AI also showed more factual errors in the blinded check, that would count against adoption even with a time saving.

A team that later automates part of this work, as in our guide to building a growth reporting agent, should keep the same measures in place.

Questions readers commonly ask

Can we just ask people how much time AI saves them?

Ask, but do not rely on it. In METR's trial, developers estimated a 20% speed-up after experiencing a 19% slowdown.[1] Use self-reports to find where to look, then measure.

Do we need a randomized trial?

Not always, but you need a comparison. A baseline plus a staggered rollout across teams is often practical. Randomizing tasks or people gives stronger evidence when the stakes justify it.

How long should a pilot run?

Long enough for downstream errors to surface and for people to get past their first attempts with the tool. There is no universal figure. Set the window from how long your errors usually take to appear.

Are AI usage metrics, such as suggestion acceptance rates, enough?

No. They measure use, not results. A high acceptance rate can coexist with heavy rework, and a low one, such as the under-44% rate in METR's trial, says nothing alone about whether the work got done faster or better.[1]

What if quality rises but total time rises too?

Report both, and decide on value. Better work at higher cost may be worth it for some outputs and not others. The point is to make the trade-off visible rather than hide it in a drafting-time number.

The next step

Once you know what to measure, read the studies with the same eye. What workplace AI studies actually show reviews the wider evidence on AI and productivity, including what each study did and did not count.

Sources and evidence note

This analysis relies on two peer-reviewed field or laboratory experiments, one peer-reviewed online experiment and its working-paper version, one randomized trial preprint with two follow-up publications from the same research group, and one vendor-published industry survey. Each result describes its own population, task, tool and period; none is a forecast for another workplace. The weekly-report scenario and its numbers are illustrative. AI Chronicle did not run any study for this article. All sources accessed 16 October 2026.

Related reading

• AI for marketing: useful tasks and required checks
• Who is accountable when an AI agent acts?