halaspace
November 3, 2026 at 12:15 am · AHMED ALMURTADHA

Measure What Ships

When a developer gives an AI agent a coding task, the agent does more than answer once. It plans, generates code, runs tests, inspects errors, and tries again. The resulting token count can be much higher than that of a basic chat interaction.

That creates a measurement problem. Token counts show consumption, not whether the work produced useful software. A better approach is to connect AI usage with completed, reviewed changes.

AI coding costs are rising with usage

A conventional chatbot may answer a prompt in one exchange. An AI coding agent usually works through a sequence of actions:

  1. Interpret the assignment.
  2. Break it into subtasks.
  3. Inspect the existing codebase.
  4. Generate or modify code.
  5. Run tests or other checks.
  6. Diagnose failures.
  7. Revise the implementation.
  8. Repeat until the task is complete or abandoned.

The source material describes a range of roughly 10 to 50 times more tokens for a single agentic coding task than for a basic chat prompt. Actual consumption will depend on the task, model, repository, tools, and number of retries.

This creates a difficult pricing problem. The workflows that use the most inference may also be the ones doing the most productive work. A developer who asks an agent to inspect a large repository, run tests, and correct several errors will generally consume more tokens than someone requesting a short code snippet.

A simple usage cap treats both situations alike.

Why blanket limits can backfire

Companies facing rising AI bills may respond with:

  • Token quotas
  • Seat limits
  • Approval gates
  • Restricted access to more capable models
  • Fixed allowances or metered credits

Those controls can reduce waste, but they can also discourage developers from using agents for complex tasks. A system that stops midway through debugging may save tokens while leaving the developer with more manual work.

The central problem is that token consumption measures activity, not value. High usage may indicate waste, such as repeated failures or an uncontrolled loop. It may also reflect legitimate engineering work that results in tested and reviewed software.

A spending cap cannot distinguish those cases by itself.

A better engineering metric: cost per merged change

The proposed alternative is to track cost per merged pull request:

Total AI-assisted development cost divided by the number of changes that pass review and are merged.

This measure is not perfect, but it is closer to the outcome a software organization actually wants. A pull request that is reviewed, accepted, and merged represents completed work rather than attempted work.

Teams should pair the metric with quality indicators, including:

  • Rework after a change is merged
  • Defects linked to AI-assisted changes
  • Reverted pull requests
  • Review time
  • Test failures
  • Security or reliability findings
  • The proportion of tasks abandoned before completion

Without those measures, a team could reduce cost per merged change by accepting smaller, lower-quality changes or by weakening review standards.

Metric What it shows What it misses
Tokens consumed Model and workflow usage Whether the work produced useful software
AI spending Direct financial cost Quality and business value
Pull requests opened Development activity Whether changes pass review
Pull requests merged Completed reviewed changes Defects, rework, and long-term maintenance
Cost per merged pull request Cost of accepted output The complexity and strategic value of each change
Defect and rework rates Quality after delivery Some benefits, such as faster exploration or learning

The metric is most useful for comparing similar categories of work over time, not for ranking every developer or task with one universal number. A small documentation fix and a major architectural change should not be judged as though they have identical complexity.

Match the model to the task

Reducing AI coding costs does not necessarily mean asking developers to use less AI. It can mean using the right model for each part of the workflow.

A capable but expensive model may be justified when an agent must:

  • Understand unfamiliar architecture
  • Plan a multi-file change
  • Resolve subtle test failures
  • Reason about security-sensitive code
  • Make changes with significant downstream effects

A less expensive model may be adequate for:

  • Boilerplate generation
  • Repetitive transformations
  • Formatting
  • Simple test scaffolding
  • Documentation updates
  • Routine code search and summarization

This kind of routing can preserve access to advanced reasoning where it contributes to successful delivery while lowering the cost of predictable work. It also reduces the temptation to impose crude limits across all tasks.

What to measure instead

The goal is not to minimize token use at any cost. It is to improve the relationship between AI-assisted development spending and useful software output.

Track token and inference costs alongside merged changes, rework, and defects. If cost per merged pull request falls as AI usage rises without compromising quality, developers are converting more consumption into finished work.

The most useful question is not how many tokens an agent used. It is how much reliable software the investment produced.

Source

  • MIT Technology Review, “Stop counting tokens, start counting what ships,” published November 2, 2026: https://www.technologyreview.com/2026/11/02/1147702/stop-counting-tokens-start-counting-what-ships/

Leave a Reply

Your email address will not be published. Required fields are marked *