#ai#team#api

GitHub Measures Code Review: Where AI Slows Teams

GitHub exposed median and p90 code review stage data via API, including Sonnet 5.5, Copilot policy validation, and merge-time metrics.

  • The occasion: five GitHub updates in one week
  • Generation is getting cheaper, and it is measurable
  • The bottleneck has moved to review
  • Policies that silently fail to apply

The occasion: five GitHub updates in one week

  1. In the last week of September, GitHub released five updates: Claude Sonnet 5.5 in Copilot, code review stages in the Usage metrics API, a validator for Copilot enterprise settings, an AI agent for security audits, and Git 2.56.

  2. Together, they show where the team loses time.

  3. Models already write code quickly and cheaply.

  4. Everything after generation remains slow: review, policy enforcement, and security checks. A company that evaluates AI by the number of accepted suggestions measures the fastest part of the chain and misses the slowest one.

Generation is getting cheaper, and it is measurable

According to GitHub’s internal tests, Sonnet 5.5 solves coding tasks at the level of Sonnet 5, but uses fewer steps, tokens, and tool calls and finishes faster. A tool call is an agent action: reading a file, running a test, or searching the repository. Every action costs time and money. Copilot charges for the model at the provider’s rate under usage-based billing, so saved tokens directly reduce the bill.

The rule for managers is clear: compare a new model by the cost and time of one completed task. The benchmark “how smart is the model”

does not answer this question.

The bottleneck has moved to review

The week's most important update looks modest. The Copilot usage metrics reports now include the `pull_request_review_times` array.

It returns the median and 90th percentile for three intervals: - from “ready for review” status to the first review; - from the first review to the final review; - from the final review to the merge.

The median describes a typical pull request

The 90th percentile (p90) describes the long tail: 9 out of 10 PRs pass the stage faster than this value, while the tenth gets stuck.

Long-tail tasks are what derail releases

In this version, both fields, `authored_by` and `reviewed_by`, refer to people. GitHub started by measuring how long human reviewers wait, which is logical.

If AI doubles the number of PRs while the number of reviewers stays the same, the queue grows and the path from idea to production may even become longer.

These three intervals are part of what DORA calls lead time for changes.

Previously, teams collected this data with their own scripts; now GitHub provides it by stage through the API.

At KT.Team, we call this TTU, time to use: the time from a task to a working result.

If a PR waits three days for its first look, coding speed has almost no effect on TTU. Git 2.56 affects the same bottleneck.

It was built by 104 contributors, 39 of them for the first time. The changes include the ability to index resolved conflicts without adding the remaining edits to the index.

While there are few branches, this feature is hardly needed.

When agents open dozens of branches in parallel, we hypothesize that merge conflicts increase (GitHub did not measure this), and resolving conflicts carefully could save reviewers hours.

Assess where AI can deliver impact in your process

clients@kt.team Telegram @kt_team_it

Policies that silently fail to apply

  1. The enterprise managed settings validator checks `copilot/managed-settings.json`, `copilot/team-mappings.json`, and related team settings.

  2. It catches malformed JSON, unsupported parameters, and incorrect team mappings—in other words, errors that prevent the policy from being applied.

  3. For each error, it identifies the file and JSON path. The cost of such an error is high.

  4. The security team believes the model or external context is prohibited for the team, but the restriction is ineffective because of a typo in the configuration. In projects LLM & Security Gateway we keep model access rules in the repository and check them before deployment: which models are available, what data may be sent externally, and what is written to the log.

  5. An untested policy protects the company only on a slide.

An agent needs a workflow

  1. The GitHub Security Lab team found 24 vulnerabilities in Android applications using its open Taskflow Agent.

  2. The researcher breaks the audit into sequential steps and packages them into a reusable workflow, taskflow.

  3. The model goes through the steps sequentially and finds complex vulnerabilities that it would miss with one large prompt.

  4. The result came from combining the model with a deliberately designed process.

  5. We build on the same principle AI-native development: the task, context from the codebase and documentation via MCP, generation, automated tests, and review.

  6. We can change the model in this workflow in a day.

  7. The surrounding process is built over months, and it is what produces the result.

  8. The setup looks simple from the outside, but it is backed by months of engineering work and well-tested team rules.

What to do before the next AI budget

Record the median and p90 for the three review stages now, before changing the model or buying new licenses. Without a baseline, you cannot prove the effect later.

Track the p90 time to merge

If it increased after AI was introduced, delivery slowed down even though generation became faster.

Compare models by cost and time to complete a task, taking steps and tool calls into account.

Validate policy configuration as code, automatically and before deployment.

Give agents explicit step-by-step workflows instead of a single “find everything” prompt.

Conclusion

We interpret these metrics as follows: GitHub suggests evaluating a code-generation tool by how long it takes code to reach a merge. Businesses pay for changes in production. If, one quarter after introducing Copilot or Claude, your p90 from “ready for review” to merge has not decreased, you bought an expensive keyboard. The figures needed for verification are now available in the API, and a year from now the CFO will not accept “the impact is hard to measure” as an excuse.

Discuss the article: GitHub measures code review: where teams…

Enter your email or phone number so we can get back to you.

Send via:

By clicking the button, you accept the offer and grant consent to the processing of personal data.

Related Cases

All cases

Related solutions

All solutions
AI-native development: agent, repository and skills libraryOfficial PHP SDK for Model Context ProtocolLLM Fine-Tuning for Company Data: When It Makes SenseAI Accountant in Your Environment: Implementation Cost

Articles on the topic

AI in Business Collection

Related videos

All videos
Shorts What “KPI” did Amazon set for developers in the AI era? AITeams & hiringManagement
34:06 Automation and AI Adoption at PIK Group | Vibe Coding, the Developer Market | Artur Ishmaev Podcast AIAutomationTeams & hiring
36:37 Neural Networks Changed IT FOREVER! How to Grow FAST from Junior to Senior with AI? AITeams & hiring

News on the topic

All news