ReviewDebt: a practical framework for scoring every pull request — Sachin Gupta, Ebay

Transcript

Hi everyone, I'm Sachin Gupta, and I'm a software engineer. The title is exactly what it sounds like. Your coding agent is creating review date. Before I start, sit with the title for a second. Notice what it is not saying. I'm not saying that coding agents are bad. I'm not saying they don't make us faster. I'm saying they are creating a kind of debt that nobody is measuring. And that debt is going to come to you. Over the next few minutes, here is what I'm going to do. I will define what review date is. I'll walk five signal families that are going to compose it. I'll score three real pull requests side by side. And I will show you a cross-repo scan of over 500 PRs, which is nothing but three public code bases. So let's go. Look here. Here's the gap nobody is measuring. GitHub's 2025 Octover's report that covers almost every public pull request on the planet shows that the commits climb 25% year over year. And now, over the same year, comments on the commits drop 27%. Now these comments are nothing but the proxy for review activity. Code production volume actually went up, but the review attention went down. They moved in opposite directions, in the same year. Now look at the teams furthest along the AI adoption curve. Faro's AI tracks this cohort in their 2026 benchmark. Median PR review time is up by 441.5%. And if you calculate, you'll figure out that the review PRs take 5.4 times longer than what they used to. And then 31% more PRs are now merged with no review at all. So AI is producing the code very fast. AI is producing the pull request very fast. But humans cannot responsibly review them at that pace. This gap is called reviewed at. It actually accrues quietly. It compounds. And right now, nobody has a number for it. But by the end of this talk, I will make sure that you will have a certain number. Now, if you look at this particular slide, here is the story every team is telling right now. PR per developer are up 16%. That's the Faro's AI acceleration whiplash benchmark, and that too from April 2026. There were 22,000 developers, 4,000 teams. And the median PR size is up 63%: 44 lines to 72 lines per pull request. That is 16 months of long-term data from DX 2026 study, which includes 400 organizations. Now, if you see the cycle time open to merge, that is modestly down. And it is coming from the same DX study. The framing is very generous. PR throughput grew actually about 8%. And the AI usage rose about 65%. The gain that we see today is real. But it is actually smaller than the hype that is present. Every one of these numbers are real. None of them is a lie. But every one is a vanity metric. PR count goes up when one PR spreads into seven PRs. Median PR size going up is not a benefit. It's actually bloating. Cycle time going down when reverse stop pushing back. These things tell you the speed of production. They do not tell you the speed of trust. Now, we will go into the next slide to see more. Now, if you go looking for the second story, here is what you're going to find. Reviewer fatigue. Engineers carry far more review load than a year ago. They're not happier about it. Late night merges. PRs are sitting unreviewed for three days, four days, and then they suddenly get a thumbs up at 11 p.m. before a Friday deadline. Test theater. Tests are getting added, but they are actually asserting what the code did, not what the code should do. They lack in behavior, including bugs. Architectural drift. Again, the same problem that is solved in three different ways in three different ways. Nobody is actually holding up the architectural thread. Finally, incident lag. When something breaks, the bug lands weeks or months after merge. Nobody connects the dot back to the AI author change. These costs don't show up in any dashboard until now. We'll look into the next slide to know more about it. So now, let me give you the definition. Review debt. It's the accumulating gap between code your agent has produced and the code humans have actually reviewed, trusted, and understood. It rhymes with technical debt, but it is more like a financial debt because it compounds. It actually accrues interest, but this interest is not money. This interest is paid in human attention. It compounds because of the three feedback loops. First, the agent learns from your code base: fine-tuning, rag-grounding, in-context suggestions. Code that was not deeply reviewed yesterday grounds tomorrow's PR. This debt becomes generative in nature. Second, reviewers seed the architectural call. When most of a PR was generated, attention contracts to syntax and obvious bugs. Big picture decision moved from review time to never. Third, velocity expectations reset. Once leadership sees the new throughput, you don't get to hire reviewers in proportion. There is no slack left to pay the debt back. Each loop on its own is survivable. Together, you will have a runway. But the next question is, how are you going to measure it? So, how do we measure review debt? There are five signal families that have 10 deterministic checks. Remember, the keyword here is deterministic. Every check is computable from a pull request and its repository. No, we don't need any language model. Why? Because LLM, when it acts as a judge, it possibly can break two things. Number one, the score becomes a moving target. The same PR, the score that you got for the same PR will score differently when your model will change. Two, the score stops being defensible in an engineering review. You cannot put it on a slide. You cannot put it in front of a leadership. So, what do you want? You want a number that is traceable to a deterministic computation, different size and coupling. Second, test evidence gap. Third, directory and ownership spread. Fourth, AI authorship indicators. Fifth, evidence and rational gaps. These are the five families that will have 10 checks under them. Let's walk each of them. The first one, the first signal that we have is def size and coupling. This is the simplest one to calculate. It's also the most often misread. What it measures is the net lines changed, the files touched, whether changes cluster in one module or are sprawling across many. Why is the agent struggling here? Agents are biased toward fix at the call site. A human engineer routes a fix to the root cause. That keeps the difference very small. But an agent will reach into many files looking for the same symptom. The reviewer cost of a sprawling difference is not proportional to the size. It is actually much steeper. So the cross-pile coupling explodes the mental model and reviewer has to hold. We'll look into the second signal, which is test evidence gap. So what is test evidence gap? Test evidence gap is nothing but test lines added divided by the production lines added per pull request. So AI-authored PRs, they ship with a far lower test-to-code ratio. Sometimes there is no test file in place, and if there is any, it's minimal. So this gap is actually a consistent gap. And the problem with this particular thing is the number. The reason this number is brutal is the agent generate test. They generate a lot of tests. But these tests basically assert what the code is doing. It does not generate the test cases what the code should actually do. They lock in the behavior, including the bugs. The ratio does not capture the quality gap. It just measures whether the tests showed up at all. So the deeper signal is the next step that is a layer down. We'll see the next one, which is directory and ownership spread. So what is directory and ownership spread? Count the distinct code owner teams whose file appear in the difference. Now what does it mean? Basically, a well-shaped PR concentrates in one team territory. But a stalling one reaches across many team files, and the reward cost is enormous in this scenario. No single human hold the whole mental model. You need multiple approvals from multiple engineers in multiple different contexts. The coordination overhead easily exceeds the time the agent save producing the code. This is where you start to feel the economics of review debt. The agent gave you three hours of typing. You spend those three hours by multi-party reviewer attention bank. And the next one we are going to see is AI authorship indicators. So what is AI authorship indicator? Before everyone gets defensive, this is not for the blame. We are not flagging that engineers use coding agents. We wait the scores to a PR shape like agent-assisted authorship get extra reviewer attention. So that way you know, this is the PR that is directly coming from the agent. And there are the very common three detection modes. I think everyone must have seen it by now. One is the co-author! Which is basically one of your strongest signal. You see that co-authored by copilot. Second one contexts. The coordination overhead easily exceeds the time the agent saves producing the code. This is where you start to feel the economics of review debt. The agent gave you three hours of typing. You spend those three hours by multi-party reviewer attention bank. And the next one we are going to see is AI authorship indicators. So what is an AI authorship indicator? Before everyone gets defensive, this is not for blame. We are not flagging that engineers use coding agents. We weight the scores to a PR shape like agent-assisted authorship get extra reviewer attention. So that way you know, okay, this is the PR that is directly coming from the agent. And there are the very common three detection modes. I think everyone must have seen it by now. One is the co-author, which is one of your strongest signals. You see that co-authored by Copilot. Second one is the branch name pattern. Generally, you must have seen it says Codex, Copilot, Cursor prefixes. Third one is generated by, assisted by, which is nothing but your PR body or commit message phrases. We did a real data check on three public repos, which comprised 524 PRs. I am not going to name the company, but for the sake of this, we are just calling it A, B, and C. Now if you see, the highest signal was the co-author footer, and the second one, the lowest signal, was 0% on repo C. Even though it was done by a coding agent, they might have blocked it, saying any co-authored or generated by, etc. The next thing that we are going to see is evidence and rationale gaps. So evidence and rationale gaps. I think this is one of the most deterministic that I personally feel. It's the one that destroys reviewability the fastest. What it measures, basically: does the PR explain the why, or just the what? On the left, if you see, we have a high gap. On the right, we have a low gap. The title says fix leaky test, and all the data that you are seeing here is actually from some public reports. Now the PR body length is 18 characters. The commit message says updates. Obviously, if you give it to me, I won't be able to review this. Okay, I cannot accept this. But on the right, if you see, when there is actually a low gap, the title actually is telling you what the change is all about. The body has a symptom, the diagnosis, the change, and the link to the benchmark. Now a reviewer can do the job. So in our regression fixtures and hand-coded teams, this signal destroys reviewability fastest. In the real open source data, PR bodies follow conventional commit format, so it fires most rarely there. But if you see, the regression fixtures are just the demos. Okay, let's see how those five signals combine. You will only have one number that is going from 0 to 100, and the exact weights are basically your defaults. Now if you are planning to adopt this, I would like you to do this first. Run it backwards over the last 200 PRs that have been merged in your company. Calibrate the weight against your team's actual experience. The score has to feel right against your actual gut. You are going to divide it into four bands. If you are 0 to 24, which means you have very low review burden. If you have 25 to 49, you are normal, proceed with the standard care. 50 to 74, you need evidence from the author before senior review. And if you are 75 and above, definitely it's high, split or request more context. Same shape as technical debt categories, but the unit is different. Now we are going to see what the scanner actually did. Now if you see, this is the clean PR. Unfortunately, I won't be able to give you a live demo today, but I will walk you through a CLI. I want to show you three scored pull requests side by side, the level where the framework becomes useful as a repeatable review conversation. This is the first one. If you see, this is what the scanner says when a PR is well shaped. Score 0 out of 100. The burden is basically nil. It's 0. We have a low review burden. The estimated minutes is 6. No checks were fired. And if you look at the structure here, that is why the list is empty. Reviewer focus is empty. Author next action is empty. The framework only generates specific advice when it has something specific to say. A healthy PR produces a tiny, almost ceremonial report. That's exactly what you want. And the takeaway is most healthy PRs produce zero noise. They get the work done without making any noise. This is not a tool that complains by default. Running it on every PR costs you one comment that says look good. That's it. Let's see the high debt PR now. Now what you see actually on the screen is from a public repo. This is a real scenario from the scanner regression suite. If you see, the score is here, 60 out of 100. It requires evidence. It's on the orange, it says needs evidence. 86 estimated minutes of review effort. Look at the structured output. It's actually not a score, a wire list explaining what fired, a reviewer focus list telling the reviewer what to do next, an author next action list telling the author how to bring the score down. This is the part most PR quality tools miss. The score alone is useful. The structured advice is what actually moves the team behavior. Now if you read the bullet, it says PR has soft indicators of AI-assisted authorship. This is information only, not a definitive claim, and not a penalty on its own. This particular sentence is in the scanner output. The AI indicator check contributes 5 of the 60 points, which is nearly about 8%. The other 55 came from a different size claim mismatch and missing tests. Those would be high-burden signals on any PR, agent-authored or not authored. This is not an anti-AI scorecard. Basically, this is more of a review burden scorecard. The agent did not cause the score, but the shape of the PR that is created by the agent did this. Let's see the next one. This is the PR, basically, which is AI-authored. It's very well shaped, and the score is 7, which is low review burden. Now if you see, the same agent, it has done the same kind of work, but the tests were added. So the CI is green. The risky path is called out in the PR description. That's why we got the score as 7 out of 100. It's very low burden, took only 14 minutes. The AI indicator check still fires, but it contributes two points. One, low severity. Second, the framework explicitly says, quoting the report itself, information only, not a definitive claim, and not a penalty on its own. So when your team is going to ask, will this tool penalize us for using coding agents? No. The answer is on your screen: low review burden. It's a very well shaped AI PR. We are going to see the next one, which is basically our three public repos over 524 PRs. I wanted to show you a 90-day AI sub-slope across three public repos and 524 pull requests. The review burden climbs even when the AI authorship doesn't. If you see this particular scan, we observed three things. One of them is the volume is the actual variable. AI authorship was flat, and it was 5-20% steady state, but the review burden was not. If you see, one repo accumulated 186 senior reviewer hours in 27 days, and another one took 43. The window length is same, but the volume was not the same. The burden was not the same. That is what PR volume is. Second is amplifier only holds up under real data, which means AI indicators fired on 5-20% of PRs every week across all three repos. None of those AI positioning contracts holds against real codebases. Third one is complexity drives burden, not authorship, which means across 524 PRs, four landed in needs evidence or high bands, which means all four were structural changes, which were large migrations, SDK rewrites, multi-team refactors. The framework scores complexity fairly. AI-driven volume creates the condition under which these accumulate. Where does the review debt actually show up in absolute terms? The scanner saw 524 real pull requests in three public repos. 12-28 senior review hours accumulated across three public repos that were scanned over 27- to 90-day windows. Second one is 9 PRs per day, sustained merge rate at high-velocity repos. So if you see, it is all about the volume that is basically driving the burden. Third one is 5036. These are the minutes that are basically spent on a single PR, which is nothing but 84 hours of review effort estimated for one single pull request. The score that we received there was 73, which said it requires an evidence spend. Now 5-20 percent of these PRs every week fire the AI indicator signal. Again, remember, this is volume which is causing the review debt, which is changing the game altogether. We'll move to the next slide. We have shown the cost. Now what do we do about it? So first thing is you should have one logical change per PR. I'm not saying the scanner saw 5-24 real pull request in three public repos. 12-28 senior review hours accumulated across three public repos that were scanned over 27 to 90 days windows. Second one is 9 PR per day sustained merge rate at high velocity repos, so if you see it, all about the volume that is driving the burden. Third one is 50-36. These are the minutes that are spent on a single PR, which is nothing but 84 hours of review effort estimated for one single pull request. The score that we received there was 73, which said it requires an evidence spend now. 5-20 percent of these PR every week, across every week, fire the AI indicator signal. Again, remember, this is volume which is causing the review debt, which is changing the game altogether. We'll move to the next slide. We have shown the cost. Now, what do we do about it? So, first thing is you should have one logical change per PR. I'm not saying not small PR in the abstract. One logical change, that is enough. Second, test shift with the change. Even if the agent wrote the code, even if the agent wrote the test, the human author confirms the test assert what the code should do, not what the code is doing, what the code is supposed to do. Okay, third, stay in one owner territory. Cross-cutting work, splitting into per-team PR, remain in your own territory. One approval, one context, one mental model. Author writes the why. The agent should not write the PR body. That's the moment the human author commits to understanding what they are actually shipping. Same review standard for the AI PRs as human PRs. The AI indicator amplifier only fires when other signals are weak. Keep the other four strong. The amplifier is invisible, so make sure there is no exception for AI. None of these is actually requiring you to have a new tool. These are the moves that you already are aware of. You just need to implement it. Now, how are you planning to adopt this? If you see, we again have five steps: back fill, threshold, surface, aggregate, and talk about it. Back fill: so you run the scorer over your last 200 merge PR. Look at the one that has the highest score. Second, threshold: set up justify line default. Let's say you want to give it 50, so any PR scored at 50 or above requires a comment from the author. As simple as that. Surface it: post the score as a PR comment on every PR. Don't block it, just for the visibility purpose, surface it so that everybody should know what's happening over there. Aggregate weekly per team, which means the slope of each team deadline is the leading indicator. That's what your engineering manager should watch. Talk about it: bring the number to a retrospective, every roadmap view, in every meeting. You should discuss that number, like what was your score when you did this particular PR? That's why this number matter, because of this conversation. Absolutely not. Without a number, what you are just converging is Y, but if you have a number, it's actually structured. You can tell, okay, with AI coding agents, we have a rollout which has added X percentage of the throughput, but you need to make sure you are also adding the Y points of the review debt that has. If not, then who is accountable when an AI author change causes an incident? Where is the audit trail? So review debt is the bridge between these two columns. It's the first number that lets you have the right-hand-side conversation without abandoning the left-hand-side gains. sinn sinn sinn sinn sinn sinn sinn sinn sinn sinn sinn sinn sinn sinn over time matters most than the level, so plot it weekly and then discuss it with your team, maybe in a meeting or in a sprint planning, whatever the way you feel like. Have this conversation. Bring the number to your next review, say that, okay, we do have extra put wide it and this is the slope, and then you are moving away from the vibe culture to the measurement culture. One quick callout on anti-patterns to avoid. First thing is approved-with-comment mergers. The AI did the boring part. No. Will you catch it in QA? No. PR are smaller. No, no. Stop writing LGTM on the PRs. Make sure whatever we have discussed, the habits that we have discussed, we are going to follow them. This is the entire point of the talk. Thank you very much for listening. any noise. This is not a tool that complains by default. Running it on every PR costs you one comment that says look good. That's it. Let's see the high debt PR now. Now you see what you see actually on the screen is from a public repo. This is a real scenario from the scanner regression suite. If you see the score is here 60 out of 100. It requires evidence. It's on the orange it says needs evidence. 86 estimated minutes of review effort. Look at the structured output. It's actually not a score. A wire list explaining what fired. A reviewer focus list telling the reviewer what to do next. An author next action list telling the author how to bring the score down. This is the part most PR quality tool miss. The score alone is useful. The structured advice is what actually moves the team behavior. Now if you read the bullet book, it says PR has soft indicators of AI assisted authorship. This is information only not a definitive claim and not a penalty on its own. This particular sentence is in the scanner output. The AI indicator check contributes 5 of the 60 points which is nearly about 8%. The other 55 came from a different size claim mismatch and missing test. Those would be high burden signals on NAPR. Agent authored or not authored. This is not an anti-AI scorecard. Basically, this is more of a review burden scorecard. The agent did not cause the score, but the shape of the PR that is created by the agent did this. Let's see the next one. This is the PR basically, which is AI authored. It's very well shaped and the score is 7, which is low review burden. Now if you see the same agent, it has done the same kind of work, but the tests were added. So the CI is green. The risky path is called out in the PR description. That's why we got the score as 7 out of 100. It's very low burden, took only 14 minutes. The AI indicator check still fires, but it contributes two points. One, low severity. Second, the framework explicitly says, quoting the report itself, information only, not a definitive claim, and not a penalty on its own. So when your team is going to ask, will this tool penalize us for using coding agents? No. The answer is on your screen, low review burden. It's a very well shaped AI PR. We are going to see the next one, which is basically our three public repos over 524 PRs. I wanted to show you a 90-day AI sub slope across three public repos and 524 pull request. The review burden climbs even when the AI authorship doesn't. If you see this particular scan, we observe three things. One of them is the volume is the actual variable. AI authorship was flat and it was 5-20% steady state, but the review burden was not. If you see one repo accumulated 186 senior reviewer hours in 27 days and another one took 43. The window length is same but the volume was not the same. The burden was not the same. That is what PR volume is. Second is amplifier only holds up under real data which means AI indicators fired on 5-20% of PRs every week across all three repos. None of those AI positioning contract holds against real code basis. Third one is complexity drives burden not authorship which means across 5-24 PRs four landed in needs evidence or high bands which means all four were structural changes which were large migration SDK rewrites multi team refactors the framework score complexity fairly AI driven volume creates the condition under which these accumulate ! where does the review date actually show up in absolute terms the scanner saw 5-24 real pull request in three public repos 12-28 senior review hours accumulated across three public repos that were scanned over 27 to 90 days windows second one is 9 PR per day sustained merge rate at high velocity repos so if you see it all about the volume that is basically driving the burden third one is 5036 these are the minutes that are basically spent on a single PR which is nothing but 84 hours of review effort estimated for one single pull request the score that we received there was 73 which said it requires an evidence spend now 5-20 percent of these PR every week across every week fire the AI indicator signal again remember this is volume which is causing the review debt which is changing the game all together we'll move to the next slide we have shown the cost now what do we do about it so first thing is you should have one logical change per PR I'm not saying like not small PR in the abstract one logical change that is enough second test shift with the change even if the agent wrote the code even if the agent wrote the test the human author confirms the test assert what the code should do not what the code is doing what the code is supposed to do okay third stay in one owner territory cross cutting work splitting into per team PR remain in your own territory one approval one context one mental model author writes the why the agent should not write the PR body that's the moment the human author commits to understanding what they are actually shipping same review standard for the AI PRs as human PRs the AI indicator amplifier only fires when other signals are weak keep the other four strong the amplifier is invisible so make sure there is no exception for AI none of these is actually requiring you to have a new tool these are the moves that you already are aware of you just need to implement it now how are you planning to adopt this if you see we again have five steps back fill ! threshold surface aggregate and talk about it back fill so you run the scorer over your last 200 merge PR look at the one basically that has the highest score second threshold set up basically justify line default let's say you want to give it 50 so any PR scored at 50 or above requires a comment from the author as simple as that surface it post the score as a PR comment on every PR don't block it just for the visibility purpose surface it so that everybody should know what's happening over there aggregate weekly per team which means the slope of each team deadline is the leading indicator that's what your engineering manager should watch talk about it bring the number to a retrospective every roadmap view in every meeting you should discuss about that number like what was your score when you did this particular PR that's why this number matter because of this conversation absolutely not without a number what you are just converging is Y but if you have a number it's actually structured you can tell like okay with AI coding agents we have a rollout which has added X percentage of the throughput but you need to make sure you are also adding the Y points of the review debt that has if no then who is accountable when an AI author change causes an incident where is the audit trail so review debt is basically the bridge between these two columns it's the first number that lets you have the right hand side conversation without abandoning the left hand side gains sinn sinn sinn sinn sinn sinn sinn sinn sinn ! sinn sinn ! sinn over time matters most than the level so plot it weekly and then discuss it with your team maybe in a meeting or in a sprint planning whatever the way you feel like have this conversation bring the number to your next review like say that okay we do have extra put wide it and this is the slope and then you are basically moving away from the vibe culture to the measurement culture one quick call out on anti-patterns to avoid first thing is approved with comment mergers the ai did the boring part no will you catch it in qa no pr are smaller no no stop writing lgtm on the prs make sure whatever we have discussed the habits that we have discussed we are going to follow them this is the entire point of the talk thank you very much for listening