When AI Reviewers Become a CI Problem

The AI Reviewer Proliferation Problem

Adding an AI reviewer to your repository takes five minutes. Adding the fifth one happens without anyone noticing. Then one day a maintainer realizes the project has quietly acquired a dozen automated voices on every pull request - and the team's real work has shifted from reviewing code to reviewing the reviewers. The failure modes are predictable. A semantic bot with merge‑blocking authority turns a false positive into a wall. Review‑driven churn sets in: code gets shaped to silence the loudest bot instead of to serve the reader. Configuration drifts between three vendor dashboards and two repo files until nobody knows which one is real. And every pull_request_target workflow that checks out PR code to "run the checks" is a supply‑chain incident waiting for a headline.

WorldScript Studio , an open-source writing app with an unusually dense reviewer setup - fifteen automated providers at last count - had to solve this deliberately. The answer is a governance layer with four mechanisms. Code references are from the repository at commit 99024a4b (2026‑09‑28), release v1.28.8; simplified excerpts are labeled.

1. A registry that says what each bot is for

The repository keeps a machine‑readable registry of every automated reviewer - not a wiki page, a JSON file with a schema version and an authority pointer: // config/reviewer-registry.json (excerpt - one of fifteen entries)

{ "id" : "coderabbit" , "role" : "semantic-ai-review" , "repoConfig" : ".coderabbit.yaml" , "configurationOwner" : "repository" , "mayMutateBranch" : false , "blockingAuthority" : false , "statusSource" : "live" }

Two fields carry the whole philosophy. blockingAuthority: false holds for every semantic‑AI reviewer in the registry. Of the fifteen registered providers, exactly four carry blocking power - CodeQL, OSV, GitGuardian, Socket - all deterministic security gates. An LLM's opinion about your variable naming can be valuable; it must never be a merge wall, because its error mode is confident nonsense. And mayMutateBranch: false for every entry: no bot writes to your branch, ever. Reviewers advise. Deterministic security gates block. Nothing else gets either power.

2. One hierarchy of truth

Fifteen vendors means fifteen dashboards, each happily showing "configured." The governance doc ends the ambiguity with an explicit authority order: executable repository policy, CI workflows, hooks and scripts first; then the agent/contributor rules; then the reviewer‑governance doc; then vendor‑specific repo adapters; then vendor dashboards; and dead last, historical audits and incident records. The load‑bearing consequence: a vendor adapter file like .coderabbit.yaml is a narrow implementation of repo policy, not a competing policy document. When a dashboard disagrees with the repo, the repo wins - and "the dashboard said so" stops being an argument.

3. The trust boundary is enforced twice

The subtle part of CI governance: the reviewer‑policy check itself runs on untrusted input. A PR that can modify the checker has already won. So the gate comes in two layers. The ordinary pull_request workflow materializes the complete base workspace and runs the base‑ref copies of the policy checkers against the PR tree as data. On top of that sits a base‑owned pull_request_target guard whose entire design is distrust: # .github/workflows/reviewer-governance-trust.yml (excerpt)

on:
  # read from the base ref; never executes the PR workspace
  pull_request_target:
    types: [ opened, synchronize, reopened, ready_for_review, edited ]
  permissions:
    contents: read

It checks out the trusted base (SHA‑pinned actions, persist-credentials: false), fetches the PR head purely as a git ref, verifies the fetched SHA matches the event SHA, fails if the protected policy files differ between base and head, and even pins the expected hash of the admission‑policy checker itself. The second layer exists so no later PR can delete or neutralize the invocation step in the first one. That is what "defense in depth" looks like when the threat model includes your own automation surface.

4. Quiescence is a defined state, not a feeling

When is a PR done with review? Without a definition, the answer is "when someone gets tired." The repo defines it: a PR is review‑quiescent only when both reconciliation loops are satisfied - the CodeAnt correction loop (fetch threads, validate each finding, fix or justify, reply citing the commit, then resolve) and the token‑free DeepSource loop. Bot channels are treated as observed, not assumed : the docs explicitly note which bot actually posts inline threads versus status checks, and instruct agents to inspect actual output per PR instead of following a fixed mental roster. The point is not ceremony. It is that "all review findings are either merged, justified, or answered" is a checkable property - and until it holds,