How to Build a Multi-Agent Review Swarm

At-a-Glance

• Verdict: Watch fully
• Core thesis: A coding agent becomes materially more useful when it operates from the company’s canonical context, executes remotely, and is governed by a multi-model, domain-specific review loop that it must resolve before opening a PR.
• Why it matters: The video provides a concrete control-plane pattern for turning planning artifacts, codebase access, CI feedback, and code review into one asynchronous agent workflow rather than a sequence of manual handoffs.
• Best use: Use it as a reference architecture for an agentic software-delivery loop: spec-to-task generation, remote execution, structured work journaling, CI remediation, and swarm-based quality review.

Executive Summary

A Notion AI engineer describes an internal workflow in which Notion is not merely a documentation layer but the primary interface and context store for coding agents. He starts tasks through unstructured voice input, asks the agent to explore the codebase through a remote development VM, and has it turn the resulting understanding into a detailed task with code pointers, requirements, and self-verification criteria.

The implementation agent then receives the linked task, builds the change in a cloud-hosted coding environment, maintains notes as it works, opens a PR, and monitors CI. In the demonstrated case, it started at 12:53 and completed at 2:27 while the engineer attended meetings and initiated other work; the stated goal is not one-shot perfection but removing execution busywork after careful human planning and constraint-setting.

The most reusable technical idea is the “review swarm” skill. It partitions a change set into domains such as front end and back end, assigns separate correctness and maintainability reviewers for each domain, runs both GPT and Opus over those reviews, aggregates findings through a top-level agent, and loops the implementation agent until the resulting issues are addressed.

The broader operating lesson is that consolidation matters: connecting the agent to the canonical task system, company knowledge, codebase, remote compute, and CI reduces context drift and app-switching. The speaker also argues that teams should build bespoke agent skills around their own standards rather than wait for an off-the-shelf workflow to fit.

Key Takeaways

• Claim: Treating the work-management system as the agent’s native operating surface can turn organizational context into executable engineering tasks. | Evidence: The agent is asked to investigate a missing model-picker migration using access to the codebase, then creates a task in Notion’s M2 milestone project containing code pointers, requirements, and self-verification steps. | Implication: Ken should design agents to write back structured plans, provenance, and acceptance checks into the same system where work is assigned, rather than leaving analysis in ephemeral chat threads. | Caveat: This depends on the task system having reliable permissions, accurate source material, and tool access to both the relevant workspace context and engineering environment.
• Claim: Voice-first, free-form task intake can yield richer agent instructions than tightly edited prompts because it captures reversals, nuance, and contextual detail. | Evidence: The speaker begins tasks by “yapping” through the situation and says models tolerate, and may benefit from, corrections such as “actually, never mind, do that instead.” | Implication: Use conversational intake for discovery and context capture, but have a planning agent normalize it into an executable task contract before an implementation agent acts. | Caveat: Free-form intake still needs a later conversion into explicit requirements and verification criteria before autonomous implementation.
• Claim: Remote, preconfigured compute is a key enabler of asynchronous coding agents because it eliminates local-environment and repository-state friction. | Evidence: Notion AI can invoke a computer; instead of a bare Vercel sandbox, the workflow uses internal “Boxy” coding VMs with remote compute on demand and codebase access. The speaker contrasts this with managing whether a local repo, Codex installation, and environment are current and correctly connected. | Implication: For Ken’s agent systems, standardized ephemeral or remote workspaces should be treated as infrastructure, with deliberate environment bootstrap, scoped credentials, and auditability rather than ad hoc local-machine dependence. | Caveat: The transcript does not address isolation boundaries, credential handling, or how remote VMs are prevented from accessing unintended production resources.
• Claim: The human’s highest-value role shifts toward planning, constraints, and tool provisioning, while the agent absorbs implementation and remediation busywork. | Evidence: After the engineer supplies the task and a review-swarm instruction, the agent builds the feature, opens a PR, and fixes CI issues autonomously. The demo reports a 12:53 start and 2:27 completion while the engineer was in meetings; the speaker explicitly says the value was not simply asking it to make no mistakes. | Implication: Measure agent leverage by reduction in engineer coordination and repair time after planning—not by one-shot completion claims—and retain human responsibility for scope, acceptance criteria, and release decisions. | Caveat: The example is a single successful internal change and does not establish a general reliability rate, review quality benchmark, or safe autonomy threshold.
• Claim: A review swarm should separate correctness review from maintainability review, because bug detection alone does not protect the long-term architecture of large changes. | Evidence: The speaker’s skill slices a change set into domains such as front end and back end, then delegates to a correctness reviewer for bugs and a maintainability reviewer for reuse, existing patterns, and designs that could fail at scale. | Implication: Ken should encode architectural standards as a distinct review lane, not bury them in generic code review; domain decomposition also makes reviewer prompts and ownership more precise. | Caveat: The transcript does not specify reviewer prompts, severity thresholds, false-positive handling, or who adjudicates conflicts among reviewers.
• Claim: Using multiple model families and a top-level synthesis agent creates a practical ensemble review loop rather than relying on a single model’s judgment. | Evidence: For every domain slice, the workflow runs both GPT and Opus; a top-level agent collects their findings into an action report, and the implementation agent iterates until the findings are resolved. | Implication: Build review orchestration around structured findings, deduplication, severity ranking, and termination criteria, rather than simply increasing the number of agents or model calls. | Caveat: Multiple-model review may increase cost, latency, and correlated false positives; “until things are good” requires an explicit stopping rule in production use.
• Claim: Agents should maintain a living work journal while executing, making the task system both the specification source and the operational record. | Evidence: A review-swarm skill instructs the agent to take notes, and the speaker says Notion is updated with what the agent found, issues it encountered, and progress during feature construction. | Implication: Require agents to emit durable intermediate artifacts—decisions, discovered dependencies, failed attempts, and validation results—so humans and subsequent agents can audit or resume work without reconstructing context.

Detailed Brief

Workflow design: from exploratory request to PR

• Claims: The demonstrated task began as an investigation of how a newly built model picker worked and how to migrate it into an alpha version of the product.; The agent can use shell and computer tools for work that exceeds its native tool set, including exploring code and generating files.; The workflow is intentionally asynchronous: the engineer can close the tab after dispatching a task and return to completed artifacts later.; The speaker’s personal task-management practice is minimal: a canonical company task board exists, but he tracks active work largely through pinned chats and focuses on one task at a time.
• Evidence: The prior workflow used Codex connected to a repository, producing a Markdown file or using a Notion MCP integration to create a task; the native Notion workflow removes those app transitions.; The speaker describes Vercel sandboxes as useful but bare-bones compared with the internal prebuilt coding VM environment.; The agent’s output is positioned as a prebuilt task ready for implementation, rather than a vague summary of its codebase exploration.
• Caveats: The presentation is a product-adjacent internal demo and does not show failed runs, task rejection, PR review by humans, deployment, rollback, or security controls.; The transcript repeats its final review-swarm segment, so it contains less distinct material than its stated word count suggests.
• Implications: A robust agent workflow needs explicit state transitions across planning, execution, review, CI repair, and handoff; a chat interface alone is insufficient.; The canonical workspace can function as an operational memory layer when agent outputs are captured as tasks and journals rather than isolated conversations.

Custom skills as organizational policy

• Claims: The speaker frames agent skills as highly malleable organizational tools rather than fixed vendor features.; His review swarm was created by asking Codex to analyze existing skills, preserve what he liked, reject what he did not, and build a new workflow around his own priorities.
• Evidence: The stated priorities include maintainable feature and infrastructure changes, reuse of existing code patterns, and prevention of designs that may break under scale.; The speaker calls the resulting workflow the best review-loop flow he has personally experienced.
• Caveats: A generated skill should be tested against representative historical changes before it becomes a required quality gate; the transcript offers no evaluation methodology.
• Implications: Ken can treat skills/prompts as versioned operating policy: derive them from existing standards, test them against known edge cases, and evolve them where they produce poor findings or miss important risks.

Notable Concepts & Terms

• Review swarm skill: A custom orchestration workflow that decomposes a code change, assigns specialized reviewers, aggregates their findings, and drives implementation iterations.
• Correctness reviewer: A reviewer lane focused on bugs and functional defects in a defined slice of the change set.
• Maintainability reviewer: A reviewer lane focused on reuse, adherence to established patterns, infrastructure quality, and potential scalability failures.
• Boxy: Notion’s internal nickname for remote, on-demand coding VMs that provide a prepared development environment to the agent.
• Canonical source of truth: Notion is positioned as the unified location for specifications, tasks, company knowledge, execution notes, and resulting work records.
• Living work journal: Agent-maintained notes recording discoveries, problems, and progress during implementation, intended to preserve operational context.
• Multi-model review: Running GPT and Opus independently over each domain slice before a higher-level agent synthesizes actionable findings.
• Notion MCP: A prior integration path the speaker used to have Codex create Notion tasks, contrasted with a more native in-product workflow.

Operator Notes / Why Ken Should Care

• Prototype a code-change control loop with explicit artifacts: intake transcript, normalized task contract, codebase findings, implementation journal, reviewer findings, CI status, and PR handoff.
• Define a review matrix for each change domain: correctness, maintainability, security/auth where relevant, test adequacy, and operational impact; do not use a single generic “reviewer” prompt.
• Add a finding schema and stop policy before deploying a swarm: severity, evidence location, recommended remediation, deduplication across models, owner, resolution evidence, and escalation conditions.
• Require remote agent workspaces to use scoped credentials, isolated environments, reproducible setup, and logs of tool actions before granting repository-write or PR-opening privileges.
• Evaluate any custom review skill on a benchmark of prior incidents and accepted PRs to measure missed defects, false-positive burden, model disagreement, latency, and cost.
• Keep a human approval gate for scope expansion, sensitive files, production-impacting changes, and final merge until empirical reliability and governance are established.

Source/Metadata

• Title: How to Build a Multi-Agent Review Swarm
• Transcript words: 2407
• Duration seconds: 721
• Timestamp note: No timestamps or chapter markers were provided. The transcript repeats the final discussion of context consolidation and the review-swarm design.