Introduction

AI coding agents can now open more pull requests in a single day than many engineering teams can review in an entire week. An AI software factory is designed to turn that surge of machine-generated code into shippable features, but in practice it simply relocates the bottleneck from writing code to verifying it. This guide examines the eight essential building blocks of an AI software factory, the tools that support each one, and what breaks when a block is missing.

What Happened

The industry has moved toward systems where multiple coding agents run in parallel loops, each pulling from a queue of specifications and tickets. According to a widely cited definition by Addy Osmani, a loop is a single agent that repeats a cycle: it gathers context, makes a change, checks the outcome, and begins again. A harness wraps that loop with sandboxes, tools, memory, and completion rules. A factory consists of many harnessed loops operating at once under human oversight.

Dan Shapiro's five levels of autonomy, published in early 2026, offer a shared scale ranging from "spicy autocomplete" to environments where nobody reads the code. Osmani further distinguishes between light factories, where humans remain involved in design and review, and dark factories, where code ships without any human reading it and only machines verify the result.

The eight building blocks include work queues and specs, coding agents with harnesses, rules files and MCP servers, isolated PR builds, unit tests and CI, running-app verification, risk-tiered review gates, and rollback with production signals. Leaving any one out creates a specific failure mode, from agents building the wrong thing quickly to broken code merging undetected.

Why This Matters

Code generation now scales with compute, but verification does not. The amount of autonomy a team can safely grant its agents depends directly on how well it can verify the running application from a user's perspective. Unit tests and AI review bots examine the code, but they cannot confirm that a customer journey actually works.

Consider a redesigned checkout flow. Unit tests pass, a review bot leaves style comments, and continuous integration is green. Yet on the preview deployment, a new layout pushes the saved-card selector below a sticky banner, blocking returning customers from completing payment. Nothing in the code diff looks wrong, but the product is broken.

This is where running-app verification becomes essential. AI QA agents can interact with a live web or mobile build like real users, pursuing goals such as "pay with a saved card and see the confirmation." They can also check email and SMS or place API calls during a test, verifying that a receipt email actually arrives. Verification happens both before a pull request exists—inside tools like Claude Code, Cursor, or Codex via an MCP server—and on the pull request itself against a preview deployment.

A fully dark factory is rarely the right goal. Osmani warns about "comprehension debt," the widening gap between the volume of code produced and how much any human still understands. Regulated domains, legacy codebases, and products where usability matters more than raw throughput are generally better suited to a light factory model, where people retain responsibility for judgment while agents handle repetitive checks.

Key Takeaways

  • Work queue and specs: Turns intent into tasks agents can build. Without it, agents build the wrong thing quickly.
  • Coding agents and harness: Writes changes within defined limits. Without it, loops wander, stall, or edit unintended areas.
  • Rules files and MCP servers: Provides agent context and boundaries. Without it, the same mistakes repeat on every pull request.
  • Isolated PR builds: Gives every change its own running build. Without it, there is nothing real to verify before merging.
  • Unit tests, CI, static analysis, and review bots: Catches code-level issues early. Without them, broken code gets merged.
  • Running-app verification: Checks user journeys in the live application. Without it, correct code can ship a broken product.
  • Risk-tiered review gate: Routes changes through the least expensive safe review. Without it, reviewers get overwhelmed or rubber-stamp changes.
  • Rollback and production signals: Undoes bad changes and feeds incidents back into the queue. Without it, escaped bugs pile up unnoticed.

Teams should build these blocks in order of what each one unlocks. Start by giving every pull request a preview deployment, verify the running app on every pull request using a tool that did not write the change, add risk tiers to the review gate, and raise agent autonomy only as fast as verification can keep up.

Conclusion

An AI software factory is only as strong as its weakest verification block. Generating code is no longer the constraint; checking that the product works is. By treating running-app verification as a first-class station in the factory—rather than an afterthought—engineering teams can safely increase agent autonomy without shipping broken experiences to customers.

Whether you operate a light factory with human oversight or move toward darker automation, the principle remains the same: agents can draft the code, but independent verification must decide whether it is ready to ship.