110 agents. One repository.

Fourteen of 69

Half an hour into the first run, about 110 agents had opened 69 pull requests in one repository on our hosted demo. Each one touched only documentation and carried a passing check, which under the repository’s policy was enough to land without a person. We stopped the load, gave them 15 more minutes and watched the count. Fourteen landed.

The other 55 hadn’t been refused. Each had opened, passed its check and then sat there until the clock ran out. Finding out what held them up took the rest of October 15 and ran past midnight UTC, and the answer kept changing under us. Along the way we took our own demo down and found three bugs that only show up at agent speed.

The line people never see

Source control grew up around people. A developer pushes a few times a day, opens a pull request or two and waits hours for a review. At that pace a forge can send every write in an installation through one coordination point, and the line in front of it stays so short that nobody knows it’s there.

Agents live in that line. An agent commits, opens a pull request, waits for its check, lands and starts again a minute later, while a hundred others do the same in the same repository. Golf on Cloudflare was built around one such point: a stateless Worker hands each request to a single Durable Object per installation, the Workspace, which admits requests through a queue and coordinates writes. D1 holds the records and Artifacts holds the Git data.

Five load processes played the agents against a test repository. Thirty ran the whole loop: commit through the Source API to a new branch, open a pull request over MCP, report the check and wait to land. The other 80 or so committed, pushed and cloned around them. Each paused 30 to 90 seconds between operations, so for 30 minutes roughly a hundred writers each touched the repository about once a minute. Golf never lands a commit that no check has reported on, and we counted landings, because landing is where an agent’s work ends.

Waiting for the sweep

The first suspect for the 55 was landing itself. Golf landed pull requests only in its maintenance sweep, every five minutes and at most 20 a pass, and a passing check triggered nothing. On an idle system one pull request landed 193 seconds after its check passed. Under load, agents opened pull requests faster than the sweep could clear them.

Merges made it worse. Every landing is a merge, and a merge reserved the whole repository for a few seconds while other writers failed instead of waiting. Pushes were rejected and 14 to 27 percent of commits were refused.

Both choices hold up when people land a few changes a day. By 14:56 UTC we had replaced them. Golf now evaluates a pull request the moment its check is reported or it opens, with the sweep as a backstop, and writers behind a merge wait their turn: a commit within its 30-second budget, a push for up to 20 seconds. Once our test harness reported checks reliably, a single pull request landed 5.4 seconds after it opened. That looked like the answer to the 55.

Landing faster slowed everything else

Under the full mix again, 77 of 105 agent pull requests landed within 15 minutes, at a median of 64 seconds. Almost everything else got slower.

When a person’s change lands, their next one is usually hours away. When an agent’s lands, it starts the next within a minute or two. Agents that landed sooner came back sooner, with 105 pull requests and 203 commits against 69 and 71 before. Clones timed out 62 times in 68, the median push took 100.9 seconds, and successful writes fell from 0.75 a second to 0.54. We had turned refusals into waiting without adding capacity, so the extra work went straight to the next limit. That kept happening all day: each fix moved the bottleneck.

Twenty-eight pull requests still hadn’t landed, the rest of the forge was crawling, and the obvious suspect was the processor.

Forty-seven seconds in line

The Workspace is one single-threaded object that built packs and merged trees for the whole installation. To see whether it had run out of processor, we started recording each response’s Server-Timing header, which splits a request’s time into waiting for admission to the Workspace and doing the work.

The first numbers seemed to settle it. Admission wait was close to zero, so requests got in at once and then ran slowly, the way they would on a starved processor. We wrote that down as the finding. Then we looked at whole headers. A response carries two queue stages, and our parser had picked up the small one, a nested request’s own few tens of milliseconds. We corrected it at 16:40 UTC.

Three Source API commits sampled with full headers told a different story. They waited 14.5, 31.4 and 47.4 seconds for admission, then did 0.95 to 1.26 seconds of work, with D1 taking 0.3 to 0.6 seconds of that and Artifacts 0.2 to 0.5. The slowest one spent about 50 times longer in line than at work.

Cloudflare’s analytics agreed. Between 16:20 and 16:56 UTC the Workspace used 1.3 to 2.9 seconds of CPU a minute, about 4 percent of one core, while staying busy all 60 seconds of every minute. It spent that time waiting on I/O, so a faster processor would have had almost nothing to speed up.

The admission rules explained the rest. Up to 8 reads and 24 Git transfers could run at once. Anything exclusive, such as opening a pull request or reporting a check, waited until every running read and transfer had finished, and nothing behind it was admitted in the meantime. Clones and pushes ran for tens of seconds under load and exclusive requests arrived every few seconds, so each one started a convoy. Every agent cycle sent two, which is why faster landings had made everything else worse. The answer to the 55 had moved from the sweep to this line.

The shortcut that took the demo down

The cheapest fix sat right there in the rules. Many of the exclusive writes didn’t need the repository to themselves, because D1 already ordered them with its own reservations, so we let them overlap like transfers. The change passed all 722 forge tests. On the hosted demo, a single pull request landed 5.8 seconds after it opened. We started the full mix.

Eight minutes in, the whole demo stopped answering. Source API commits reset or timed out at 60 to 90 seconds, and the home page, which has nothing to do with the test repository, timed out at 20. We stopped the load at 17:50 UTC. At 17:52 the demo answered again, slowly, with the home page taking 9.6 seconds. At 17:53 we rolled the Worker back, and the home page answered in 0.26 seconds.

Slow as it was, the convoy had been the backpressure that kept the Workspace alive. With it gone, every long wait piled into one object that is single-threaded, has 128 MB of memory, can’t grow and serves every repository in the installation. The line we had been trying to shorten was also holding the object up. Another single object would hit the same wall, so the capacity had to come from something that spreads out.

Moving the work out of line

The Worker in front of the Workspace is stateless and scales horizontally, and correctness had never depended on the Workspace’s queue. D1 already ordered every write with a branch reservation, a journal of merges and guarded inserts. So the Workspace kept activity, webhooks, automations and alarms, and the long, I/O-bound work moved to the Worker.

We moved it in steps between 18:27 and 21:17 UTC, measuring each at about 30, 60 and 110 agents. Clones and fetches went first. At about 30 agents the median clone dropped to 2.7 seconds, while pushes, still in line, sat at 21.9. Pushes, Source API commits and branch updates went next, and the median push fell to 1.8 seconds. Writes carry the caller’s grant in their D1 reservation and reads check it again after storage answers, so revoking access still works without the queue.

MCP came last. Agents waiting to land poll their pull request, and at about 60 agents that polling was most of the Workspace’s load: 871 polls against 23 opened pull requests. Once the Worker served MCP reads, opening a pull request waited 1.6 seconds at the median instead of 18.6.

Back at about 110 agents, a probe asking the Workspace for an exclusive slot every 20 seconds waited 0 ms at the median.

Pull requests that never landed

With the line gone, more pull requests reached landing, and landing went the wrong way. At about 30 agents, 16 of 23 landed, then 11 of 18. Three bugs were behind it, and each needed the agents’ speed or the repository’s size to show itself. Two of them left no trace at all, only pull requests with passing checks that stayed open.

Eighteen minutes with a passing check

At about 60 agents, 15 of 23 pull requests timed out with passing checks. The policy allowed them, and they stayed open while others landed around them. To land a branch on a base that has moved by a merge, Golf compares the two tips, and it read up to 500 commits of each one’s history. The test repository had grown past that. For every pull request whose base had moved, the evaluation ran out of history and the landing step skipped it without a trace. Golf now finds the merge base with Git’s own date-ordered walk, bounded at 2,000 commits. One pull request that had waited 18 minutes with a passing check landed at the first sweep after the fix went out.

Four minutes of nothing

At about 110 agents, the next run broke where we could see it. Pushes failed 233 of 325, refused because the repository was finishing a merge, and D1 showed writes arriving in bursts with four-minute gaps between them. For minutes at a time, every write to the repository stalled.

The gaps traced back to storage. Under this load Artifacts answered some merges with “busy” before running any command. Golf’s merge journal recorded each one as an unknown outcome, which fenced the whole repository until maintenance settled it two to seven minutes later. Golf now settles a busy answer as refused at once, and the landing retries. In the next run the provider still said busy twice in the first eight minutes, and writes kept going at 48 to 67 a minute, every minute.

Ranked 206 to 326

By late evening every Git and Source API operation at about 60 agents succeeded, and landing still stopped at 38 of 49. Only the landed count disagreed with the latency charts. Just before midnight we ran 16 agents through the loop alone for five minutes to look closer. Thirty-six of 48 landed. The other 12 had passing checks, and Golf’s own landing evaluation, asked live, said land. None of them had a land operation, because Golf had never tried to land them.

The fast landing step began by listing open pull requests in the sweep’s fixed order, capped at 200. The repository had 359 open, most of them stale from earlier runs, and every stuck pull request ranked between 206 and 326. They were waiting in a line we hadn’t known existed, past the point where the landing step stopped reading. That step now starts from the branches at the commit whose check just passed. We deployed it and ran the loop again.

Seventy-one of 71

At 16 agents running the loop alone, all 71 pull requests landed, at a median of 5.9 seconds and 11.5 at p95. The run ended a little after midnight UTC, about nine and a half hours after the first fix went out.

The run at about 60 agents just before that fix shows the rest of the forge: pushes went 88 of 88 at a median of 1.6 seconds, Source API commits 272 of 272 and pull requests opened 49 of 49, and every clone and fetch succeeded. The only refusals were two pushes that lost a race on a shared branch and were told so, which is Git doing its job.

Where the agents wait next

So what held up the other 55? At first a sweep that ran every five minutes, then a line in front of one object, then limits nobody had reached before: a history read that stopped at 500 commits, a “busy” from storage that froze the repository for minutes, and a list that stopped at 200. By a little after midnight, for 71 pull requests in a row, nothing held them up.

People never notice the line in front of a forge, because they don’t stand in it long. Agents stand in it all day, and at their speed you find out exactly where your system waits, even when it looks healthy: our coordination object sat at about 4 percent of a core while commits waited 47 seconds to reach it. Every limit we removed showed us the next one, and we expect that to keep happening.

An agent pull request could land in 5.9 seconds only because a person had decided in advance what may land that way: in the test repository, documentation changes, and only after the required check passed. People still decide what lands. Writing that decision down once, as policy, is what makes the speed usable, because nobody has to stand over each merge.

We’re already going after the next limits. The last full run at about 110 agents came before the landing fixes, left 38 of 149 pull requests unlanded and lost its 30 Source API committers to a failed setup fetch, so first we run the whole mix again with everything deployed. Then comes the distance to the database. Each D1 call took about 100 ms from the Worker against about 20 ms from the Workspace, which runs next to the primary, and Smart Placement for the Worker should win most of that back. The Worker now caches Git objects, which removed most of its reads from storage, and we want to see that as fewer billed storage operations.

CI has to keep pace with the agents too. Runner capacity in Golf now grows by request and approval, with every change audited, and we want a run that shows CI keeping up at a newly approved limit. We want names for the rare failures we still can’t explain, like the few connection resets and the pushes that ended in a sideband disconnect. A coordination object per repository stays in the plan for the day the data asks for one, which so far it hasn’t. After all that comes a bigger run.

All of this happened in one repository on our hosted demo, with scripted agents on one workstation reporting a check that always passes, and Golf is alpha, with breaking changes still to come. We’ll publish the next numbers here the way we published these. The day started with 14 of 69 and a line nobody had looked at. It ended with 71 of 71 and a longer list of places to look than we had at the start, and the next run picks up from the top of that list.