At Abliteration, coding agents help us work on multiple changes in parallel. Each change still needs to be built, tested, and reviewed. As more work happens at once, the time between pushing a change and getting a useful CI result matters more.
We have used Blacksmith to shorten that wait. In a matched evaluation of our GitHub Actions workflows, the Blacksmith configuration reduced median time to required CI feedback from 12 minutes 26 seconds to 3 minutes 35 seconds—about 71% less waiting.
More changes, more checks
An agent's work rarely ends with its first commit. It makes a change, runs checks, reads a failure, and tries again. Several agents doing that across several pull requests create repeated demand for builds and tests. A slow check delays the next correction on every attempt.
That changes what we need from CI. Fast code generation is useful when validation can keep up. We want feedback soon enough for an agent or engineer to act on it while the change is still in context, with the required checks intact.
What we measured
We compared our GitHub-hosted and Blacksmith runner configurations using the same source revision, Node runtime, worker settings, and required test coverage. We collected four matched pairs of cold runs. The measurement ended when the required checks had succeeded and their reports were available.
| Runner configuration | Median required CI feedback |
|---|---|
| GitHub-hosted | 12m 26s |
| Blacksmith | 3m 35s |
That is roughly nine minutes less waiting per validation cycle at the median. The reduction across the four individual pairs ranged from about 67% to 72%. The result compares the runner configurations we tested; it is not a universal speed claim for every repository or job.
The timing covers required CI feedback. It does not include completion of the separate advisory browser suite, measure Production deployment time, or establish long-term reliability. We also keep cold-run results separate from warm-cache experiments and other workflow optimizations.
Faster feedback for agents and engineers
Shorter validation cycles let agents respond to failures sooner and give reviewers results earlier. With several changes in flight, that reduces the time each one spends waiting for a decision about what to do next. It also makes repeated verification more practical when a change needs several corrections.
Blacksmith works with GitHub Actions, so we could evaluate the runner change within our existing workflow. We checked that the required coverage was preserved alongside the timing results. A faster run is useful only if it still performs the validation the change needs.
Choosing where speed matters
We evaluate runner choices by both time to feedback and total cost. A shorter critical path does not automatically mean a smaller bill: parallel jobs, retries, and a growing test suite all consume compute. The right choice depends on the workload and how much waiting it removes.
Our experience with Blacksmith gave us a measured way to make that tradeoff. As coding agents take on more development work, we are applying the same standard to CI: preserve the checks, compare equivalent work, and measure how quickly the team can act on the result.