All insights

Article

What a Six-Week Pilot Proves, and What It Cannot

A pilot is only useful if it is designed to answer questions you would otherwise guess at. Most AI pilots in software delivery are demonstrations: someone shows a plausible change, the room is impressed, and nobody knows any more than they did before about whether it works on their own backlog.

Ours is six weeks, on one product, paid, at a fixed fee credited against setup. The limits, the ceilings and the logs are all on from day one, because a pilot with the safeguards removed is not testing the thing you would buy. Three organisations run it today, 678 stories delivered.

The four questions it answers

How many stories cleared. Yours, on your product, with your Product Owners writing the stories. This is the throughput question and it is the one your board will ask.

What each story cost in AI usage. Because every story carries a cost ceiling and every action is logged with its cost, you end the pilot with an average rather than an estimate.

How much reviewer time it consumed. This is the number most pilots omit and it decides the case. If the platform creates review work faster than your reviewers can absorb it, throughput has not improved, it has moved.

Whether your reviewers were comfortable. Not whether the code compiled. Whether the people who have to sign their name to a change felt able to understand what they were approving.

Why week three matters

Installation takes two weeks. The first story runs in week three. That schedule is deliberate: it means weeks three to six are spent on delivery rather than on configuration, and it means the pilot measures steady state rather than a first-day novelty.

What the pilot does not tell you

It does not tell you what happens across five products at once. Separation is enforced by identity, queue and network rules per product, so the model scales, but your review capacity and your Product Owner capacity are the constraints, and a single-product pilot will not surface them.

It does not tell you how the platform behaves on a codebase with no tests. If your existing code is untested, the first weeks go into building a testing habit around the stories, and that is real work before it is a gain.

It does not replace a security review. Your security team should be in the room in week one, reading the configuration rather than the brochure.

What the limits do during a pilot

They are the reason the pilot is worth running at all. The AI never touches production, so nothing in the pilot can reach a customer. It never merges its own work, so every change in the pilot has a named human approval. It never changes the approved tests, so the test evidence you review at the end is the test evidence that was agreed at the start.

The full set of seven is enforced during the pilot rather than held back until payment. Every action is logged against its story, with its cost, and at the end the story log is the pilot report.

What the reviewers actually see

During the pilot a reviewer is not asked to trust anything. For each story they see the acceptance criteria, the tests that were approved before implementation, the results of those tests, a working preview running against its own test database, and the record of what the AI did with its cost.

That is the difference between evaluating a tool and evaluating a process. If your reviewers spend the pilot unable to explain what they approved, the pilot has told you something important regardless of how fast the queue moved.

What happens at the end of six weeks

You get a product with real stories in it, a story log per change, a set of approvals with names attached, and four measured answers. The decision that follows is yours: continue to setup and a monthly subscription per product, or stop. If you stop, you keep your code, repository, logs and test records, our right to run the platform ends, and we remove it within 30 days.

How to run it well

Pick one product with real work and a real user, not a toy. Pick stories that are genuinely specified, because the platform is only as good as the acceptance criteria it is given. Name the reviewers before the pilot starts and protect their time. Agree in advance what number would make you continue and what number would make you stop.

That last point is the one most organisations skip, and it is why pilots drift into renewals nobody argued for. Who does the reviewing is the constraint worth resourcing before you start, and the financial case is written for whoever signs it off.

What to do next

The 30-minute assessment is where you decide whether a pilot is worth six weeks of your team's attention. Bring the product, the stories and the reviewers.

What a Six-Week Pilot Proves, and What It Cannot | Gyzer