How to run an AI agent pilot without burning $20K

A firm reaches out. They have read about AI agents. They know competitors are experimenting. They want to try one but they are not sure where to start and they are worried about spending serious money on something that might not work. If you are in that position, this note is for you. It describes how to run a 14-day AI agent pilot that produces a real decision instead of a vague impression.

Why most AI agent pilots fail before they start

Most pilots fail for one reason: there is no measurement plan. A vendor sets up a demo, runs some sample outputs, and asks the firm whether it "feels right." The firm says yes because the demo looked impressive. Three months and twenty thousand dollars later, nobody can say whether the agent did anything useful.

A 2026 analysis of over 800 agent deployments found that 76 percent experienced critical failures within weeks. The surprising finding was not the failure rate. It was that the model quality was rarely the root cause. The failures were upstream: no defined success criteria, no baseline measurement, no decision framework. The technology worked. The evaluation did not.

Checkpoint Zero: pick the one number

Before you build anything, name exactly one number the pilot will move. Not three numbers. Not a dashboard. One. Examples: "increase dormant client reactivation rate from 4 percent to 8 percent." "Reduce the time between a new lead entering the pipeline and the first personal follow-up from 72 hours to 4 hours." "Increase repeat customer rate from 22 percent to 28 percent."

The number must be measurable with data you already have. If you need to build a tracking system first, that is a separate project. The pilot measures what the agent does against what is happening now. You need the "now" number before the agent starts.

Checkpoint One: scope one narrow workflow

The biggest mistake in agent pilots is scope. Firms want the agent to "handle marketing" or "manage client relationships." Those are not workflows. They are departments. A pilot needs one narrow workflow with clear inputs and outputs.

Good pilot workflows: reactivate dormant clients who have not purchased in 12 months. Draft follow-up emails after a discovery call, personalized to the conversation. Identify which current clients are most likely to need a specific service based on purchase history. Bad pilot workflows: "improve our marketing," "automate our sales process," "make our CRM smarter."

• One workflow, not three.
• Clear input: the data the agent reads at the start.
• Clear output: the artifact the agent produces at the end.
• Human review gate: a person checks the output before it goes anywhere.
• Measurable outcome: the output connects directly to the one number from Checkpoint Zero.

Checkpoint Two: run the holdout measurement

Days 1 through 7 are the measurement period. The agent runs on a defined subset of clients, leads, or accounts. At the same time, you track what happens with an equivalent group that does not receive the agent output. This is the holdout. Without it, you cannot separate the agent effect from everything else that changed during the week.

A clean holdout does not need to be statistically perfect. It needs to be honest. Do not cherry-pick the best accounts for the agent group and leave the worst for the holdout. Split evenly. If the agent moves the number when the holdout does not, you have a signal. If both groups move the same way, the agent did not cause the change.

Checkpoint Three: the scorecard, not the feeling

Day 14 is decision day. You review a scorecard with four questions. Did the agent move the one number? Did it stay within acceptable error bounds? Did the human reviewer spend less time on the workflow than before? Did the output quality meet the firm standard? Four yes answers means go. Three yes answers means go with adjustments. Two or fewer means stop.

The scorecard matters because it removes the most expensive variable in any evaluation: the impression. Without it, the decision comes down to whether the demo was exciting and whether the person running it was likeable. Those are not good reasons to spend twenty thousand dollars.

Red flags that mean stop now

Some pilots should stop early. If the agent cannot produce output that is safe to review, stop. If the data the agent needs is spread across five systems and nobody can consolidate it, stop and fix the data first. If the workflow the firm picked is actually broken as a human process and cannot be documented, stop. Automating a broken process produces broken output faster.

The most common early-stop reason we see: the firm realizes during Checkpoint One that they cannot describe the workflow they want to automate. That is useful information. It means the workflow needs to be defined and practiced by humans before an agent can run it. That is not a failure. It is the right discovery at the right time.

What happens after a go decision

If the pilot returns four yes answers, the next step is not "scale everything." It is to lock the workflow, define the error budget, and expand the agent scope by one adjacent workflow. The error budget is critical. It is the percentage of agent outputs that can need human correction before you consider the agent broken. For most workflows, 5 to 10 percent is a reasonable starting budget.

The second workflow should be adjacent to the first. If the pilot automated dormant client reactivation, the next workflow might be new lead qualification using the same customer data and the same voice rules. Adjacent workflows reuse context, which makes the second agent cheaper and faster to build than the first.

A short pilot with a clear measurement plan protects you from the two most expensive mistakes in AI adoption: buying something that does not work, and waiting so long to evaluate that you have already spent the budget before you know. The framework here works for any vendor. If you want to discuss running one with a custom growth operator built specifically for your firm, the Growth Audit Call is the place to start.