A pilot is the easiest place to defer governance, and the most expensive place to have deferred it.
Somewhere in your organization, a promising tool is about to be piloted. The word is doing more work than it appears to. "It is just a pilot" lowers procurement scrutiny, postpones the policy conversation, and reassures oversight that nothing has really been decided yet. Everyone relaxes, because a pilot is an experiment, and experiments are reversible.
Except that pilots create facts. The moment a system touches live work, data begins to flow to a vendor, staff begin to form habits around the tool, and — in human services — the tool begins to influence decisions about actual people. None of that waits for the evaluation report. A person affected by an AI-assisted recommendation during a pilot is exactly as affected as they would be after full adoption. The organization's accountability does not shrink to pilot size.
This is not an argument against piloting. Pilots are how responsible organizations learn, and refusing to try anything is its own governance failure. The argument is narrower: the questions most organizations plan to answer after the pilot are precisely the ones that must be settled before it.
First, what decision does the tool influence, and who owns that decision during the pilot? Not the project — the decision. If the tool suggests priority, eligibility, or risk, someone must hold the same accountability for those calls that they held the day before the pilot began. If the answer is "we will see how the tool performs," the tool has quietly become the owner.
A pilot that cannot fail is not a pilot. It is a rollout with better manners.
Second, what leaves the building? Which data goes to the vendor, where it is stored, who can see it, and whether it trains anything. These are contract questions, and a pilot agreement is a contract. The data shared during a six-week trial is not returned when the trial ends.
Third, what does stopping look like? Define, in writing and in advance, what result would end the pilot, and name the person with the authority to end it early. An exit that has to be argued for after staff have built the tool into their day is not an exit; it is a negotiation the tool usually wins.
Fourth, who is told? Staff should know what the tool does, what it cannot do, and what to record when they disagree with it. And if the pilot touches decisions about people, the organization needs a position on what those people are told — a position it can defend later in plain language.
Finally, what is written down? A pilot is an evaluation, and an evaluation needs evidence: where the tool was right, where it was wrong, where staff overrode it and why. If the pilot ends and the only record is enthusiasm, the adoption decision will be made on enthusiasm.
None of this requires a governance department. Most of it fits on two pages, drafted in the week before the pilot starts, by the people who would otherwise draft it under pressure a year later. The organizations that struggle with AI governance are rarely the ones that moved too fast. They are the ones that called the first move an experiment and never noticed when it stopped being one.