Fifteen questions to ask before you greenlight another AI analytics pilot.
Your AI team can build an impressive demo of this. In three weeks, with a good code assistant, you will have scheduled extracts with AI-written commentary. It will work. It will demo well. And then the meeting will end.
What follows is what happens after the meeting. These are not gotcha questions. They are the questions we had to answer when we built this ourselves. A good team will say "we'd have to build that" to most of them. That is the right answer. The follow-up question is whether building it is their day job.
On correctness
When the peer comparison silently breaks and the AI writes "you match your peers exactly" (grammatical, plausible, false): what catches it before a district manager reads it?
This is not a hypothetical. It happened to us. There is now an automated gate for it, and that gate exists because we were burned into building it. Ask what the gate inventory looks like on day one versus after two years of production.
Your fiscal year ends September 30. The system compares week 39 against week 39 last year. One of those weeks contains a public holiday. What corrects for it?
This is the most common way a competent team publishes a confidently wrong number. Arithmetically nothing went wrong. Nothing in a standard pipeline flags it. The error is real, consistent across every location, and points in the same direction. It looks like a trend.
You have a balance-sheet metric alongside flow metrics. What prevents the system from summing four weekly snapshots of a balance into a number four times too large?
Stock-versus-flow semantics have to be baked into the query engine. They cannot be layered on top later. If the team is using Power BI or a rented semantic layer as the query engine, ask how it extends into fiscal calendar awareness and comparable-period pinning.
How does the system know that an 18% February revenue drop is tax-refund season, not a crisis?
The answer has to be versioned configuration: something written down, with a date and a reason, auditable. If the answer is "it's in the prompt," the follow-up question is: what happens when the person who wrote the prompt leaves?
On production volume
The pilot runs across ten locations. What does the architecture look like at four hundred? At fourteen hundred?
Ten locations is a math problem. Fourteen hundred locations at thirty to fifty queries each is sixty thousand query results per cycle. No context window holds that. This is an architecture problem, not a prompting problem, and the naive version of the pilot architecture costs roughly a hundred times more at scale than it costs at pilot. By the time that becomes visible, the pilot has already been declared a success.
When stage four of seven fails at 3am on delivery night because a provider rate-limited you: does the run resume from a checkpoint, or does it start again from the beginning?
The second time the run restarts from the beginning, someone is manually babysitting a pipeline at 3am. That person now has a second job. Ask what the checkpointing and resume logic looks like before the first production failure, not after.
How are inference costs budgeted at scale: per month, per run, or per investigation?
If the answer is per month or per run, the incentive is to run cheaper models everywhere to hit the budget, including where the answer matters. The right architecture screens cheaply and universally, then routes expensive, targeted investigation only where a trend or anomaly justifies it. That routing logic has to be in the system from the start, not retrofitted when the bill arrives.
On knowledge and maintenance
Where does the system's judgment live: what patterns matter, when a finding is a false alarm, how to measure it?
There are two possible answers. One is a prompt: unversioned, unauditable, living in one person's head. The other is versioned configuration, with a change history. Every threshold has a reason and a date. Ask to see the structure of the second before you assume it exists.
When the one person who understands the system leaves, who owns next Tuesday's four hundred briefings?
This is not a people question. It is a documentation question. The knowledge the system runs on has to be externalised into something readable, maintainable, and not dependent on institutional memory.
How does the system handle a metric that changes definition mid-year?
Schema changes are constant in operational data. The system's view of "revenue" in January and "revenue" in August may be measuring different things. Ask how the knowledge layer tracks that, and how it prevents silent drift between what the system thinks it's measuring and what it's actually measuring.
On trust and adoption
In week six, a district manager asks: "is this new advice, or am I retreading week two?" How does the report know?
Actions need identity across cycles. A recommendation that reappears without acknowledging it has appeared before destroys credibility on the second encounter. Managers stop reading reports that don't know their own history, and you do not get a second launch.
A manager declines a recommendation. What happens the following cycle?
If the answer is "it appears again," the system does not respect human judgment, and managers will treat it accordingly. Dispositions (accepted, declined, in progress) have to persist and be reflected in what gets surfaced.
When a recommendation is marked complete and the metric does not improve, what updates?
Progress has to be measured by replaying the exact frozen query that set the baseline. Not by asking a model to recall a number. Models recall numbers fluently and sometimes incorrectly. The baseline has to be a stored result, not a remembered one.
On governance and control
If the system produces a finding your legal team would not want published: what is the gate, and who controls it?
A chart with a wrong number looks wrong. AI prose with a wrong conclusion reads fluent and confident. A field organisation revokes trust after one bad sentence, and it does not come back. Ask what the pre-publish gate inventory looks like, and whether configuration of those gates requires an engineer or an operator.
When you stop using the system, what do you keep?
The configuration (the written record of how your best operators think, what they investigate, what they ignore, and why) should be yours. Ask whether it is exportable, in a format a human can read, independent of the vendor's continued operation.
What to do with the answers.
A good team will say "we'd have to build that" to most of these. That is genuinely the right answer, and it's the answer we'd give.
The follow-up question is what else they were going to do with the next two years.
If you want to see how we answer all fifteen, and what we've shipped on the other side of each of them: talk to us.
Book a discovery call
Reach out to start a two-way conversation about your industry, your performance variations across locations, and whether Scoop is the right fit for you.