Dashboard summaries are easy. Investigation is not.

Brad Peters

Summarizing six metrics is a solved problem. Ask Claude Code for last month's revenue, transaction count, and margin by store, and it returns a clean, formatted answer. That part works.

What is real analytics?

Real analytics is a chain of follow-up questions where each answer decides the next question.

A number moves. You ask why. The answer points somewhere.

You check whether that somewhere is the actual cause or a coincidence. Then you rule out the thing that would make the whole conclusion wrong. Which question comes second depends entirely on what the first one returned.

What is real investigation?

Investigation needs a playbook that is written down, versioned, and identical across every run. Claude Code does not have one. It improvises the second question every time, or it never asks it at all.

What is the difference between a dashboard summary and an investigation?

A dashboard summary answers the question you asked. An investigation answers the question you should have asked next.

There are two kinds of analytics work, and they need different machinery:

Question and answer

Take a question, run a query, format the result. "What was Store 42's revenue last month?" One pass. One answer. Claude Code handles this well.

Investigation

Take a result and ask what it means and what to look at next. "Revenue is down 22%. Is that this store, this district, or the whole chain? Is the reporting period even complete?" Many passes. Each one shaped by the last.

The line between description and diagnosis

Brad Peters, who built Scoop after founding Birst, puts the gap in plain operator terms. A leader sees a dashboard, a number moves, and in his words:

"You see this dashboard, and something goes up or down, and that just begs the question of, I gotta drill in trying to figure out what the hell."

The dashboard raises the question. It does not answer it.

That drill-in, the "figure out what the hell" part, is the entire job. It is also the part a single prompt cannot do, because the drill-in is not one step. It is a sequence, and the shape of the sequence is not known until the data starts talking back.

This is the line between descriptive and diagnostic analytics, and it is worth being precise about it.

Descriptive work reports what happened. Diagnostic work explains why.

Why can't Claude Code follow a thread?

Claude Code cannot follow a thread because following a thread requires knowing which thread to pull, and that knowledge is not in the model. It is in the operator's head, and nobody wrote it down.

Consider what a real diagnostic pass looks like:

• Round one, broad scan: Pull revenue, transactions, and margin for the store. Finding: revenue down 22%, margin stable.
• Round two, hypothesis test: A stable margin with falling revenue rules out a discounting problem. So the drop is volume, not price. Is it this store or the district? Check the neighbors.
• Round three, drill: District is healthy. This store is isolated. What changed near this store in the last two weeks? A competitor opened four blocks away, and the timing lines up.

Investigation goes beyond the dashboard

Notice that round two only exists because of what round one returned. Stable margin sent the investigation toward volume and away from price. A different round-one result would have spawned a completely different round two.

This is the core of AI investigation that goes beyond the dashboard: the branching is the work, and the branch you take is decided by the last answer, not by the prompt.

Claude Code does not have that branching logic written down. So one of two things happens. It asks a generic second question that ignores what the first one returned. Or it formats the first plausible answer confidently and stops.

Either way, the thread goes unpulled.

The result shows up as sameness. Every store report reads identically:

• Sales declined, review staffing.
• Traffic is down, check marketing.
• Margin slipped, look at discounting.

The spawn problem: which diagnosis follows which symptom

The spawn problem is that a symptom has to route to a specific diagnosis, and Claude Code does not know the routing because the routing is business-specific and undocumented.

In a well-built investigation system, each result triggers a defined next move. Scoop calls this spawning.

A screen trips, the system decides whether to open an investigation into that unit, and the investigation that opens is the right one for that specific flag.

Here is what that looks like in a retail chain that runs on lending and resale, like a pawn operation:

• An origination decline (fewer new loans written) spawns one follow-up path: check loan sizing, check redemption rates, check whether staff are turning away business they should be writing.
• A false-security pattern (numbers look fine now but a lead indicator is drifting) spawns a completely different path: check the balance ratios that degrade with a lag, the early signals of what the EZ Corp team calls the death spiral.

The Drift Problem

The routing between them is not something a model infers. It is config, written by sitting with the operator who has run this business for two decades and knows that an origination decline and a false-security pattern are not the same emergency. The standards slide quietly before the numbers crater.

Without that routing written down, every flag triggers the same generic deep-dive.

The acute case queues behind the mild one. The store that is three months from a revenue crater gets the same boilerplate as the store having an average week.

Autonomous investigation without a routing table is just a louder version of the shallow report.

Which second question follows which first answer needs to be written down. Claude Code improvises it every time, or does not ask it at all.

When the obvious answer is wrong

The Confounder Problem

The confounder problem is that the most visible explanation is often not the real one, and only a disciplined investigation checks the boring alternative before drawing a conclusion.

A store's numbers look terrible. Revenue down 50% against last month. The obvious read is performance. Something is broken at that location.

The real explanation: March has 14 days of data loaded, not 31. The month is not over, or the pipeline is behind. The store is fine. The data is partial.

Claude Code sees a 50% drop and flags a crisis

A system with investigation discipline checks whether the reporting period is complete before it says a single word about performance.

That check is not clever. It is a rule, sitting at the front of the playbook, that says: before you diagnose a decline, confirm the decline is real.

The difference between a demo and a deployment

A tool that skips the confounder check will be right most of the time and catastrophically wrong occasionally, and it will deliver both with the same confidence.

The diagnostic analytics that operators trust is the kind that rules out the partial month, the calendar shift, the one-time event, before it escalates anything.

Real investigation is disciplined precisely because the obvious answer is a trap. The first plausible explanation is often a confounder wearing a costume.

The cost of improvising at scale

Improvising the investigation is not just less accurate. It is more expensive, and the expense compounds with every location.

Without a tiered loading system, every question pays the token cost of the entire knowledge base.

The tool re-reads everything it might possibly need on every pass, because it has no map telling it what each stage actually requires.

A well-designed system reads only what the current stage needs:

• The broad scan loads the store's core metrics and nothing else.
• The hypothesis test loads the district comparison context.
• The confounder check loads the data-completeness rules.

Each stage pulls its slice. Claude Code, with no staged plan, reloads the whole context window every time.

At one store, this is manageable. You barely notice.

At three hundred stores, running every cycle, it is an inference bill and a latency problem at the same time. Frontier model pricing is public. Do the arithmetic on 300 investigations, each dragging a full context window through multiple rounds, on a biweekly cycle.

The number is not a rounding error. It is a line item, and it grows linearly with every location you add, which is exactly the direction operations leaders are trying to grow.

This is why the improvisation approach does not just produce worse analysis. It produces worse analysis that costs more the more you use it. The economics run backwards.

Isn't this just a prompting problem?

No. This is the objection worth taking seriously, so here is the honest answer: better prompts help with the descriptive analytics and do almost nothing for real analytics.

You can prompt your way to a good dashboard summary. You can specify the metrics, the format, the tone, the time window, and get a clean result. That is a retrieval task, and retrieval responds to instructions.

Investigation does not, because the instruction you would need to write does not fit in a prompt. The instruction is:

"When revenue drops and margin is stable, check volume by district before anything else. When it's a pawn origination decline, check redemption rates. When it's a false-security pattern, check the balance triangle. And before any of that, confirm the reporting period is complete".

That is not a prompt. That is a decision tree with dozens of branches, specific to one business, captured from one operator's two decades of pattern recognition. It is closer to how agentic analytics actually works than to prompt writing.

Brad Peters describes how that capture actually happens, and it is not a prompt-writing exercise:

"If I took a tape recorder and recorded everything you thought as you looked at your BI reports, we stick that into the system so it could do that on your behalf."

The tape recorder is the point. The knowledge being captured is not a set of instructions the operator can type out.

It is the running commentary in their head as they read a report, the thing they check first, the threshold that makes them worry, the pattern they have learned to distrust. That gets recorded, structured, and versioned. It becomes the routing table. It does not change between runs, and it does not get re-improvised every time someone asks a question.

A model with a great prompt still improvises the branches. A system with a written playbook does not. That is the structural gap, and no amount of prompt engineering closes it.

Where Claude Code stops and Scoop begins

The investigation method is the product. Not the model.

This matters because the instinct, when the summaries come out shallow, is to reach for a better model or a cleverer prompt. Neither addresses the actual gap.

The gap is that the investigation logic, the part that decides which second question follows which first answer, is not written down anywhere Claude Code can read it.

Scoop's investigation playbook is versioned customer config. It specifies what to ask, what counts as trouble, and which question follows which answer. It is captured once, from the operator who runs the business, and then it runs everywhere, the same way, every cycle. That is the AI performance management layer sitting on top of the BI you already have.

The mechanics are deterministic where they need to be:

• When an origination decline fires, a specific diagnosis context spawns.
• When a false-security pattern fires, a different one does.
• The routing is written down, not inferred, so it does not drift between runs or between stores.

In production today: biweekly investigation cycles for a NASDAQ-listed specialty retailer. Tens of thousands of computed results per cycle. Every store analyzed on its own data with its own drill-downs, and every one judged by the same rules, in the same words. One tailored report per store, at scale, with zero AI calls on re-render, because the analysis is computed config, not a live model call every time someone opens the file.

That last part is the tell. When the investigation is written down, re-rendering a report costs nothing. When it is improvised, every render is another inference bill.

One store is a demo. A thousand stores, bespoke and comparable, is the product.

That gap is structural. It is not a prompt problem, and it is not a model problem.

It is the difference between a tool that answers the first question and a system that knows which question comes next.

Which brings up the assumption underneath all of this.

Both the summary and the investigation assume the data feeding them is clean, complete, and consistent across every location. Article 4 is about what happens when it is not, and why adding engineers does not fix it. The team behind Scoop built the layer around exactly that problem.

Frequently asked questions

What is the difference between a dashboard summary and an investigation?

A dashboard summary reports what happened: revenue, transactions, margin, formatted and delivered. An investigation explains why it happened and what to do about it. The summary answers the question you asked. The investigation answers the follow-up questions, in sequence, where each answer determines the next one. Summaries are a retrieval task. Investigations are multi-pass reasoning with business-specific routing.

Why does every store report read the same?

Because the tool stops at the first plausible explanation. Without a written playbook telling it which second question follows which first answer, it defaults to a generic deep-dive for every flag. Sales down, check staffing. Traffic down, check marketing. The routing that would send an origination decline down one path and a false-security pattern down another is not documented, so every report collapses into the same shallow template.

Can better prompts fix shallow AI analysis?

Better prompts improve dashboard summaries and do very little for investigations. A summary is a retrieval task that responds to clear instructions. An investigation requires a branching decision tree, specific to one business, that does not fit in a prompt. You can prompt the format. You cannot prompt two decades of an operator's pattern recognition into a single instruction. That has to be captured and written down as config.

What is the spawn problem in AI analytics?

The spawn problem is routing: a symptom has to trigger the correct diagnosis, and the correct diagnosis is business-specific. An origination decline and a false-security pattern show up on the same dashboard but require completely different investigations. A model does not infer that routing. It has to be written down by sitting with the operator who knows the difference. Without the routing table, every flag triggers the same generic response.

Why is improvised investigation more expensive at scale?

Because it reloads the entire knowledge base on every question instead of reading only what each stage needs. At one store the cost is invisible. At three hundred stores on a recurring cycle, each investigation dragging a full context window through multiple rounds, it becomes a real inference bill and a latency problem. A staged system reads its slice per stage. Improvisation pays the full cost every pass, and the cost grows with every location added.

How does Scoop keep investigations consistent across locations?

Scoop captures the operator's interpretation logic once, structures it as versioned config, and runs it identically across every location every cycle. The routing is deterministic: a specific flag spawns a specific diagnosis context, the same way for every store. Results synthesize per location and roll up through district, region, and division, each role scored critical, severe, normal, or healthy. Because the logic is written down rather than improvised, reports re-render with zero AI calls.