Tl;dr

  • I discuss how the questions that real people ask analytics agents differ from those encoded in analytics evals.
  • I describe a 3-part workflow for answering analytics questions, and show how traditional evals narrowly focus on the middle part, the “data pull”.
  • I talk about how “broad evals” might fix this problem, and, in doing so, might give us more realistic expectations of accuracy in the real world. That being said, these evals are much harder to build and maintain.
  • I develop a scheme to classify “refusals”: analytics questions that should be refused by an agent.

Background

Recently I’ve been working on Slack’s self-serve analytics agent. (A self-serve analytics agent is an AI agent that can answer analytics questions in a “self-serve” way — without needing the guidance of an expert, like a data scientist. It can run read-only queries against the data warehouse, and it is usually seeded with a lot of context about the data and how to query it.)

Companies, like Anthropic, usually suggest using “offline evals” (”evals”, for short) to evaluate the accuracy and performance of such an agent:

A common pattern we see is that data teams will set up elaborate analytic environments without having any process to understand the accuracy of their analytics agents.

One way of addressing this gap is via offline evals, which are simple question / answer pairs. You can think of offline evals similar to offline testing for an ML model in that they don’t tell you the performance of your online agents, but they do give you a good sense of whether you’ll have any critical gaps.

A question might be something like “On October 6 2025, what was Slack WAU?”. The answer would be however many millions of weekly active users Slack had on that date (although I can’t divulge the answer, you can join Slack if you wish to find out; we’re hiring!)

A case study

If you’ve ever worked with a non-technical stakeholder — the kind of person who might benefit from a self-serve analytics agent — you’ll know that they tend not to ask precise and hyper-specific questions like, “On October 6 2025, what was Slack WAU?”

So let’s pose a more realistic question. Let’s suppose the end-user asks, “what % of teams end up paying for Slack?”

That kind of question, without any additional clarification, is probably unanswerable or at least meaningless. Is it fair to lump together teams who sign up for Slack through the website (”self-serve”) with ones who were courted by an account executive (”enterprise”)? How long of a window should we wait for payment/conversion? What date range should we look at? If we look too far in the past, we might miss out on more recent behavior. But if we look too close to the present, and our window is too long, we risk running into the “censoring” problem, wherein cohorts haven’t had enough time to mature.

The issues don’t stop there. Should we count test teams towards the denominator? Or teams we suspect of spam? Or low-intent teenagers who joined because of Slack’s collaboration with Mr. Beast? Or teams whose Slack workspace was auto-generated by virtue of being a Salesforce customer? Etc.

This example reveals two big problems.

The first is that it’s difficult to “eval” the original question. The question is flawed, so there’s more than one “right” answer. One way of addressing this problem is to make the eval question more explicit, more precise, less ambiguous. One could ask, “what percentage of self-serve, non-test, non-spam, non-auto-created teams with X proxy for intent, created between February 2026 and May 2026, purchase a paid self-serve plan within 40 days?” All of the implicit pieces of knowledge, the ones that led to the ambiguity discussed previously, are made explicit.

Where this approach falls down is that no one talks like this. If my PM asked this question, I’d be nervous.

The other option is to test the agent on its knowledge of implicit facts: domain knowledge, general business sense, etc. The agent should probably know to filter out test and spam teams. So perhaps this qualifier doesn’t need to be included in the question. The agent should probably also know that, at Slack, the new team conversion rate metric uses a window of 40 days. So the “within 40 days” qualifier might also be unnecessary. We can potentially strip down the question, and make it sound more like a real human question, by relying on the agent to fill in the gaps.

For the non-implicit facts — like, self-serve vs. enterprise — the agent could refuse the initial question and work with the stakeholder to turn it into a better, more “answerable” one. In this way, a non-technical stakeholder could ask a “normal” question, and the agent would respond with something sensible.

The second problem, though, is that numbers aren’t the same as insights. What % of teams end up paying for Slack is a descriptive claim, not a normative one. It doesn’t tell us whether that number is good or bad, whether it might be increased further or has reached its ceiling, or how much more money we’d get by increasing it. (In fact, the original question is probably already too far from being a representative end-user question. Something like “is team conversion healthy?” might be more representative.) If the end-user wants an insight, not a number, how would we “eval” that?

A framework

Let’s try to generalize from this example. Answering a self-serve analytics question is a three-part process:

Flowchart for answering analytics questions

  1. Clarification/refusal. The agent pushes back against the end-user’s vague question, and replaces it with one or more precise questions, after a conversation with the end-user. In our example above, the agent would refuse the initial question, and ask about cohort dates and maturity windows, dimensions like “self-serve”, and gotchas like Slack’s Mr. Beast campaign or team auto-creation. (It might not need to ask about filters like “test teams” if these are deemed “common knowledge”.)
  2. Data pull. The agent answers each of the precise questions by querying the data warehouse. It might rely on a routing layer (a skill or some other source of context) to find the right tables, and a semantic layer to understand how a metric is defined on top of those tables.
  3. Narration. The agent takes the data it has pulled and crafts a narrative. It tries to go beyond mere description and provide an insight, a point of view, something that puts these raw numbers in context. (This doesn’t necessarily have to be “actionable” — agents are often annoying about proposing next steps, even when we don’t need them to.)

The flow isn’t unidirectional, either. It’s possible a question can’t be answered with the data available. But the agent doesn’t necessarily know that fact in advance; it likely has to proceed to step 2 in order to refuse the question in step 1. Similarly, when a data scientist builds an “insight”, they often collect data they didn’t anticipate collecting at the start. The questions you end up answering are often different than the one you started with. The flow is more of a loop than a linear process.

What does an eval really measure?

Fitting evals into the framework above isn’t straightforward. One approach — the simplest — is to use evals just on step 2: the “data pull”. Each question in the eval pair is totally unambiguous; each answer is an indisputable numerical fact.

My guess is that when Anthropic reports “~95% accuracy in aggregate” for its agents on “business analytics queries”, or Block claims its employees can go from “a business question to a trusted answer through an AI agent, with 100% metric accuracy for governed metrics with no SQL review required”, they are referring to step 2.

(It’s just a “guess” because these blog posts don’t divulge a key piece of information: what kinds of questions are being eval-ed. It’s also possible to boost “accuracy” by tilting the dataset towards questions that are very easy to answer, although that’s a separate topic.)

But 95% accuracy on step 2 does not imply 95% accuracy for the entire workflow! Failures are common in steps 1 and 3 — especially step 1. We might be guilty of the “drunkard’s search”: we focus on step 2 because it’s the easiest to grade (there’s a numerical answer; we don’t need an LLM to adjudicate things).

If an agent needs to be told everything relevant upfront, then it might perform flawlessly on precisely-worded eval questions, but badly on vaguely-worded stakeholder questions. The gap between online and offline performance would reflect how challenging it is for agents to deal with ambiguity. If a data scientist succeeds where an agent fails, it is because they ask good questions, not because they write good SQL. (There’s obvious parallels to other disciplines, like software engineering. Crudely speaking, “technique” feels less important now, and “judgment” more important.)

It’s worth noting that such an agent — one that excels at eval-type questions, but not at real-world questions — might still be (very) useful for data scientists or other domain experts. It would speed up the most tedious parts of data analysis. While that’s a win for labor, it’s not a win for capital. It doesn’t automate away an entire category of jobs, which is the ultimate goal of frontier labs and tech CEOs.

Relatedly: in my experience, agents rarely refuse to answer data questions. But many end-user questions should be refused, at least in their original form! (See the “ask a DS” Slack channel at your company if you need proof.) If an eval set contains only questions that are answerable and ought to be answered, “eval accuracy” might again overstate “true accuracy”.

The alternative to “narrow evals” is “broad evals”. Instead of evaluating only step 2, evaluate the other steps, either solo or in combination. This is a lot harder. Grading requires LLM-as-a-judge and a subjective rubric. It becomes more difficult to generate evals from existing dashboards or past SQL queries. The eval set must incorporate refusals, and enough of them that the agent can’t “pass” just by answering answerable questions. Some answers will have to be insights, not just numbers, and some of the questions in the eval set will have to be open-ended and ill-posed. Etc.

I believe broad evals are better than narrow evals, but they are also much more onerous to build and to maintain. People who write about “self-serve analytics” should, in my view, should discuss these issues instead of simply making breathless pronouncements about “95% accuracy”. (I don’t doubt that 95% accuracy in self-serve analytics will be possible some day, but I don’t see it yet, based on my day-to-day work.)

Digression: on refusal

While we’re on the topic of question refusal, I thought it might be useful to share a typology of reasons a data question should be refused. This is probably non-comprehensive, so suggestions are welcome!

  • Not a data question: The question involves enough outside-of-data judgment that an agent can’t answer it. To make progress
    • Example: How should Slack’s self-serve business make more money?
  • Vague/imprecise: The question isn’t precise enough to be answered. To make progress requires clarification, stating of assumptions, etc.
    • Example: see discussion above!
  • Insufficient data: The data needed to answer a question doesn’t exist, even if the question is well-posed. To make progress requires requires more instrumentation, data ingestion, etc.
    • Example: How many users were active at X point in time across all Slack workspaces?
  • Performance: The raw data exists, but querying it isn’t performant. To make progress requires a data pipeline.
    • Example: How many users saw X impression event, month-by-month, over the last 3 years?
  • Complexity: The question is well-posed, but is complex enough that it can’t be broken down into a set of N data pulls that are answerable in a reasonable amount of time. To make progress requires a data scientist (or burning a lot of tokens, potentially fruitlessly).
    • Example: How much extra money might we get from changing our pricing model from X to Y, under assumptions Z1, Z2, and Z3?

Summary

There’s a big gap between the questions real people ask of “self-serve” analytics agents, and the kinds of questions I’ve seen in eval sets for self-serve analytics agents. The former are often meaningless, ill-posed, or too vague to be answered. The latter, by construction, admit a single correct answer.

Relatedly, what evals test is just one step in an overall analytics workflow. They basically assess whether an agent can go from a well-posed data question to SQL that answers that question. But they don’t evaluate how to turn an ill-posed question into a well-posed one, or whether the data from a data pull is used in a “correct” way to develop an insight.

My theory is that the kinds of figures quoted in blog posts (95% accuracy, 100% accuracy, whatever) are misleading for these reasons. If you’re publishing a blog post about evals, you should reveal much more information about the kinds of questions you’re evaluating the agent on.

Separately, I think the topic of “refusals” is interesting, and I’ve constructed an initial typology. Ideally, if an agent refuses a question, it should provide a reason why, and that reason should fall into one of these categories. Refusals are just as important as “regular” evals are.