Skip to content
Agentic Engineering Fieldbook

Data and Analysis Agents

Data-analysis agents can feel magical: upload a spreadsheet, ask a question, receive a chart, a conclusion, and sometimes the code that produced it. In enterprise settings the same pattern extends to SQL generation, metric exploration, notebook drafting, data-quality checks, and business reporting.

The risk is that analytical fluency can hide the work that matters most. Which data did the agent access? Which rows were excluded? Which join changed the answer? Which code ran? Was the chart reproducible? Did the metric definition match the business question? A data-analysis agent should therefore be judged by executable evidence and validation, not by whether the final explanation sounds like an analyst.

Thesis

Section titled “Thesis”

A data-analysis agent is fit for delegation when data access is scoped, analysis steps are executable and inspectable, results are validated against domain and statistical checks, and the final explanation preserves assumptions, uncertainty, and reproducibility.

The accepted outcome is not a chart or a paragraph. It is an analysis artifact that can be inspected, rerun, challenged, and connected to the data and permissions that shaped it.

Start with the analytical contract

Section titled “Start with the analytical contract”

Every analysis request should be translated into an analytical contract:

  • the decision or question the analysis supports;
  • the dataset, snapshot, source system, and freshness requirement;
  • the population, filters, time window, and unit of analysis;
  • metric definitions and business semantics;
  • allowed tools, libraries, and execution environment;
  • privacy, tenant, and export constraints;
  • required validation checks;
  • the form of result and the level of uncertainty to disclose.

“Why did revenue fall?” is not yet an analysis task. It may require revenue definition, product segment, time zone, refunds, currency conversion, seasonality, acquisition channel, and data-latency assumptions. The agent can help clarify those assumptions, but it should not silently choose them and present the result as fact.

Data access is an authority boundary

Section titled “Data access is an authority boundary”

Analysis agents often need access to sensitive data. They may query warehouses, inspect uploaded files, generate temporary tables, export charts, or combine internal and external sources. Each of those steps can leak information or create an unauthorized inference.

The access model should specify:

  • which principal the agent acts for;
  • which datasets and columns are available;
  • whether row-level, tenant, or purpose restrictions apply;
  • whether generated artifacts can leave the environment;
  • whether code execution has network access;
  • how temporary files and cached results are retained;
  • how access is logged and reviewed.

If the agent can see more than the user is allowed to know, the system has created a high-risk deputy. If the agent can export raw rows when the user asked for an aggregate, the system has expanded the effect beyond the analytical purpose. Data minimization should be enforced by tools and query layers, not left to prompt wording.

Queries and code are intermediate artifacts

Section titled “Queries and code are intermediate artifacts”

OpenAI’s data-analysis product documentation shows a concrete pattern: a model-mediated system can inspect uploaded files, run Python, and create output artifacts (OpenAI Help). The durable engineering lesson is that code and queries are part of the answer. They should be captured, reviewed, and connected to results.

For SQL agents, the generated query should be visible or at least recoverable. For code agents, the notebook, script, or execution transcript should be available. For dashboards, the semantic layer or metric definition should be named. Without these artifacts, reviewers cannot tell whether the model:

  • joined on the wrong key;
  • filtered out nulls incorrectly;
  • mixed time zones;
  • treated missing data as zero;
  • double-counted entities;
  • used a sample as a population;
  • chose a misleading chart;
  • inferred causation from correlation.

Text-to-SQL benchmarks such as Spider and BIRD show that database-grounded language questions remain difficult even when schemas and execution checks are available (Yu et al., 2018; BIRD). In production, the problem is harder because business definitions, permissions, and data freshness matter as much as SQL validity.

Validation should be designed into the loop

Section titled “Validation should be designed into the loop”

An analysis agent should not wait until the final answer to discover that the data was wrong. Useful validation includes:

  • schema and type checks;
  • row counts before and after filters;
  • missing-data and outlier summaries;
  • joins checked for multiplicity;
  • totals reconciled against known reports;
  • sampled rows inspected against source records;
  • metric definitions compared with the semantic layer;
  • code rerun from a clean environment;
  • sensitivity checks for key assumptions;
  • reviewer challenge for consequential conclusions.

Computational notebooks can support reproducibility when they preserve narrative, code, data, dependencies, and execution order (Rule et al., 2019). They can also create misleading confidence when cells are stale, dependencies are missing, or hidden state affects the result. Agent-generated notebooks need the same discipline, plus traceability for model-mediated choices.

Explain the result without hiding the work

Section titled “Explain the result without hiding the work”

The user-facing explanation should answer:

  • What question was answered?
  • Which data was used, from what date or snapshot?
  • What transformations were performed?
  • What checks passed or failed?
  • What assumptions materially affect the answer?
  • What uncertainty remains?
  • What decision is, and is not, supported?

An executive summary can be short, but the evidence should remain available. A chart should include definitions and filters. A forecast should state its horizon and evaluation limits. A comparison should name the denominator. A surprising result should include the checks that make it credible or the reasons it remains tentative.

Design options

Section titled “Design options”
Option Best fit Main risk
Uploaded-file analysis Individual spreadsheet or CSV exploration Hidden privacy, stale files, and unreproducible local state
SQL assistant Query drafting over governed data Valid SQL with wrong business semantics
Notebook agent Exploratory analysis and reproducible narratives Stale cells, dependency drift, and unreviewed assumptions
Metrics copilot Governed KPI explanation through semantic layer Over-trusting predefined metrics or hiding anomalies
Autonomous analysis monitor Recurring anomaly detection or report generation Alert fatigue, silent data drift, and unowned conclusions

For recurring business metrics, a governed semantic layer and deterministic dashboard may be better than an agent. Use the agent for interpretation, investigation, and exception handling when the metric path is already controlled.

Failure modes

Section titled “Failure modes”
  1. Executable nonsense. Code runs successfully but answers the wrong question.
  2. Semantic mismatch. The agent uses raw fields instead of approved metric definitions.
  3. Data leakage. The analysis reveals rows, columns, or inferred facts outside the user’s authorization.
  4. Stale snapshot. The answer depends on old or partial data without disclosure.
  5. Irreproducible artifact. The result cannot be rerun because code, data, environment, or execution order is missing.
  6. Statistical overreach. The agent claims causation, significance, or forecast confidence beyond the design.
  7. Chart persuasion. Visualization choices make a weak or noisy result look decisive.

Fieldbook guidance

Section titled “Fieldbook guidance”

Treat data-analysis agents as executable systems. Capture code, queries, parameters, data snapshots, and environment. Keep data access purpose-bound. Validate intermediate steps before trusting final prose.

Prefer governed definitions over raw table discovery for recurring business questions. When the agent departs from the semantic layer, require it to state why and show the consequences.

Make uncertainty concrete. “Sales fell 8%” is incomplete without time window, currency, refund treatment, missing-data handling, and baseline. The user does not need every trace token, but they do need the assumptions that could change the decision.

Checklist

Section titled “Checklist”
  • Is the analysis question tied to a decision, population, time window, and metric definition?
  • Are dataset source, snapshot, freshness, and permission boundaries recorded?
  • Does the agent run code or queries in a governed execution environment?
  • Are generated SQL, code, parameters, and artifacts inspectable or recoverable?
  • Are joins, filters, missing data, row counts, and totals validated?
  • Does the explanation state assumptions, uncertainty, and what the result does not prove?
  • Are exported artifacts checked for privacy and tenant leakage?
  • Can a reviewer rerun or reconstruct the analysis from recorded inputs?

References and how they are used

Section titled “References and how they are used”