AI Engineer

State of Data — Sean Cai, Independent / State of Data

2331 summary words 10 min summary Watch video

Start with the signal

10 min read

Summary

At-a-Glance

Verdict: Watch fully
Core thesis: The durable value in AI data is shifting from scalable but contrived expert-generated tasks to live, process-level enterprise workflows and the infrastructure that continuously turns those workflows into post-training and RL environments.
Why it matters: This is directly relevant to agent systems: long-horizon agent performance depends less on generic benchmark gains and more on real task trajectories, reliable verification, environment design, model routing, and retraining across changing base models.
Best use: Use the talk as a strategic framework for evaluating AI-data vendors, designing agent evals and RL environments, and identifying control-plane opportunities around enterprise-owned intelligence.

Executive Summary

Sean Cai argues that conventional data labeling is no longer the important frontier. The scarce input is process-based data: the sequence of decisions, tool calls, state changes, corrections, and outcomes that show how professionals actually produce work. He distinguishes Type 1 data—captured from real workflows—from Type 2 data—expert-created examples in artificial settings—and argues that the industry often misrepresents the latter as the former.

His market framework is verification. A domain becomes trainable and commercially viable when correctness can be cheaply decomposed and checked, when people agree on what correct means, and when fresh verified examples are plentiful. Coding matured first because repositories, tests, and commits provide all three. Higher-value domains such as finance, law, healthcare, biology, security, and taste are harder because their work is long-horizon, less objectively verifiable, and locked within private enterprise systems.

Cai is especially skeptical of benchmark-driven data markets. He says vendors can manufacture difficult-looking tasks, select failures, package them as benchmarks, then sell data designed to improve those same benchmarks. A headline score under one agent scaffold is therefore an unreliable measure; harness and infrastructure choices can produce hidden false positives and false negatives. His finance examples suggest models with similar aggregate scores can have opposite failure profiles, such as arithmetic strength versus methodological strength.

The investment and product implication is that data businesses must become enterprise AI infrastructure businesses. Enterprises that want to own rather than rent intelligence will need a layer for routing models by cost/latency/performance, migrating RL datasets and post-training across base-model changes, and translating messy business context into executable training and evaluation environments. Cai calls these bespoke translation systems “Antikythera mechanisms.”

Key Takeaways

Claim: The key AI-data bottleneck is no longer generic annotation volume; it is high-fidelity process data that captures real professional work from initial context through decisions and final outcomes. | Evidence: Cai contrasts state-based data such as ERP rows or saved files with process-based data: trajectories, reasoning traces, and sequences of decisions. He characterizes the legacy labeling market as roughly $10–15 billion per lab annually but calls it the least interesting part of the market. | Implication: For agent products, prioritize instrumenting live workflows—tool use, intermediate states, recoveries, approvals, and independently observed outcomes—rather than treating static document corpora or isolated prompt-response pairs as the core proprietary asset. | Caveat: Type 2, contrived expert-created data remains useful as an initial training mechanism; Cai's claim is that it is insufficient for moving models from competent performance toward robust expert performance.
Claim: AI data supply chains are permanently fragmenting because quality does not scale linearly with quantity, allowing specialists to outperform vertically integrated providers at sourcing, environment design, rewards, and evaluation. | Evidence: He says labs increasingly mandate diversification across roughly 20–30 vendors because they distrust any single vendor's ability to scale quality. He contrasts this with the prior model in which firms such as Mercor, Handshake, Surge, and Scale AI had to operate as vertically integrated providers. | Implication: Avoid assuming a single data or RL-environment vendor can be a full-stack strategic dependency. Build a vendor portfolio and retain internal ownership of task definitions, quality standards, data schemas, and evaluation logic. | Caveat: The assertion is presented as Cai's market observation, not as independently substantiated vendor-revenue or procurement data in the transcript.
Claim: Verifier's Law predicts which agent markets mature first: tasks are easier to train when their outcomes are decomposable, their definitions of correctness have consensus, and fresh verified examples occur frequently. | Evidence: Cai attributes the principle to researcher Jason Wei and divides verifiability into asymmetry of verification (checkable steps), veracity of verification (consensus on correctness), and proliferation of verification (the supply of new verified examples). He explains coding's early maturity through unit tests, shared standards for working code, and abundant GitHub commit histories. | Implication: Score prospective agent verticals before committing: select initial wedges with clear step-level checks, observable outcomes, recurring work, and enough real examples, then use those workflows as a foundation for harder long-horizon tasks. | Caveat: High verifiability is not equivalent to high business value; it explains trainability and speed of product maturity, while many economically valuable workflows remain difficult to verify.
Claim: Most synthetic expert benchmarks are structurally vulnerable to Goodharting and do not establish long-horizon agent reliability. | Evidence: Cai describes a Type 2 loop in which vendors ask experts to generate plausible tasks via chat, solve them, cherry-pick cases where models fail, market them as difficult benchmarks, and sell training data intended to improve those exact tests. He argues these benchmarks usually test isolated in-distribution questions, not dependent episodes of work. | Implication: Separate task/eval ownership from data-vendor incentives. For agent systems, test multi-step episodes with constrained state transitions, non-interchangeable tool calls, forced recovery from failure, and outcome-based grading—not merely standalone task completion. | Caveat: The talk does not claim all benchmarks are invalid; its narrower argument is that a single benchmark, especially when controlled by the same party selling the associated data, is an inadequate evidence standard.
Claim: Aggregate model leaderboard scores conceal meaningful model-specific failure modes, so evaluation must be rubric-level and scaffold-aware. | Evidence: Cai says cross-harness and cross-infrastructure differences cause benchmark divergence, including hidden false-positive and false-negative rates in SWE-style benchmarks. On private finance tasks—ARR waterfall reconciliation, an LBO evaluation memo, and a long-short pair trade—he reports that Opus 4.8 could underperform 4.7 on some rubrics, while GPT 5.5 and Opus 4.8 scored within three points overall but failed in opposite ways: GPT handled arithmetic better and Opus methodology better while missing arithmetic. | Implication: Do not route models using a general leaderboard alone. Build a workload-specific scorecard that isolates relevant dimensions—calculation, planning, policy compliance, tool execution, recovery, and domain methodology—and validate each model in the production harness. | Caveat: These are Cai's internal/private evaluations, with no task specifications, sample sizes, scoring protocol, or independently reproducible results provided in the transcript.
Claim: Data-market purchasing can serve as an upstream signal for forthcoming frontier-model application priorities, but only where the underlying modality and verification setup are sufficiently settled. | Evidence: Cai says Anthropic spent heavily on cybersecurity data in January and on biology data in March–April, followed two to three months later by Cyber and Bio/Life Sciences product efforts. His screen for robust long-horizon work includes enforced step length, heterogeneous non-interchangeable tool calls, state transitions that constrain subsequent actions, mandatory recovery, sequential decisions on a single entity, expert action at each step, and independently recorded outcomes. | Implication: Monitor enterprise data procurement and vendor formation as leading indicators, but avoid investing heavily in a data modality until its training representation, verifier design, and operational use case are clear. | Caveat: He cites robotics as a counterexample: choices among egocentric data, teleoperation, and UMI remain unresolved research questions, and he considers many robotics-data vendors unsophisticated. Data demand alone is therefore not a sufficient market signal.
Claim: The enduring enterprise opportunity is a model-agnostic control plane that turns proprietary work into continuously maintained AI capability; pure data businesses will be pushed toward services and application-layer ownership. | Evidence: Cai says successful data companies, including Mercor and Handshake by his account, are already generating substantial enterprise revenue. He names needs to route smaller models by cost, latency, and performance; retain RL datasets through base-model migration by rerunning post-training instead of restarting; and create “Antikythera mechanisms” that translate business context into training/evaluation systems. He also argues that model alternatives such as GLM 5.2 demonstrate application-layer companies can decouple from a single foundation-model provider. | Implication: The defensible architecture is not a one-model application or a static dataset; it is a portable operating loop: capture work, evaluate it, train or adapt models, route among providers, and repeat as models and workflows change. | Caveat: Claims about particular vendors' enterprise revenue and GLM 5.2's relative performance are unsupported in the transcript and should be independently checked before informing a market or investment conclusion.

Detailed Brief

The long-horizon opportunity and why live enterprises matter

Claims: The largest remaining automation opportunity lies in deeply dependent, long-horizon white-collar work rather than the short-horizon tasks that are currently addressable.; A dataset has a limited useful life at the frontier because models eventually absorb it; a live business relationship creates a renewable stream of work trajectories and outcomes.; RL-environment companies should be viewed primarily as research accelerators, and become venture-scale only when their generalized infrastructure supports a real enterprise application.
Evidence: Cai depicts task horizon on the horizontal axis and share of white-collar work on the vertical axis, arguing that the addressable threshold shifts rightward when a real-world data pipeline is built.; He calls startup codebases a frequently purchased but ultimately finite source of data, contrasting them with ongoing enterprise workflows.; He warns against vendors offering large volumes of weakly relevant video—for example, '100,000 hours of iPhone video'—as a substitute for a coherent robotics-data strategy.
Caveats: The speaker does not provide empirical estimates of the size of the long-horizon market, the cost of acquiring workflow rights, or the economics of enterprise data partnerships.; Capturing actual workflows raises unaddressed issues around privacy, IP ownership, consent, auditability, and security controls.
Implications: A durable data-rights strategy should target recurring operational systems where outcome quality can be observed over time, rather than one-time corpus purchases.; The hard problem is not merely collecting traces; it is preserving the context and state needed for those traces to remain useful as executable agent environments.

Antikythera mechanisms as the missing enterprise translation layer

Claims: Generic foundation models cannot directly consume messy enterprise context as a reliable training signal; organizations need bespoke systems that translate business rules, tools, constraints, and outcomes into evaluable environments.; Cai positions his newly announced work as building these systems and delivering RL as a service with companies that monetize their data assets.
Evidence: He repeatedly labels the mechanism an “Antikythera mechanism,” borrowing the image of a specialized machine that converts complex inputs into a structured output.; The talk identifies enterprise-owned intelligence as the end state, replacing permanent dependence on rented frontier-model capability.
Caveats: The presentation does not define the full five-job architecture; it explicitly names model routing, RL-data management across base-model migration, and Antikythera mechanisms, while leaving the remainder unspecified.; This portion is also a product announcement, so its framing should be read as strategic positioning as well as analysis.
Implications: The control plane for enterprise agents should model workflow semantics explicitly: permissions, tools, intermediate artifacts, state transitions, acceptance criteria, exception paths, and feedback sources.; The enterprise vendor likely to retain value is the one that operationalizes these semantics across model changes, rather than the one that merely brokers annotators or sells a snapshot dataset.

Notable Concepts & Terms

Type 1 data: Real workflow capture with minimal artificial reward shaping, such as code commits or session replays; Cai considers it the route to realistic expert-level model behavior.
Type 2 data: Contrived examples manufactured by hired experts in an artificial setting; useful for bootstrapping but vulnerable to benchmark gaming and weak transfer to real work.
Process-based data: Decision trajectories, reasoning traces, tool usage, and workflow states rather than final records; it is the central data asset for long-horizon agents.
Verifier's Law: Jason Wei's framing that a task's trainability rises with ease of verification; Cai uses it to predict which AI application domains will mature first.
Asymmetry, veracity, and proliferation of verification: Cai's three-part diagnostic: whether work can be checked stepwise, whether correctness has consensus, and whether verified examples are continuously generated.
Cross-harness / cross-infrastructure differencing: Performance variation caused by the agent scaffold and execution setup rather than the base model alone, making single benchmark results unreliable.
Antikythera mechanisms: Cai's term for bespoke enterprise systems that convert messy operational context into usable evaluations, rewards, and RL environments.
Neolabs: Data companies evolving into enterprise AI infrastructure, service, and application companies because durable value lies beyond data collection.

Operator Notes / Why Ken Should Care

Create an internal agent-evaluation policy requiring workload-specific, multi-step tests in the exact production harness before changing models or vendors; prohibit model selection from a single public benchmark.
Build an eval suite around real operating episodes with explicit state, required tools, recovery paths, independent outcome signals, and separate rubrics for methodology versus arithmetic/execution.
Treat workflow telemetry as a product capability: determine which agent interactions, human corrections, artifacts, approvals, and outcome signals can be legally captured and reused for evaluation or post-training.
Design model abstraction now: keep prompts, tools, evals, training data, and routing policies portable across providers and open-weight bases, with a defined revalidation process after each base-model change.
Due-diligence data/RL vendors for provenance, whether tasks are live or synthetic, evaluator independence, harness sensitivity, long-horizon specifications, and rights to use operational data; do not accept benchmark difficulty as proof of transfer.
Monitor cybersecurity, biology, finance, healthcare, law, and other high-wage domains for evidence that verification and enterprise-workflow access have improved; treat raw data spending as a lead signal rather than a commitment trigger.

Source/Metadata

Title: State of Data — Sean Cai, Independent / State of Data
Transcript words: 5374
Duration seconds: 1102
Timestamp note: No timestamps or chapter markers were present. The transcript contains a substantial duplicated segment and repeated extraction noise near the end.

Search this video Resume transcript at 0%

Full transcript 3257 words · 24 min read