The Base Model Is Dead — Varun Singh, Arcee AI

At-a-Glance

• Verdict: Watch fully
• Core thesis: The traditional web-trained base model is no longer the main determinant of LLM capability; supervised training should increasingly be designed as a prior that prepares models for large-scale reinforcement learning, reasoning, and agentic work.
• Why it matters: For teams building agent systems, the implication is that model quality will depend less on generic web knowledge and more on whether pre-training data, architecture, and training distributions anticipate tool use, long-horizon traces, and RL environments.
• Best use: Use this as a concise technical framework for evaluating model-training claims, especially when comparing general-purpose base models against models optimized for coding, reasoning, and agents.

Executive Summary

Varun Singh argues that the old concept of a base model—a broad representation of human internet knowledge acquired primarily through next-token prediction over web text—is becoming inadequate. Historically, pre-training consumed most compute and determined downstream quality, while post-training mostly made a capable model usable in chat. In the reasoning-and-agent era, RL is no longer cosmetic alignment: it can materially improve performance on mathematics, software engineering, tool use, and other verifiable tasks.

This changes the role of supervised learning. Rather than treating pre-training as the product and RL as a final polish, Singh proposes viewing supervised learning as preparation: it should establish the atomic skills, data distributions, task formats, context lengths, and stable model representations that RL can later compose and optimize. The relevant prior is increasingly reasoning and agentic behavior rather than a maximal general-web prior.

He points to a data-mix shift across modern training recipes. GPT-3 used roughly 85% web text including Wikipedia, while newer models sharply reduce general web data and increase code, STEM, instructional, and synthetic data. Microsoft MAI Thinking 1 reportedly limits web text to 15% while avoiding synthetic model-generated data, whereas NVIDIA's Nemotron 3 Ultra explicitly brings SFT-style question-answer data into pre-training and leans heavily into synthetic data.

The operational message is not that base models literally no longer matter. They matter differently: a base model must expose the capabilities that RL will need, including task-specific formats and possibly long agent traces. This is particularly consequential for MoE training, where an abrupt shift from generic pre-training data to post-training data can destabilize expert routing and require aggressive late-stage load-balancing intervention.

Key Takeaways

• Claim: RL has shifted from a minor post-training alignment layer to a major source of capability gains, making the original base-model objective insufficient as the central measure of model quality. | Evidence: Singh contrasts earlier chat-oriented post-training with OpenAI o1's reasoning results, DeepSeek R1's public demonstration of reasoning-model construction, and coding agents that can be trained end-to-end to interact with software environments. He also cites Xiaomi Mimo Labs as allocating roughly equal compute to pre-training and post-training, and says Kimi's K2.5 reportedly used more RL compute than supervised-learning compute. | Implication: When evaluating frontier models for agentic work, assess their RL environment, reward/verifier quality, and training distribution—not just base-model size, benchmark pretraining quality, or general knowledge coverage. | Caveat: The speaker does not claim supervised learning can disappear: human language remains an exceptionally broad distribution, and he frames a full AlphaGo-like replacement of supervised learning by RL as uncertain for language models.
• Claim: Modern pre-training data is shifting away from broad web text toward code, STEM, instruction-shaped, and agentically useful data. | Evidence: GPT-3's mix was described as roughly 85% web text including Wikipedia, while LLaMA 3 still used 50% general-knowledge tokens. Singh says MAI Thinking 1 reduces web text to 15%, and notes that code—absent as a dedicated GPT-3 subset—has become a dominant component of current recipes. | Implication: Generic internet-corpus breadth is becoming a weaker proxy for usefulness in coding and agent deployments; data composition should be judged against the target interaction and environment. | Caveat: The transcript presents selected published recipes rather than a universal optimal mix; MAI Thinking 1 and Nemotron represent contrasting approaches.
• Claim: Synthetic data can improve pre-training when it is used to clean, reshape, and diversify useful seed information rather than indiscriminately inflate token counts. | Evidence: Arcee's Trinity Large Thinking used web-scale synthetic rephrasing to present the same seed information in multiple forms. Singh also cites Kimi K2's broad use of this approach and Swallow Code/Swallow Math as earlier examples. He argues synthetic generation can yield higher-quality tokens and make pre-training examples resemble instruct or agentic tasks. | Implication: For a proprietary agent or domain model, synthetic traces, reformulations, and task-shaped examples are potentially valuable, but should be treated as curated training assets with validation—not cheap substitutes for quality source data. | Caveat: Singh acknowledges the concern that blindly adding synthetic data can cause model collapse or degrade performance; data generation and filtering quality are therefore central.
• Claim: Introducing post-training-like distributions earlier can be especially important for MoE models because late distribution shifts can create expert-load imbalance. | Evidence: Singh describes MoE load balancing as a core difficulty: experts specialize during training, and objectives try to keep utilization broadly balanced. He says MAI Thinking 1 encountered a large pre-training-to-SFT distribution shift and addressed it by substantially increasing the load-balancing coefficient during SFT. | Implication: MoE model builders should test routing and utilization under the eventual instruction, reasoning, and agent-trace distribution early, rather than assuming generic-corpus balance will persist through post-training. | Caveat: The proposed remedy is architectural/training-process specific to MoEs; it is not presented as a general reason every dense model must mix SFT data into pre-training.
• Claim: The useful role of a base model is to supply atomic skills that RL can compose, not merely to store a broad world model. | Evidence: Singh cites work suggesting that a model needs exposure to the atomic skills required for a downstream task, after which RL can extrapolate and compose them if the environment is sufficiently difficult. He compares the direction conceptually with AlphaGo, where RL ultimately surpassed supervised learning. | Implication: Before investing in agent RL, identify the irreducible prerequisite skills—tool syntax, code patterns, planning formats, domain primitives, and long-context behavior—and ensure they are present in supervised training. | Caveat: This depends on having an RL environment with adequate difficulty and feedback; RL cannot reliably create capabilities from entirely absent prerequisite representations.
• Claim: Training data should include novel interaction forms and test-time-compute patterns before RL so the model can explore them effectively later. | Evidence: Singh identifies reasoning traces as a novel format that does not closely resemble ordinary human output, and points to xAI's XIA1 paper as an example of warming models during SFT or pre-training for test-time-compute schemes. | Implication: If Ken's systems depend on deliberation loops, structured tool trajectories, or extended contexts, model selection and fine-tuning should prioritize prior exposure to those exact interaction shapes over generic chat fluency. | Caveat: The transcript does not specify which test-time-compute formats generalize best or quantify their incremental value.

Detailed Brief

Two competing recipes for the new base-model prior

• Claims: The field has not converged on whether synthetic data is necessary for the next generation of base models.; Both the human-data-heavy and synthetic-data-heavy approaches still depart from the older web-dominant paradigm.
• Evidence: MAI Thinking 1 reportedly emphasizes avoiding synthetic data and filtering web scrapes to exclude data from other language models.; Nemotron 3 Ultra reportedly incorporates SFT-prefixed, question-answer-style data directly into pre-training, rather than reserving it for post-training.; Singh frames these as opposing examples despite their shared reduction in the proportional role of general web text.
• Caveats: Published data recipes offer incomplete visibility into filtering, quality controls, generation methods, and experimental controls, so their reported mixes should not be read as direct causal proof.
• Implications: The strategic decision is not simply human versus synthetic data; it is whether the resulting corpus creates robust representations for the downstream training regime and deployment interface.

Replacing phase labels with a two-paradigm view

• Claims: Singh considers the labels pre-training, mid-training, post-training, and RL increasingly blurry and less useful than distinguishing supervised next-token learning from RL.; Mid-training reflects the practical need to expose models to distributions closer to RL and agent deployment, including longer contexts.
• Evidence: He describes mid-training as bringing in the distribution likely to appear in post-training RL and adding longer context for agentic traces.; He argues that datasets used at mid-training could often be introduced earlier to establish more stable representations from the start.
• Caveats: This is a conceptual reframing, not a claim that pipeline stages have disappeared operationally.
• Implications: Training-pipeline discussions should focus on distribution continuity and capability prerequisites across stages, rather than treating phase names as hard boundaries.

Notable Concepts & Terms

• Base model prior: Singh recasts the base model as the representation and skill foundation needed for later RL, rather than a general-purpose compression of all web knowledge.
• Atomic skills: Prerequisite capabilities that supervised learning must establish before RL can combine and optimize them into more complex behavior.
• Synthetic rephrasing: Generating alternate expressions of seed data to increase coverage, improve task shape, and reinforce information without relying solely on new raw-source tokens.
• SFT data in pre-training: Moving instruction-style question-answer examples upstream so models learn expected downstream task formats before conventional post-training.
• MoE load balancing: The process of preventing a mixture-of-experts model from overusing or underusing particular experts; sharp distribution shifts can destabilize it.
• Mid-training: An intermediate stage that acclimates a model to longer contexts and distributions closer to reasoning, agent traces, or RL workloads.
• Test-time compute: Inference-time methods that let a model spend additional computation on reasoning or search; Singh argues models can be warmed up for these schemes in training.

Operator Notes / Why Ken Should Care

• Add a model-evaluation criterion for distribution match: compare candidate models' exposure to tool calls, coding tasks, structured reasoning, long contexts, and agent trajectories against the target workflow.
• For any proposed agent-RL program, write an atomic-skill inventory first and use it to identify missing supervised-training or fine-tuning data before building rewards.
• If considering MoE-based models or custom MoE training, require routing-utilization testing on the intended production distribution, not only generic pre-training or chat benchmarks.
• Treat synthetic agent traces and rephrased domain data as an experiment with quality gates: retain source provenance, use held-out evaluations, and test for degradation before scaling generation.
• When assessing vendor claims about a strong 'base model,' ask how much of the observed capability comes from supervised data, RL compute, reward environments, and test-time compute rather than parameter count alone.

Source/Metadata

• Title: The Base Model Is Dead — Varun Singh, Arcee AI
• Transcript words: 2102
• Duration seconds: 1064
• Timestamp note: No timestamps or chapter markers were present in the supplied transcript.