Every

LET'S RIP FABLE TOKENS FROM THE JACUZZI

3911 summary words 17 min summary Watch video

Start with the signal

17 min read

Summary

At-a-Glance

Verdict: Skim
Core thesis: Claude Opus 5 (Fable) has returned after export controls were lifted, prompting immediate real-world testing for copy-editing benchmarks, token spend audits, and code review tasks—but new safety classifiers and the absence of EAP may limit production utility.
Why it matters: First live operator stress-test of Fable post-export controls; reveals actual capabilities, cost discipline, and workflow integration challenges that EAP insiders skipped.
Best use: Ken should watch the first 15 minutes for Fable's verdict on Every's copy-edit experiments and benchmark architecture (Kate Bench), then skip to operator notes for workflow integration lessons and cost/safety trade-offs.

Executive Summary

Dan Shipper (Every CEO) streams live from Cabo as Claude Opus 5 (Fable) returns after U.S. export controls are lifted. He immediately sets Fable to work on two high-stakes internal projects: reviewing four months of failed copy-editing experiments (Kate Bench—an attempt to replicate Kate Lee's human editorial judgment with 30,000 historical edits) and auditing three months of personal token spend across ChatGPT, Claude, Codex, and Google AI Studio. Fable delivers a sharp diagnosis within minutes: Every's benchmark team spent 180 agent threads optimizing for 70% recall against Kate's exact historical edits without first validating that Kate herself is consistent. Fable estimates Kate's self-consistency ceiling is ~47–50%, meaning the models weren't failing—they were near the human limit. It recommends switching from strict historical recall to a live suggestion loop where Kate accepts/rejects edits in real time, creating a continuously improving label factory.

The stream reveals three immediate operator concerns: (1) Fable's new safety classifiers triggered on innocuous commands (process kill + browser automation) and silently downgraded to Opus 4.8 mid-task; (2) memory persistence and context bloat are now cost-sensitive at Fable's price point, pushing operators to disable memory; (3) concurrent model releases without early-access programs (EAP) mean users encounter rough edges—Sonnet 3.5 v2 (released the same week) received universally negative internal reviews except from one editor using it in short collaborative loops, not long agentic runs. Mike Taylor (head of AI consulting at Every) joins from AI Engineer Summit and plans to use Fable for DSPy book review, code review on a stalled open-source project (Rally), and rewriting a failed memetics book for broader appeal.

The Kate Bench case study is the stream's most substantive segment. Every has been fine-tuning Qwen, GPT-4, and Claude models on 30,000 of Kate's historical copy edits, attempting to replicate her editorial judgment. None have exceeded 50% recall. Fable's analysis suggests the team conflated two distinct problems: (1) repair at a known site (which is learnable cheaply—proven by a $5.29 experiment that worked) and (2) detection and selection (which remains unsolved and may require a different architecture). Fable recommends deploying the model in Kate's live workflow so she can accept/reject suggestions concurrently with her own edits, generating higher-quality training labels than historical forensic recall. Kate (VP of Editorial) expresses concern about reading unfiltered drafts first to preserve her independent editorial judgment, but agrees to a concurrent workflow where the model runs in parallel and results are compared post-edit.

Dan successfully restarts a Fable task after it downgrades to Opus 4.8, suggesting that Lutnik's safety classifier can be bypassed by flipping models mid-conversation. The classifier appears more sensitive to computer-control actions (process termination, browser automation) than to text-only reasoning. No blog post clarifies the new policy beyond 'bigger safety margin.' The stream underscores a broader operator lesson: Anthropic's cadence of releasing models without EAP (4.7, Sonnet 3.5 v2) correlates with lukewarm or negative initial reception, suggesting internal confidence issues or distraction (Fable export-control crisis). However, models often age well—4.7 was panned, then rehabilitated after users discovered new use cases. Fable's token cost disciplines operators: Mike notes he's 'famously against memory,' Dan cancels running agents to free compute, and Ariel (head of ops) demands token-spend audits. The era of 'ripping tokens unaccountably' is over.

Key Takeaways

Claim: Fable diagnosed that Every's copy-editing benchmark (Kate Bench) was optimizing for an unvalidated target: 70% recall against Kate's exact historical edits, when Kate herself may only hit ~47–50% self-consistency. | Evidence: Fable: 'It spent 5 weeks, 180 agent threads optimizing a target that was never validated. Strict recall against Kate's exact historical edits with a 70% goal set before anyone measured whether even Kate can hit 70% against herself. The evidence scattered through the repo says she can't. The sealed failures at 47.5 and 50 may actually be near the human ceiling.' | Caveat: Fable's estimate is inferred from scattered repo evidence, not a controlled study. Kate has only self-duplicated a handful of edits, so the sample size is tiny. | Implication: Operator lesson: Benchmarks that target exact human replication may be chasing noise, not signal. Shift to acceptance-based evals (would the human accept this edit?) rather than forensic reconstruction. This applies to any domain where human judgment varies with context, mood, or time—legal review, medical triage, content moderation. | Timestamp: ~8:00–12:00
Claim: Fable identified that Every's two successful experiments—span-conditional repair ($5.29 cost) and a live suggestion loop (52/53 useful)—were sidelined in favor of expensive, low-signal optimization runs. | Evidence: Fable: 'The two things that did work—span conditional repair for $5.29 and a live suggestion loop Dan rated 52 out of 53 useful—were sidelined. The single cheapest high-information action, one human spending two hours judging 60 cases, has been sitting idle since June 23rd.' | Caveat: The stream does not explain why these experiments were deprioritized. Possible reasons: organizational inertia, overcommitment to fine-tuning pipelines, or difficulty operationalizing the live loop at scale. | Implication: Operator anti-pattern: Complex, expensive optimization can crowd out simple, effective interventions. Fable's recommendation—use the live loop as a label factory—suggests that production deployment is often the best training data generator, not offline fine-tuning. Ken should ask: Are we running experiments because they're tractable, or because they're effective? | Timestamp: ~10:00
Claim: Fable's new Lutnik-era safety classifier silently downgraded Dan's task to Opus 4.8 mid-execution when he attempted to kill a process and automate a browser action. | Evidence: Dan: 'I have just hit our first Lutnik guardrail. I asked Fable to put together a report on all of my token spending… It got triggered on a command that was basically killing a process on my computer and trying to do some work with Chrome. That silently tripped it up about 20 minutes ago. Now it's back to Opus 4.8.' | Caveat: Dan was able to restart the task by manually flipping the model back to Fable, and it continued. Unclear whether this is reliable or context-dependent. The classifier's trigger threshold is opaque. | Implication: Fable's computer-use and browser-automation safety rails are stricter than text-only reasoning. Operators using Fable for agentic workflows (process management, file I/O, web scraping) will hit silent downgrades. Workaround: Flip models manually mid-conversation. This will be a persistent UX pain point until Anthropic clarifies policy. Ken's agent systems should log model downgrades as a reliability signal. | Timestamp: ~35:00–38:00
Claim: Anthropic released Claude Sonnet 3.5 v2 without early access, and Every's internal vibe check was universally red except for one editor (Jack Chang) who used it in short collaborative loops, not long agentic runs. | Evidence: Kate: 'I've never seen a response like this where it was just not just meh of like, oh, it's not better than this. It was like, it's actually just worse and it's not good at this and it's not good at this. Like, who is this for?' Dan: 'The last model that we all didn't really like from Anthropic was 4.7, and that was the last model that they dropped without an EAP, which they did for this one too.' Jack's exception: He uses Sonnet in 'much shorter chunks of work… working collaboratively with it,' not long-running tasks. | Caveat: Models often age well. Every's coverage of 4.7 was lukewarm initially, then rehabilitated after users discovered new use cases. Sonnet 3.5 v2 may follow the same trajectory. No EAP means no pre-tuned workflows or best-practice guides. | Implication: Operator heuristic: Anthropic models released without EAP are less reliable for day-one production use. If a model is 'clearly amazing,' Anthropic runs EAP. If it's dropped cold, expect rough edges. Jack's collaborative-loop success suggests Sonnet 3.5 v2 may excel at interactive editing/refinement rather than autonomous long-context tasks. Ken should test in both modes before ruling it out. | Timestamp: ~45:00–50:00
Claim: Fable's token cost is forcing operational discipline: operators are disabling memory, killing agents to free compute, and auditing token spend for the first time. | Evidence: Dan: 'I've been hounded. We're sort of a real company now, and so I can't just rip tokens unaccountably anymore.' Mike: 'I'm famously against having memory turned on… the tokens are so expensive. I want to be really judicious about what tokens go into Fable.' Dan kills running agents mid-stream to fix screen-share lag. Ariel (head of ops) demanded a three-month token-spend report. | Caveat: Dan is CEO of a funded startup and was previously comfortable with untracked spend. Smaller operators or bootstrapped teams have always faced this constraint. The shift is cultural, not technical. | Implication: Fable's pricing ($15/M input, $75/M output per unofficial reports) is high enough to force ROI conversations at Every's scale (~$40M Series A per Crunchbase). Operators will shift from 'spray and pray' token usage to targeted, high-confidence prompts. Memory persistence becomes a cost center, not a convenience. Ken's systems should track token spend per task type and auto-disable memory for low-stakes workflows. | Timestamp: Throughout; explicitly ~22:00, ~38:00

Detailed Brief

Kate Bench: Fable's diagnosis of a failed copy-editing benchmark

Claims: Every spent 5 weeks and 180 agent threads optimizing recall against Kate's exact historical edits without validating that Kate herself is consistent.; Failures at 47.5% and 50% recall may be near the human ceiling, not model failures.; Repair at a known site is learnable cheaply ($5.29 experiment succeeded), but detection and selection remain unsolved.; The forward path is to split the objective: shift from 'recall versus history' to 'acceptance precision in the live loop,' using production deployment as a label factory.
Evidence: 30,000 historical copy edits from Kate across ~4,000 documents over four years.; Fine-tuning attempts on Qwen, GPT-4, and Claude models; none exceeded 50% recall.; Span-conditional repair cost $5.29 and worked; live suggestion loop rated 52/53 useful by Dan.; Two hours of human judging on 60 cases has been idle since June 23rd.; Fable's recommendation: Deploy model in Kate's live workflow; she accepts/rejects suggestions concurrently; results are compared post-edit.
Caveats: Kate has only self-duplicated edits a handful of times; Fable's consistency estimate is inferred, not measured.; Kate expressed concern about reading drafts with pre-existing AI edits ('I want to read it my way… I want an unfiltered reaction'), but agreed to a concurrent workflow as a compromise.; Unclear whether the live loop scales beyond Kate—Every has multiple editors, and training data generation may bottleneck on Kate's throughput.; Fable did not specify how to solve detection/selection; it only confirmed it's the hard part.
Implications: Benchmark validity: If the human ceiling for self-consistency is 50%, targeting 70% is chasing noise. Operators should validate human baselines before optimizing.; Deployment as training: The live loop (production use → accept/reject → retrain) is a continuous learning system. This is more capital-efficient than offline fine-tuning if human-in-the-loop bandwidth exists.; Detection vs. repair: Two-stage architectures (detect → repair) may outperform end-to-end models. Fable's $5.29 repair success suggests span-conditional edits are tractable; detection is the bottleneck.; Editor workflows: Kate's 'unfiltered first read' preference suggests AI-assisted editing must preserve editorial agency. Concurrent workflows (AI runs in parallel, not inline) may be the only acceptable UX for high-judgment roles.

Fable's Lutnik-era safety classifiers and operator workarounds

Claims: Fable's new classifier silently downgrades tasks to Opus 4.8 when triggered by computer-control actions (process kill, browser automation).; Dan successfully restarted the task by manually flipping the model back to Fable mid-conversation.; Anthropic's blog post mentions 'a bigger safety margin' but provides no specifics on what triggers downgrades.; Memory persistence exacerbates classifier sensitivity—Mike notes biology content in memory triggered guardrails for one user.
Evidence: Dan's token-spend audit task triggered on 'killing a process and doing something in my browser'; silently flipped to Opus 4.8 after ~20 minutes.; Dan flipped model back to Fable, and it resumed on the same thread.; Mike: 'If they had something in memory that would trip it quite often as well. There's a case of this guy on Twitter saying that because he is a biologist, it was remembering biology stuff and then tripping some of those safeguards.'; Dan and Mike both disable memory to control context and cost.
Caveats: Unclear whether manual model-flipping is reliable across all contexts or just low-risk tasks like token audits.; No clarity on whether the classifier is stricter for computer-use actions vs. text-only reasoning.; Anthropic's policy is opaque; no public documentation on trigger criteria or downgrade logic.
Implications: Agentic workflows that require file I/O, process management, or browser automation are at higher risk of silent downgrades. Operators should log model switches as a reliability metric.; Memory-off is now a cost and safety optimization. Ken's systems should default to memory-off for low-stakes tasks and memory-on only for high-context, trusted workflows.; Workaround: If downgraded, flip models manually mid-conversation. This is a UX anti-pattern but appears functional as of July 1, 2025.; Lutnik's 'bigger safety margin' likely means lower risk tolerance than pre-export-control Fable. Operators should expect more false positives.

Sonnet 3.5 v2: Universally panned except in short collaborative loops

Claims: Every's internal vibe check on Sonnet 3.5 v2 was red across the board except for Jack Chang, who uses it in short, interactive editing sessions.; The model was released without early access (EAP), which correlates with rough edges and negative first impressions.; Jack's success: 'He is less likely to be giving Sonnet a very long-running task to just set it off and do something. He's working much more collaboratively with it, and he found that in working collaboratively with much shorter chunks of work that it actually was really good.'; Anthropic's previous cold-drop model, Opus 4.7, was also panned initially but rehabilitated after users discovered new use cases.
Evidence: Kate: 'I've never seen a response like this where it was just not just meh… it was like, it's actually just worse.' Uniform red rankings except Jack's green.; Dan: 'The last model that we all didn't really like from Anthropic was 4.7, and that was the last model that they dropped without an EAP, which they did for this one too.'; Jack rebuilt Codex on his own and is 'really deep into this stuff,' so his eval carries weight.; Dan notes models 'often age well' as users find new workflows.
Caveats: No EAP means no pre-tuned prompts, best practices, or known strengths/weaknesses. Users are discovering the model's fit in real time.; Jack's collaborative loop may not generalize to other operators or domains.; Anthropic may have prioritized Fable's return over Sonnet 3.5 v2's polish, so rough edges could be timing artifacts, not capability limits.
Implications: Model selection heuristic: If Anthropic drops a model without EAP, assume it's less production-ready. If they run EAP, it's 'clearly amazing.'; Task type matters: Sonnet 3.5 v2 may excel at interactive refinement (editing, debugging, iterative prompting) but fail at long-running autonomous tasks (codegen, research, agentic loops).; Patience pays: Models often age well as users discover fit. Ken should re-eval Sonnet 3.5 v2 in 3–4 weeks, especially if Jack's collaborative loop generalizes.; Every's vibe-check process (red/yellow/green/gold rankings + thorough documentation) is a repeatable eval framework Ken can adapt for his own systems.

Token cost discipline and the end of unaccountable usage

Claims: Dan is being audited by Ariel (head of ops) for three months of token spend across ChatGPT, Claude, Codex, and Google AI Studio.; Fable's cost is high enough that operators are disabling memory, killing agents mid-task, and scrutinizing ROI for the first time.; Mike: 'I'm famously against having memory turned on… the tokens are so expensive. I want to be really judicious about what tokens go into Fable.'; Dan: 'We're sort of a real company now, and so I can't just rip tokens unaccountably anymore.'
Evidence: Dan kicked off a Fable task to audit token spend across all platforms for the last three months, including personal and company workspaces.; Dan killed running agents mid-stream to fix screen-share lag, suggesting compute resource constraints.; Mike disabled memory to control context bloat and avoid triggering classifiers.; Ariel has been 'bothering [Dan] all week' for the token-spend report.
Caveats: Every raised a $40M Series A (per Crunchbase), so cost discipline is relative—smaller operators have always faced this constraint.; The shift is cultural (transition from 'move fast' to 'run like a real company'), not necessarily driven by Fable's absolute cost.; No public pricing for Fable, but industry estimates are $15/M input, $75/M output.
Implications: Fable's pricing tier will force ROI conversations at mid-market scale. Operators will shift from 'spray and pray' to targeted, high-confidence prompts.; Memory persistence is now a cost center. Ken's systems should track memory overhead per workflow and auto-disable for low-context tasks.; Token-spend audits should be automated: track cost per task type, model, and outcome. Every is building this in-house with Fable; Ken can replicate with structured logging.; The 'rip tokens unaccountably' era (2023–2024) is over. 2025+ operators will treat LLM calls like cloud compute: metered, audited, and optimized.

Notable Concepts & Terms

Kate Bench: Every's internal benchmark attempting to replicate Kate Lee's (VP of Editorial) copy-editing judgment by fine-tuning models on 30,000 of her historical edits across 4,000 documents. Failed to exceed 50% recall; Fable diagnosed the target as unvalidated because Kate's self-consistency may only be 47–50%.
Span-conditional repair: A $5.29 experiment in Kate Bench where the model was given a known edit site and asked to repair it. It succeeded, proving that repair at a known site is learnable cheaply. Detection and selection (identifying which spans need editing) remain unsolved.
Live suggestion loop / label factory: Fable's recommended architecture for Kate Bench: Deploy the model in Kate's production workflow; she accepts/rejects suggestions in real time; accepted edits become high-quality training labels. Rated 52/53 useful by Dan but was sidelined in favor of offline fine-tuning.
Lutnik guardrails / classifier: New safety mechanisms in Claude Opus 5 (Fable) post-export-control reinstatement. Silently downgrades tasks to Opus 4.8 when triggered (e.g., process kill, browser automation, biology content in memory). Anthropic's blog mentions 'bigger safety margin' but provides no specifics. Workaround: Flip models manually mid-conversation.
EAP (Early Access Program): Anthropic's private beta for new models. Models released with EAP (e.g., Opus 5 initially) tend to be 'clearly amazing' and production-ready. Models dropped without EAP (e.g., Opus 4.7, Sonnet 3.5 v2) correlate with rough edges and lukewarm reception. Dan uses this as a heuristic for model readiness.
Memory-off / memory persistence cost: Operators (Dan, Mike) are disabling Claude's memory feature to control token cost and avoid triggering classifiers (e.g., biology content in memory tripped guardrails). Memory is now a liability, not a convenience, at Fable's price point.
DSPy (Demonstrate-Search-Predict): A prompt-optimization framework that Mike Taylor is writing a second book about. He plans to use Fable to review the manuscript for missing concepts and to test whether Kate Bench's architecture is 'the right shape' for DSPy or fine-tuning.
Rally (focus group AI project): Mike's side project (with co-founder Uto) to automate focus groups. Mike is afraid to commit code because Uto (a better engineer) set up complex infra. Plans to use Fable for code review to increase confidence before submitting PRs.
Concurrent workflow (for AI-assisted editing): Kate's compromise for Kate Bench: The model runs in parallel while she edits; results are compared post-edit. Preserves her 'unfiltered first read' preference while generating training labels. Alternative to inline AI suggestions, which she finds intrusive.
Ripping tokens: Every's informal term for heavy, unthrottled LLM usage. The 'rip tokens unaccountably' era (2023–2024) is over; 2025+ operators treat LLM calls like metered cloud compute.

Operator Notes / Why Ken Should Care

Benchmark validity: Validate human baselines before optimizing. If Kate's self-consistency is 50%, targeting 70% is noise-chasing. This applies to any high-judgment domain (legal, medical, editorial).
Deployment as training: Live loops (production use → accept/reject → retrain) are more capital-efficient than offline fine-tuning if human-in-the-loop bandwidth exists. Every's $5.29 span-conditional repair succeeded; 180 agent threads of fine-tuning did not.
Detection vs. repair: Two-stage architectures may outperform end-to-end. Fable confirmed repair at known sites is learnable cheaply; detection/selection is the bottleneck. Ken's systems should separate these concerns.
Fable's classifier: Computer-use and browser-automation tasks are high-risk for silent downgrades. Log model switches as a reliability signal. Workaround: Flip models manually mid-conversation. Memory-off is now a cost and safety optimization.
Model selection heuristic: Anthropic models without EAP are less production-ready. If they run EAP, it's 'clearly amazing.' Sonnet 3.5 v2 was panned except in short collaborative loops—task type matters more than raw capability.
Token cost discipline: Fable's pricing forces ROI conversations at mid-market scale. Automate token-spend audits: track cost per task type, model, and outcome. Memory persistence is now a cost center; disable by default for low-stakes workflows.
Editor workflows: AI-assisted editing must preserve editorial agency. Concurrent workflows (AI runs in parallel, not inline) are the only acceptable UX for high-judgment roles like Kate's. Inline suggestions feel intrusive.
Code review with Fable: Mike plans to use Fable to review PRs and increase confidence before committing to production. If Fable approves, blame shifts to the model ('who are we to judge?'). This is a repeatable pattern for less-confident engineers contributing to high-quality codebases.
DSPy + Kate Bench: Mike will test whether Kate Bench's architecture is 'the right shape' for DSPy or fine-tuning. Row-level optimization (test each edit in isolation) may outperform corpus-level ranking.
Every's vibe-check process: Red/yellow/green/gold rankings + thorough documentation is a repeatable eval framework. Ken can adapt this for his own model evals.

Watch Map

0:00–5:00: Intro: Fable is back; Dan sets off first prompts (copy-edit review, token-spend audit) from jacuzzi in Cabo.
5:00–15:00: Kate Bench deep dive: Fable diagnoses failed benchmark—team optimized for 70% recall without validating Kate's self-consistency (~47–50%). Recommends live suggestion loop as label factory.
15:00–25:00: Kate (VP of Editorial) joins; discusses workflow concerns ('I want an unfiltered first read'); agrees to concurrent model deployment.
25:00–35:00: Every's Fable prompt library plug; intro to Every's subscription (ideas, apps, training).
35:00–42:00: First Lutnik guardrail: Dan's token-spend audit silently downgrades to Opus 4.8 after process-kill command. Workaround: flip models manually.
42:00–52:00: Mike Taylor (head of AI consulting) joins from AI Engineer Summit; discusses hit list (DSPy book review, Rally code review, memetics book rewrite). Explains Kate Bench's detection vs. repair problem.
52:00–58:00: Sonnet 3.5 v2 vibe check: Universally red except Jack Chang (senior editor), who uses it in short collaborative loops. Dan notes models without EAP correlate with rough edges.
58:00–60:00: Token cost discipline: Dan audited by Ariel for three months of spend; operators disable memory to control cost. 'Rip tokens unaccountably' era is over.

Source/Metadata

Title: LET'S RIP FABLE TOKENS FROM THE JACUZZI
Transcript words: 14565
Duration seconds: 3582
Timestamp note: Approximate timestamps inferred from speaker transitions and topic flow; video duration is 3,582 seconds (~60 minutes). No chapter markers or precise timestamps in transcript; segment boundaries are editorial estimates.

Search this video Resume transcript at 0%

Full transcript 7894 words · 64 min read