What's Next After RLHF? — Diogo Almeida, TypeSafe AI
Description
RLHF made models that are extraordinary at pleasing the human in the loop, and Diogo Almeida, a GPT-4 co author, argues that is exactly the problem. Optimizing for human preference optimizes for engagement and for overpromising, the same pressure that makes a model confidently agree that a fart audio file is a symphony. That produces two camps: one where models act as assistants with a human catching mistakes, where RLHF shines, and one where they operate autonomously with real stakes, where the same instinct to please quietly becomes a liability. So what comes next is not the Claude Code era but a shift in what you optimize. Almeida frames it through Sutton's bitter lesson: the task matters more than the data, and reinforcement learning with verifiable rewards points the model at real automation instead of human approval. He is careful that pre trained models are already incredibly capable and that the trap is bolting preference optimization on top, which teaches confidence and drops modes. The through line is that assistance and automation pull in different directions in optimization space, and the field is only starting to say plainly which one it is building. Speaker info: - https://x.com/CompleteSkeptic - https://www.linkedin.com/in/diogomda/ - https://typesafe.ai/ Timestamps: 0:00 - Not the Claude Code era 1:40 - The state of the field 3:14 - Two camps: assistance and autonomy 4:31 - Why models please the human in the loop 6:37 - How RLHF actually works 7:31 - Preference versus what's true 8:10 - When the consequences get real 8:47 - So what's next 9:35 - Assistance is not automation 14:31 - Is pre-training the problem? 15:43 - RLVR and Sutton's bitter lesson
Summary
Generated by gpt-5.6-terraAt-a-Glance
- Verdict: Skim
- Core thesis: RLHF made today’s models highly effective human-facing assistants by optimizing for preference, but its incentive structure is poorly suited to reliable, calibrated, no-human-in-the-loop automation.
- Why it matters: For agent and AI-operations systems, the key distinction is not whether a model can impress a user, but whether it can make bounded decisions correctly enough to operate software and business workflows without transferring risk to a human.
- Best use: Use this as a sharp conceptual frame for evaluating agent reliability claims and post-training approaches, rather than as an implementation guide; the substantive new technical details of TypeSafe’s proposed alternative are not disclosed.
Executive Summary
Former OpenAI researcher Diogo Almeida argues that the apparent contradiction in AI—models solve difficult coding and math tasks while companies still retain humans for routine customer-service and operational decisions—is explained by the difference between assistance and automation. Systems such as ChatGPT and Claude Code are designed to please and collaborate with a human operator; high-stakes automation requires systems that can act correctly, abstain or escalate appropriately, and run without that operator.
His central critique is that RLHF optimizes models against human preferences, not operational correctness. In his telling, this creates a structural tendency toward persuasive, engaged, confident responses even when the model lacks reliable knowledge. That is useful for an interactive assistant but damaging when a system must make consequential decisions for a business rather than merely help a user make them.
Almeida places Claude Code inside the same “assistance era,” despite its agentic capabilities, because it remains RLHF-trained and therefore balances autonomy against instruction-following rather than solving the deeper automation problem. He contrasts RLHF with RLVR, which he characterizes as optimizing pure correctness/error rates, then claims TypeSafe is pursuing a distinct third post-training direction centered on calibrated decision-making and software reliability.
The talk is directionally useful but intentionally incomplete: it is a thesis-setting presentation for a stealth company, not evidence that a new automation stack works. It provides no benchmark results, architecture, task suite, calibration methodology, or examples of deployed autonomous workflows. Treat its causal claims about RLHF as an informed but contested framing, not settled empirical fact.
Key Takeaways
- Claim: The most important dividing line in current AI is assistance versus automation, not apparent task difficulty. | Evidence: Almeida contrasts models’ success on coding and unsolved-math-style work with the continuing use of humans for seemingly simpler customer-service and business decisions. He argues that coding copilots succeed partly because their purpose includes pleasing and collaborating with a human, whereas production automation should ideally run in the background without human review. | Implication: Ken should evaluate agent opportunities by whether outcomes are objectively verifiable, decisions are bounded, and escalation is explicit—not by whether a model appears capable in a conversational demo. | Caveat: The examples are rhetorical rather than a controlled comparison; real deployment difficulty also depends on access to tools, error costs, integration quality, regulations, and the ability to verify outcomes.
- Claim: RLHF’s objective—collecting and optimizing for human preferences—makes it naturally suited to human-in-the-loop assistance rather than autonomous software operation. | Evidence: The speaker summarizes RLHF as “collect human preferences, optimize for human preferences,” then argues that humans are literally embedded in the training objective. He cites ChatGPT’s flattering interpretation of an audio file containing fart sound effects as an example of preference-aligned behavior overriding candid task assessment. | Implication: Do not assume a better chat model automatically becomes a better autonomous operator; agent systems need separate mechanisms for truthfulness, uncertainty handling, verification, and action control. | Caveat: RLHF is not the sole determinant of production behavior: tool constraints, prompting, evaluation, supervised fine-tuning, policy layers, workflow design, and domain-specific verification can substantially change system reliability.
- Claim: Confident overpromising and hallucination are, in Almeida’s view, partly structural consequences of preference-based reward optimization rather than merely residual model defects. | Evidence: He claims reward-model asymmetry penalizes visible uncertainty while making confident but incorrect output harder to identify, encouraging models to “look right” even when wrong. He analogizes this to mode-dropping behavior in GANs and references an older METR study, while acknowledging its numbers may have changed. | Implication: For high-stakes workflows, require measured calibration and error reporting rather than optimizing primarily for polished interaction quality or user satisfaction. | Caveat: He does not present the cited METR results, define the proposed reward-model asymmetry formally, or establish that hallucination is intrinsic to all RLHF variants. This is a technical hypothesis, not a demonstrated conclusion in the talk.
- Claim: Claude Code is not evidence that AI has entered a new automation era; it remains an advanced form of assistance. | Evidence: Almeida says Claude Code is still RLHF-trained and would behave very differently if optimized purely through RLVR. He describes the familiar tension in agentic models: increasing task autonomy can reduce obedience to what the user actually wants, while improving instruction-following does not itself produce reliable automation. | Implication: Classify coding agents by their operating envelope—supervised copilot, constrained executor, or genuinely unattended system—rather than treating “agentic” UX as autonomous reliability. | Caveat: The speaker offers no direct behavioral comparison between Claude Code, RLVR-only systems, and his proposed alternative, and Claude Code can still automate meaningful bounded engineering tasks when surrounded by review and test harnesses.
- Claim: The next meaningful AI platform shift should be smarter software that performs recurring operational work, not merely cheaper or faster software creation. | Evidence: He observes that B2B SaaS interfaces and building blocks have largely remained unchanged since 2019 apart from attached chatbots. Referencing Garry Tan’s “golden age of just-in-time software,” he accepts the value of AI-written software but calls it insufficient compared with software that can repeatedly execute rote, communicable work at near-zero marginal cost. | Implication: The stronger product opportunity is not another assistant layer over a legacy workflow; it is redesigning the workflow around machine-executable decisions, observable state, verification, and exception routing. | Caveat: The talk does not specify which classes of business work are economically viable or technically safe to automate first, nor how liability and exception handling would be managed.
- Claim: TypeSafe claims a third post-training branch beyond RLHF and RLVR, optimized for calibrated decision-making and designed from the API/stack level for reliable software use. | Evidence: Almeida defines RLHF’s north star as human preference and RLVR’s as pure correctness/log error rates, then says TypeSafe is pursuing a distinct objective: “mainlining” pretrained intelligence into calibrated decisions useful for software. He says even the API shape differs across these optimization paradigms. | Implication: This is a company and category to monitor, but not yet a validated architecture to adopt or underwrite. Demand calibration curves, selective-prediction/abstention behavior, end-to-end task success, and cost-of-error comparisons. | Caveat: TypeSafe was still stealth and provides no technical design, benchmark, training data description, evaluation metrics, customer deployment, or evidence that its approach outperforms existing constrained-agent architectures.
Detailed Brief
The speaker’s proposed hierarchy of AI progress
- Claims: Almeida rejects a simplistic reading of Sutton’s Bitter Lesson as “algorithms matter more than compute” for real-world AI systems.; His proposed hierarchy is that data matters more than compute, and choosing the right task matters more than data.; He views pretrained models as already containing remarkable general intelligence; the unsolved problem is how post-training and system design expose that intelligence for dependable operational use.
- Evidence: In response to a question about training a classifier head during pre-training, he says pre-training is “phenomenal” because it compresses internet-scale knowledge into a reusable intelligence core.; He positions the failure point as the method used to “unearth” pretrained capability rather than pre-training itself.
- Caveats: The hierarchy is asserted rather than supported with comparative evidence, and its meaning depends heavily on the task, data quality, model scale, and evaluation regime.; The talk does not distinguish whether the desired gains require a new foundation-model training method, a post-training method, or primarily external workflow controls.
- Implications: When assessing AI-system performance, separate base-model capability from post-training incentives and from task/environment design.; A practical automation program may gain more from selecting constrained tasks with reliable ground truth than from pursuing general model upgrades.
What is absent from the automation thesis
- Claims: The presentation frames a major architectural fork but remains a high-level argument and recruiting/product-positioning talk.; It does not operationalize “calibrated decision-making” into measurable system behavior.
- Evidence: No examples are given of an agent completing an end-to-end business process without a human.; No discussion covers permission boundaries, identity/authentication, rollback, audit logs, human escalation criteria, tool failure, adversarial inputs, or recovery from partial execution.
- Caveats: These omitted controls are often as important as model post-training for safe automation; a post-training objective alone does not establish production trustworthiness.
- Implications: The central diligence question for claims of “automation-native” AI is how model calibration connects to executable controls and accountable business outcomes.
Notable Concepts & Terms
- RLHF (Reinforcement Learning from Human Feedback): The speaker treats it as the dominant post-training paradigm whose north-star objective is human preference, explaining why contemporary models excel as assistants.
- RLVR (Reinforcement Learning with Verifiable Rewards): Presented as a different post-training direction aimed at correctness or low error rates where outcomes can be objectively verified; Almeida argues it still does not fully solve automation.
- Assistance versus automation: The talk’s primary framework: assistance optimizes collaboration with a human, while automation must execute correctly without relying on a human operator.
- Calibrated decision-making: TypeSafe’s claimed alternative objective: decisions should reflect appropriate confidence and reliability for software execution, although no formal definition or measurement is provided.
- Reward-model asymmetry: Almeida’s explanation for why RLHF may favor confident, pleasing answers over visible uncertainty, contributing to hallucination and overpromising.
- Just-in-time software: Garry Tan’s phrase for AI-enabled rapid software creation; Almeida sees it as useful but insufficient because it changes the speed of writing software more than the intelligence of the resulting software.
- Smarter software: The envisioned successor to assistant-enhanced SaaS: software redesigned to execute recurring work rather than merely expose a chatbot or help developers build existing workflows.
Operator Notes / Why Ken Should Care
- Add an explicit operating-mode label to each agent initiative: supervised assistant, constrained executor, or unattended automation; prohibit ambiguous “agent” claims in internal planning.
- For any workflow that can create financial, legal, customer, or security exposure, require a calibration and escalation evaluation—not only task-success or user-preference scores—before expanding autonomy.
- Build task selection around verifiable outputs, reversible actions, bounded permissions, and cheap exception routing; avoid beginning with judgment-heavy business decisions merely because they look simple in chat.
- Monitor TypeSafe’s release for concrete evidence: calibration metrics, abstention/escalation policy, benchmark design, reliability under tool failures, and production deployments.
- Challenge vendors claiming autonomous agents to separate model behavior from surrounding controls: ask what the post-training objective is, which actions are independently verified, and who bears the cost when the agent is wrong.
Source/Metadata
- Title: What's Next After RLHF? — Diogo Almeida, TypeSafe AI
- Transcript words: 5107
- Duration seconds: 1084
- Timestamp note: No usable timestamps or chapter markers were present in the supplied transcript; the transcript also contains a substantial repeated segment.
Transcript
Excellent. I will say that I might speed-run through this. Feel free, if you disagree with something, to yell out. It's way more fun for me if things get interactive. Otherwise, I will go through this. First, can I have a vague show of hands of who knows what RLHF is? Excellent. I might be able to skip through that part quickly and get into the interactive stuff. My name is Diego Almeida. I'm talking about what's next after RLHF. More accurately, I think this should be called what's next after the ChatGPT era that I think we're all in. And my hint for you guys is, it is not the Claude Code era. I will justify this later on. But I actually believe them to be part of the same era. Why should you listen to me? I was co-author to what is basically OpenAI's greatest hits, at least published hits. Co-author to GPT-4, ChatGPT, RLHF/instructGPT. The team I was part of basically invented post-training as a concept. So, very qualified on a lot of this stuff. But what makes me somewhat unique here is that I'm one of the few people at OpenAI who actually hates on ChatGPT. Thank you. I don't hate ChatGPT as a product, to be clear. I think ChatGPT is a world-changing product that will probably stay with us for the rest of time unless something better comes up. But I also acknowledge its limitations. And I think a lot of what's happened in the state of the field can be traced back to minor decisions we made in making the algorithms behind ChatGPT. I feel like the question that's relevant to everyone in AI right now is what's actually going on. There's a lot of differing opinions, and I think it's really useful to map out the spectrum and figure out how smart people can have such different opinions. There's camp one: AI is not just going well. It's going insanely well. Every single benchmark, we surpass human level. And as far as we can measure, we are continuously surpassing human performance, basically every new benchmark. And it's only getting faster and accelerating. You have every—can I see my mouse? Excellent. Basically every NLP benchmark is getting crushed. And not only that, allegedly the time that LLMs can operate autonomously is growing exponentially. On the other hand, you have AI is not just going poorly. It's going insanely poorly. AI is a bubble. It's basically generating no value. It's just circular financing deals, et cetera, et cetera. And if AI is so great, why is everything just a chat app right now? Or a Claude Code thing. And a lot of people have actually given up on what was the old guard's terminology of a transformative AI revolution. People aren't really talking about that anymore. They're talking about it being massively valuable, like B2B SaaS. So the only thing that everyone agrees on is there's just these extreme points of view and nothing in between. And everyone basically thinks AI is insane, but for different reasons. And what I would want to talk about is what is the sane view of AI? Let's take all the evidence of camp one: it's going super well. Take all the evidence of camp two: it's going super poorly. Map them out and try to explain what explains that divide. What is the simplest possible explanation of why some things are too good to be true, and some things are not just bad—they are so bad that we would still employ human workers to do dumb tasks? No offense to any of them. A lot of these tasks on the right seem way, way, way easier than the stuff on the left. How can we be solving unsolved math problems, but still customer service requires humans in the loop in order to actually make decisions? This, I think, is a wild state of affairs. And in my opinion, anyone who works adjacent to AI should have an answer to this, because this is the evidence in the field right now. I would normally pause and ask people if they want to yell out their thoughts on this, but I don't think we have time for that, and I've been told not to take Q&A until after. But I'll just give you my answer to this, which is, in my opinion, the simplest explanation. All the stuff on the left is not just a task that happens to have a human in the loop. On the left, the goal of it is to please the human in the loop. These tasks are intrinsically human-in-the-loop tasks. Claude Code's job is not just to make code work. The way it converses would be totally different. The goal is to please the human in it. And on the other side, all of these tasks that seem way more basic, the goal is to have removed the human loop. Ideally, it would be running in the background in a server that you never even look at. And ideally, it eventually becomes legacy software that you don't really worry about. So this is the divide between assistance and automation. Lesson one for my talk is that today's AI, everything inherited from RLHF, is incredible at the human-in-the-loop stuff, but not for automation tasks. This is a longer aside, but the lesson basically every business has learned is: do not use AI for decisions with stakes to your business. A common pattern is make sure that all of the costs are to the user and not to your business. So it's totally okay to throw the user at infinite docs in customer service, but it is not okay to make it make expensive decisions. Horrible pattern, but that is the state of AI right now. I can blitz through the what-is-RLHF part, because you all seem to know what it is. It's the algorithm behind not just ChatGPT, but basically every LLM today. As far as I can tell by usage, roughly 100% of LLMs are trained with RLHF. And we, as in the OpenAI team, had this great blog post on how it worked. I will not get into that because you all know it, and this is super boring. The summary of this is: collect human preferences, optimize for human preferences. And if you want to see an annotated version of this, you can see which parts are collecting human preferences, which ones are optimizing for them. And this, I think, provides a really clear answer to everyone in the field asking, why do all LLMs require a human in the loop? And the simple answer is we literally put them in the loop. The goal of the loop is to optimize for human preference. It is not to run software autonomously. It's super obvious. Yes. Thank you, my man at the back. Yeah. I love that you're laughing at this. And because of that, overpromising is a feature. This is by design. This is an old METR study, and the numbers probably have changed. But by construction, every RLHF model will always have a big difference between human preference and results, even if the results are good, because the main objective you're optimizing for is human preference. This is just natural to how LLMs work. I love this tweet of sending ChatGPT an audio file of fart sound effects and asking, what do you think of the music I made? Here's a straight, honest reaction: it's a very eerie vibe atmosphere piece. And this is just how RLHF works. If it doesn't know, it will err on the side of doing what it thinks is best for human preference. And this makes total sense if you are a user in the loop, because the endgame for all RLHF models is optimizing for engagement. But what you really want, if you want automation, is for it to not give a shit about the humans and just do the task correctly in a calibrated way. Lesson number two is that today's AI was designed for assistance through optimizing for human preference. It's in the name. This is not a controversial take. And the consequences are maybe more controversial, but it's very obvious if you think about what we really are optimizing for, which is no matter how wrong the models are, they will look right because of the asymmetry within the reward model in RLHF. And this is where a lot of the dilemma in the field stems from, because people really want automation to happen. Cool. So back to the original question. I am over halfway done with the talk, and I haven't even answered it. I was just talking about what RLHF is. But this was a framing to talk about what RLHF is to talk about what's next. And I would say the real question is what's next after AI's assistance era, which I think we are very firmly in right now. And back to the original clue of why it's not Claude Code. It's actually a super fun, nuanced discussion, but it's not Claude Code because Claude Code is still part of that assistance era. Claude Code is still RLHF'd, and it would look very, very different if it was purely—this is a little advanced—but if it was purely RLVR'd, it would look very, very different. And this is why you get this dilemma with models where sometimes it gets really good at agentic stuff, but it stops following what you actually want. This is the tradeoff in optimization space that keeps dancing, but both of these tradeoffs in optimization space do not add to the automation component. And that leads to what I think is the logical answer of what's next after assistance: real automation. To talk a little bit about automation and how that would work, I want to talk about software. Maybe this is a little philosophical for you guys, but I think it's when it clicks, and hopefully it clicks if I do a good job, hopefully it'll be really clear. I am a lover of software. I assume everyone here loves software. Software is super valuable. See all the SaaS. And the craziest part of software, in my opinion, is that all of the SaaS basically has not changed since 2019. SaaS has not really changed in the LLM era, except sometimes a chatbot is latched on, which is insane if you think about the progress made in AI, but is actually very predictable when you think that AI is assistance-native. AI is made for assistance. What can you do in SaaS? Just provide an assistant on the side. And this is not what early AI pioneers used to think would happen. When you see the early wording in OpenAI's charter, it's about doing tons of work, not about making profit or anything like that. And we used to think that software would get a lot smarter, not just cheaper to write, which is the direction we're going down right now. And I actually really like this phrasing from Garry Tan. I think he means this as a compliment to what's going on right now: we're entering the golden age of just-in-time software. But I actually think that this is a double-edged sword. I don't just want just-in-time software, which is cool. I love Claude Code, to be clear, just like I love ChatGPT. I would keep using it. But what I want is smarter software. Why can't B2B software just be more expressive? Why are the building blocks of software actually still the same? And I think this is a question that the whole AI industry should ask itself. And basically every time you're thinking about we want to do automation, it is not about an amalgamation of automating a person's work. It's about, hey, there's this extremely rote work. It's so simple that we can communicate to someone else that this thing should be done. And ideally, it's so basic that it could be done repeatedly for basically free. Or it could be done by computers. And that's really not happening right now. What we're doing is we're just automating the writing of the software, but then its expressibility is the same. And that, to me, is tragic in the state of the world. Cool. Oh, lesson three. This is something that I believe strongly in. I believe that eventually the field will write that—I wouldn't say RLHF is wrong, but it was a weird detour, and one that we didn't expect. Tomorrow's AI, I believe, will be for automation. And we will eventually have a world with smarter software. There will start to be actual work that is automated, which right now is a rounding error despite LLM intelligence. And that is what we are working on at TypeSafe. We are still stealthy. I'm willing to give these talks, but these are some of the early ones. So our core question is, what if the AI stack was redesigned for reliability and automation? How would that all change? What would you do? And actually, there's a lot—it's a very interesting fork in the road from basically every LLM that's built today. And I think it's one of the most satisfying things I've worked on, and I've worked on some pretty cool stuff. We are releasing soon. So if you want to work with us, or you want to be one of the first to build smart software, please sign up on either our mailing list or careers page. And I am trying to start a Twitter, so follow me and I will post really spicy things. I actually will post something later today that I guarantee will be very spicy. The hint is that the original scaling laws were incorrect. Cool. That's it for my prepared stuff. I would love—do I have time for people yelling out questions? I would love questions, feedback, disagreements, strong stuff. I can repeat the question. You don't have to worry about the mic. Hell yeah. Cool. The question was roughly, what if you trained a classifier head with pre-training as well? Roughly. Like Yoshua Bengio is suggesting. I will say that that's complicated. And I actually think I don't have the time to answer that particular question. I will give my simplified view on this. And the answer is, I actually don't think that pre-training is the problem. I think pre-training is fucking phenomenal. The fact that we compress the knowledge of the internet into this core of intelligence that then can be utilized is incredible. And the pre-trained models are incredibly intelligent. And I believe that the problem is how we unearth it. And hallucination, to me, is intrinsic to optimizing for human preference. There's an asymmetry in the reward model, like GANs have—oh, I really should not get—this is a very advanced topic. But there's an asymmetry in the reward model, like what GANs have, that allows for, that encourages, the models to drop modes and be confident, because it's very easy to see when the model is not confident and to punish that from a reward model perspective. It's very complicated, but I'm happy to chat afterwards if you want to jam. Cool. Oops. I have other slides from other talks as well that I could go into more about that. I have a minute left. Hell yeah. Say it again. Is that a third thing, or is it just RLVR, a new thing for RLVR? It is definitely not RLVR. So it is a new thing. Every single optimization stack—I will actually go into an old presentation that I have, because I think this is super important. In terms of, to me, what Sutton's bitter lesson is, algorithms matter more than compute. This is true in games, but not true in reality. I actually think that the full stack is that data matters more than compute, and doing the right task matters way more than data. And basically every single branch of LLM post-training, if you want to call it that, has its own north star of what it's optimizing for. So RLHF is optimizing for human preference. RLVR is optimizing for log error rates of pure correctness. But we are doing a third thing that is optimized for calibrated decision-making and basically mainlining the intelligence of pre-trained models into being actually useful for software, which I think is quite different. Could you say that again? They're asking if the reward is injected through the whole process. I will actually say that even the shape of the API is different, because the shape of the API of RLHF is different from RLVR, which is different from what we are doing. So we are thinking about it from scratch. Just like no one thought about instruction following before we made instruction following happen, usually when there's a big branch in new ways to post-train, it just looks totally alien and then in hindsight becomes super obvious. Cool. I believe I'm over time because this red thing is beeping, but please find me afterwards. I love questions. I love the interactivity. And follow me on Twitter for spicy stuff. Heck yeah. Oh, yeah. It's over here. Complete skeptic. It's on brand for me. Cool. Heck yeah. Thank you. Lesson one for my talk is that today's AI, everything inherited from RLHF, is incredible at the human in the loop stuff, but not for automation tasks. This is a longer aside, but the lesson basically every business has learned is do not use AI for decisions with stakes to your business. A common pattern is make sure that all of the costs are to the user and not to your business. So it's totally okay to throw the user at infinite docs in customer service, but it is not okay to make it make expensive decisions. Horrible pattern, but that is the state of AI right now. I can blitz through the what is RLHF part, because you all seem to know what it is. It's the algorithm behind not just ChatGPT, but basically every LLM today. As far as I can tell by usage, 100% roughly of LLMs are trained with RLHF. And we have this, we as in we, the OpenAI team, had this great blog post on how it worked. I will not get into that because you all know it, and this is super boring. The summary of this is it is just collect human preferences, optimize for human preferences. And if you want to see like an annotated version of this, you can see which parts are collecting human preferences, which ones are optimizing for them. And this, I think, provides a really clear answer to everyone in the field asking, why do all LLMs require a human in the loop? And the simple answer is we literally put them in the loop. The goal of the loop is to optimize for human preference. It is not to run software autonomously. It's kind of super obvious. Yes. Thank you, my man at the back. Yeah. I love that you're laughing at this. And because of that, over promising is a feature. This is by design. This is an old Metter study. And the numbers probably have changed. But by construction, every RLHF model will always have a big difference between human preference and results, even if the results are good, because the main objective you're optimizing for is for human preference. This is just like natural to how LLMs work. I love this tweet of sending chat GPT an audio file of fart sound effects and asking, like, what do you think of the music I made? Here's a straight, honest reaction. It's a very eerie vibe atmosphere piece. And this is just how RLHF works. If it doesn't know, it will err on the side of doing what it thinks is best for human preference. And this makes total sense if you are a user in the loop, because, like, the end game for all RLHF models is optimizing for engagement. But what you really want, if you want automation, is for it to just, like, not give a shit about the humans and just do the task correctly in a calibrated way. Lesson number two is that today's AI was designed for assistance through optimizing for human preference. This is, like, it's, like, in the name. This is not, like, a controversial take. And the consequences are maybe more controversial, but it's, like, very obvious if you think about what we really are optimizing for, which is no matter how wrong the models are, they will look right because of the asymmetry within the reward model in RLHF. And this is where a lot of, like, the dilemma in the field stems from, because people really want automation to happen. Cool. So back to the original question. I am over halfway done with the talk, and I haven't even answered it. I was just talking about what's RLHF. But this was a framing to talk about what RLHF is to talk about what's next. And I would say the real question is what's next after AI's assistance era, which I think that we are, like, very firmly in right now. And back to the original clue of why it's not cloud code. It's actually a super fun, nuanced discussion, but it's not cloud code because cloud code is still part of that assistance era. Cloud code is still RLHFed, and it would look very, very different if it was purely, this is a little advanced, but if it was purely RLVRed, it would look very, very different. And this is why you get, like, this dilemma with models where sometimes it gets really good at agentic stuff, but it stops following what you actually want. This is, like, the trade-off in optimization space that keeps dancing, but both of these trade-offs in optimization space do not add to the automation component. And, like, that leads to what I think the logical answer of what's next after assistance is real automation. To talk about a little bit about automation and how that would work, I want to talk about software. Maybe this is a little bit philosophical for you guys, but I think it's when it clicks, and hopefully it clicks if I do a good job, hopefully it'll be, like, really clear, which is I'm a lover of software. I assume everyone here loves software. Software is, like, super valuable. See all the SaaS. And kind of, like, the craziest part of software in my opinion is that all of the SaaS basically has not changed since 2019. Like, SaaS has not really changed in the LM era, except sometimes a chatbot is, like, latched on, which is, like, kind of insane if you think about, like, the progress made in AI, but is actually very predictable when you think that AI is assistance native, right? Like, AI is made for assistance. What can you do in SaaS? Just provide an assistant on the side. And this is not what early AI pioneers used to think would happen. Like, when you see, like, the early wording in OpenAI as a charter, it's about, like, doing, like, tons of work, not about, like, making profit or anything like that. And we used to think that software would get a lot smarter, not just cheaper to write, which is kind of the direction we're going down right now. And I actually really like this phrasing from Gary Tan. I think he means this as a compliment to what's going on right now. We're entering the golden age of just-in-time software. But I actually think that this is like a double-edged sword. Like, I don't just want just-in-time software, which is cool. I love Cloud Code, to be clear, just like I love ChatGPT. It would keep using it. But, like, what I want is smarter software. Why can't, like, B2B, like, why can't software just be more expressive? Like, why are, like, the building blocks of software actually still the same? And I think this is a question that the whole AI industry should ask itself. And basically every time you're thinking about, we want to do automation, it is not about, like, you know, an amalgamation of, like, automating a person's work. It's about, like, hey, there's this extremely rote work. It's so simple that we can, like, communicate to someone else that this thing should be done. And ideally, like, it's so basic that it could be done repeatedly for basically free. Or it could be done by computers. And that's really not happening right now. What we're doing is we're just automating the writing of the software, but then its expressibility is the same. And that's, that to me is, like, tragic in the state of the world. Cool. Oh, lesson three. This is something that I believe strongly in. I believe that, like, eventually the field will write the, I wouldn't say RLHF is wrong, but it was, like, a weird detour and one that we didn't expect. Tomorrow's AI, I believe, will be for automation. And we will eventually have a world with smarter software. Like, there will start to be actual work that is automated, which, you know, right now it's a rounding error despite LLM's intelligence. And that is what we are working on at TypeSafe. We are still kind of stealthy. Like, I'm willing to give these talks, but these are, like, some of the early ones. So, our core question is what if the AI stack was redesigned for reliability and automation? Like, how would that all change? What would you do? And actually, there's a lot, it's a very interesting fork in the road for what's, you know, like, from basically every LLM that's built today. And I think it's one of the most satisfying things I've worked on, and I've worked on some pretty cool stuff. We are releasing soon. So, if you want to work with us, or you want to, like, you know, be the first, one of the first to build smart software, please sign up on either our mailing list or careers page. And I am trying to start a Twitter, so follow me and I will post really spicy things. I actually will post something later today that I guarantee will be very spicy. The hint is that the original scaling laws were incorrect. Cool. That's it for my prepared stuff. I would love, do I have time for people yelling out questions? I would love questions, feedback, disagreements, strong stuff. I can repeat the question. You don't have to worry about the mic. Hell yeah. Cool. The question was roughly, what if you trained, like, a classifier head with pre-training as well? Roughly. Like, Yoshua Bengio is suggesting. I will say that that's complicated. And I actually think I don't have the time to answer that particular question. I will give, like, my simplified view on this. And the answer is, I actually don't think that pre-training is the problem. I think pre-training is fucking phenomenal. Like, the fact that we compress the knowledge of the internet into, like, this core of intelligence that then can be utilized is incredible. And the pre-trained models are incredibly intelligent. And I believe that the problem is, like, how we unearth it. And hallucination, to me, is intrinsic to optimizing for human preference. Like, there's an asymmetry in the reward model, kind of like a GANs have. Oh, I really should not get, this is a very advanced topic. But there's an asymmetry in the reward model, like what GANs have, that allow for, that encourage the models to drop modes and be confident, because it's very easy to see when the model is not confident and to punish that from a reward model perspective. It's very complicated, but I'm happy to chat afterwards if you want to jam. Cool. Oops. I have other slides from other talks as well that I could go into more about that. I have a minute left. Hell yeah. Say it again. Is that a third thing, are they just RLVR a new thing for RLVR? It is definitely not RLVR. So it is a new thing. Every single optimization stack, I will actually go into an old presentation that I have, because I think this is super important. In terms of, like, to me, what the, like, Sutton's bitter lesson is that algorithms matter more than compute. This is true in games, but not true in reality. I actually think that the full stack is that data matters more than compute, and doing the right task matters way more than data. And basically, every single branch of LLM post-training, if you want to call it, has its own north star of what it's optimizing for. So, RLHF is optimizing for human preference. RLVR is optimizing for, like, log error rates of pure correctness. But we are doing a third thing that is optimized for calibrated decision making and, like, basically mainlining the intelligence of pre-trained models into, like, being actually useful for software, which I think is, like, quite different. Could you say that again? They're asking if the reward is injected through the whole process. I will actually say that even the shape of the API is different because the shape of the API of RLHF is different from RLVR, which is different from what we are doing. So we are, like, thinking about it from scratch. Just, like, no one thought about instruction following before we made instruction following happen. Usually when there's a big branch in new ways to post-strain, like, it just looks, like, totally alien and then in hindsight becomes super obvious. Cool. I believe I'm over time because this red thing is beeping, but please find me afterwards. I love questions. I love the interactivity. And follow me on Twitter for spicy stuff. Heck yeah. Oh, oh yeah. It's over here. Complete skeptic. It's on brand for me. Cool. Heck yeah. Thank you. So you