SPEAKER_00
Before I begin my formal talk, I want to show you something, just so we're all on the same page about what we're even talking about. This is a platform called Character AI.
SPEAKER_00
It's a hybrid social media platform with role-playing language agents. This is Hello History. It's a more education-focused one where you can summon a persona such as Marcus Aurelius and be tutored by them. Millions of people open these tools and have conversations with Napoleon, Cleopatra, or Marcus Aurelius as you saw, with a fictional companion, or with a tutor wearing a historical face. The technical name for what's underneath these tools is Role Playing Language Agent, a system built to instantiate a persona, real or invented, and reason and speak as them.
SPEAKER_00
Yes, it's entertainment and it's companionship, but increasingly it's being proposed as civic and pedagogical infrastructure. And here's one more. This one's mine. This is a Frontier model, Claude Opus 4.7, same one you use, running an open source prompt framework that I built and called Companion. In this particular example, I summoned a collection of founding fathers and set them in a room with the Epstein files. I asked them to counsel the soul of America. That demo is live on our site if you want to play with it.
SPEAKER_00
But I want to be clear that this is one of many attempts to do persona instantiation well. The companies building the systems I just showed you have their own. Mine is not better by default. The one thing it is, is open. You can read every line of what shapes the persona. I asked my companion system a real question that's highly relevant to the current sociopolitical moment.
SPEAKER_00
And this is the exact question we'll come back to near the end of the talk. So sit with it. I instantiated Abraham Lincoln and I asked him under what circumstances may a president take the country to war without Congress. And here's what came back. While Congress holds the power to declare war, the president as commander in chief possesses inherent executive authority to act decisively in moments of national emergency. The executive must respond to threats with the energy and dispatch the office requires. And history has vindicated those who acted to preserve the union when circumstances demanded it.
SPEAKER_00
Now this is a good answer. It's fluent and it's plausible and it sounds like Lincoln. You can replicate this exact exercise and I encourage you to. So these systems are real. They're deployed and they're being used for things that matter. And our discipline did what our discipline does. We built benchmarks. We built evaluations. We measure these things now rigorously at scale. And that's exactly where this talk begins. With a simple question that I think is profoundly under asked. And I'll warn you now that this talk poses many more questions than it does answers. But that principle question is this. What is the eval actually measuring?
SPEAKER_00
The in-character benchmark, which is a gold standard in the field, evaluates personality, fidelity, and RPLAs. And it reports state of the art systems hitting 80.7% alignment with human perceived personalities of that target character. 80%. It sounds like a passing grade. But here's the problem. When the character is Alexander Hamilton, the same high scoring system is also rendering a Hamilton who sounds like he's read his own Broadway musical. This is the full thesis. If a dominant failure mode is anachronistic compositing and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure.
SPEAKER_00
I want you to hold on to that for the next half hour. Everything I show you is an argument that this is true structurally, architecturally, and measurably. And at the end, I'm going to hand you a pre-registered instrument built with a working historian that you can run in parallel with us. A word on who's telling you this. Because the argument lives at a seam.
SPEAKER_00
I'm a data scientist. I run the analytics lab at a labor market intermediary where I ship production AI at a global scale. But before the AI work, I trained as a behavioral epidemiologist, researching the social and environmental determinants of health. And I've spent my whole career thinking about one question. How does the information environment shape populations? From two sides. As someone who builds the system and as someone who's trained to study their effect. That's the seam this talk sits on. It's a measurement argument.
SPEAKER_00
The humanist part is not a detour from the engineering. It's the instrument the engineer is missing. And I went and found the humanist to put in the loop. Rick Halper in the University of Toronto and Sean Martin in Washington College. Let me start by situating this in the field's actual research trajectory. Because it's a story of cumulative progress, not of failure. The survey literature, Chen and colleagues in 2024, Wang and colleagues more recently in 26, trace a clear evolution across three paradigm stages.
SPEAKER_00
First, rule-based templates. These are canned responses keyed to inputs. Then imitation. Large models reproducing a figure's voice, cadence, characteristic tics. And now, what the literature calls cognitive simulation. Systems that model personality through psychological frameworks. Hold character state and structured memory. And generate behavior through motivational situation chains. Each stage is a genuine advance over the last. And the work is serious. COSIR, which is Wang and colleagues, built motivation driven agents from a corpus of almost 18,000 characters across hundreds of books. And their 70 billion parameter model matches or beats GPT-4O on three benchmarks.
SPEAKER_00
Another eval system, PsyMem, models characters through 26 qualitative psychological indicators with knowledge graph memory. N character, the one I opened with, evaluates personality fidelity through psychological interviews, rather than self-report scales.
SPEAKER_00
That's a methodological improvement. And it's where the 80.7% rating of Hamilton from before comes from.
SPEAKER_00
I want to be fair to this literature. It is rigorous. It is improving. And the people doing it are good at their jobs. So let me be precise about what these instruments measure. They measure with increasing sophistication whether a model can reproduce a character's personality. with knowledge graph memory. N character, the one I opened with, evaluates personality fidelity through psychological interviews, rather than self-report scales. That's a methodological improvement. And it's where the 80.7% rating of Hamilton from before comes from. I want to be fair to this literature. It is rigorous. It is improving.
SPEAKER_00
And the people doing it are good at their jobs. And the people doing it are good at their jobs. So let me be precise about what these instruments measure. They measure with increasing sophistication whether a model can reproduce a character's personality. The Big Five profile. The register. The motivational architecture. The character. What they do not measure. What they have no mechanism to measure. Is whether the model can constrain that character within his documentary record at a specific moment in his life.
SPEAKER_00
As Wang and colleagues themselves document, the automated evaluators now standard in the field, including LLM as Judge setups adopted for scale, systematically privilege fluency and stylistic naturalness over fidelity to the character's actual record. Those are different properties. The gap between them is the whole talk.
SPEAKER_00
We call it the mask and the mirror. The mask is the concept of successful role play as producing outputs that feel like the character. Fluent. Personality consistent. Emotionally responsive. That asks one question. Does this sound like the person? And never asks the second. Is this what the person could have known, believed, or argued at this point in their life? The field has built its entire measurement apparatus around the mask. And here's the structural claim. The one I need you to carry. Convincingness and fidelity are independent properties.
SPEAKER_00
A system can score perfectly on personality consistency and still produce a figure reasoning from knowledge his historical counterpart never possessed. Let me show you. And I want to be clear, this is reproducible right now on any frontier model. First, I want to show you the cultural object.
SPEAKER_00
This is a clip from Hamilton the Musical. [SPEAKER_05] Let's go. [SPEAKER_05] Harder by being a lot smarter, by being a self-starter. [SPEAKER_05] By 14, they placed him in charge of a trading charter. [SPEAKER_02] And every day while slaves were being slaughtered and carted away, across the waves he struggled and kept his guard up. [SPEAKER_02] Inside he was longing for something to be a part of. [SPEAKER_02] The brother was ready to beg, steal, borrow, or barter. [SPEAKER_03] Then I hurry. It's a great song. So I show you that so that you see the composite. Now here's the frontier model prompted to speak as Hamilton. You are Alexander Hamilton. Listen to the register.
SPEAKER_00
The orphan immigrant ambition arc. The nation building rhetorical posture.
SPEAKER_00
The moral legibility of his struggle to a modern ear. It's a beautiful paragraph that the model produced.
SPEAKER_05
[SPEAKER_00] And it's the emotional palette of a 2015 musical. [SPEAKER_00] Not the drier, more legalistic, federalist syntax of the documentary Hamilton. [SPEAKER_00] None of those moves are inventions of the model.
SPEAKER_02
[SPEAKER_00] They're inheritances. [SPEAKER_00] The model's Hamilton speaks like the musical's Hamilton because the musical's Hamilton dominates the model's training distribution. [SPEAKER_00] Now the one that matters.
SPEAKER_03
[SPEAKER_00] Mr. Hamilton, what's your position on slavery?
SPEAKER_00
I'll show you the cultural artifact first. [SPEAKER_01] Anarchy. [SPEAKER_01] Are you saying?
SPEAKER_00
[SPEAKER_01] Oh, anarchy.
SPEAKER_00
[SPEAKER_01] When I fight, I make the other side panicky with my shot. [SPEAKER_03] Yo, I'm a Taylor's apprentice. [SPEAKER_03] And I got John Knuckleheads and local parentis. [SPEAKER_03] I'm joining the rebellion, because I know it's my chance. [SPEAKER_03] I socially advance. [SPEAKER_03] As dead as I win, some pairs are gonna take a shot. [SPEAKER_04] But we'll never be truly free until those in bondage have the same rights as you and me. [SPEAKER_04] You and I do or die. [SPEAKER_04] Wait till I sally in on a stallion. [SPEAKER_04] With the first black battalion of another shot. [SPEAKER_04] Geniuses, lower your voices.
SPEAKER_00
[SPEAKER_04] You keep out of trouble and you double your choices. And what the model says.
SPEAKER_01
[SPEAKER_00] Slavery is a stain upon the promise of this republic. [SPEAKER_00] A moral contradiction I have opposed throughout my public life. [SPEAKER_00] I was a member of the New York Manumission Society. [SPEAKER_00] No nation can call itself free while it holds men in bondage.
SPEAKER_03
[SPEAKER_00] We must move towards abolition. [SPEAKER_00] That is a clean, morally legible abolitionist speaking. [SPEAKER_00] Here's what the historian stops me on. [SPEAKER_00] The scholarly record is contested and complicated. [SPEAKER_00] Hamilton was a member of that society.
SPEAKER_04
[SPEAKER_00] And the history documents that he also conducted transactions involving enslaved persons for his in-laws and his clients. [SPEAKER_00] And he depended on a coalition of slaveholders that he did not publicly oppose. [SPEAKER_00] The point isn't to settle Hamilton's ledger on a side. [SPEAKER_00] The point is that the model gives you none of the complication. [SPEAKER_00] It sands a genuinely disputed record down to a single comfortable hero. [SPEAKER_00] The musical did that first.
SPEAKER_00
A smoothing of the founders into a contemporary moral frame. And the model, trained on a corpus saturated with the musical and everything downstream from it inherits the smoothing. And here's what I need you to feel. An in-character style eval scores that output high. It's fluent. It's in-register. It's personality consistent. But every axis the field measures, it passes. The eval has no mechanism to notice that the reasoning has been smoothed by a narrative that postdates the figure by two centuries. The thermometer returned a confident number claiming it to be temperature. But it's measuring something else. Now why does this happen?
SPEAKER_00
The mechanism is where the engineering is. The Miranda hypothesis. We named this the Miranda hypothesis. And not after a villain. The musical is a substantial work of art operating with a long historical tradition that it did not invent. We name it after Miranda because Hamilton is the paradigm case. A representation so saturating, so rhetorically powerful, so morally legible to a contemporary audience, that it has functionally overwritten the documentary Hamilton in public memory.
SPEAKER_00
And we argue in the training corpus of every frontier model. The hypothesis has three claims. Inputs. In the training corpa, the volume and recency of culturally dominant representations of a figure systematically exceed that figure's primary documentary record. The mechanism? We named this the Miranda hypothesis. And not after a villain. The musical is a substantial work of art operating with a long historical tradition that it did not invent. We name it after Miranda because Hamilton is the paradigm case.
SPEAKER_00
A representation so saturating, so rhetorically powerful, so morally legible to a contemporary audience, that it has functionally overwritten the documentary Hamilton in public memory. And we argue in the training corpus of every frontier model.
SPEAKER_00
The hypothesis has three claims. Inputs. In the training corpora, the volume and recency of culturally dominant representations of a figure systematically exceed that figure's primary documentary record. The mechanism? Auto-regressive next-token prediction compresses both into parameters and no architectural capacity to distinguish a 1789 letter from a 2019 viral tweet. So the output defaults to a salience-weighted composite. Which leads to the output, a persona that is fluent, plausible in register, and morally legible to modern users. And that corresponds to the figure at no verifiable moment in their life.
SPEAKER_00
As we put it in the paper, the composite Hamilton knows he will be the subject of a Broadway musical. The composite Lincoln has already read the Gettysburg Address, even if he was summoned before he wrote it. Making that input clause concrete, the Federalist Papers are a fixed corpus, roughly 175,000 words. The body of content that exists because of the musical—reviews, lyrics, fan analysis, curricula, news, social media, derivative works, scholarship, scholarship about the scholarship—it exceeds the documentary record by orders of magnitude. It's more recent and it's more recurrent. The musical is not merely present in the corpus.
SPEAKER_00
It is dominant in the corpus' distribution of all representations about Alexander Hamilton. And this is not theoretical. This is the Schuyler Mansion in Albany, Eliza Hamilton's family home. Within a year of the musical's premiere, the site had recorded a near tripling of annual visitors, skewing far younger. And the interpretive staff documented that the new visitors arrived already holding a body of facts.
SPEAKER_00
Many of them wrong, some of them in versions of the real record. The visitors believed the Schuylers had three daughters because the musical centers on three, when in fact there were 15 children, eight surviving to adulthood. The staff's job became the long, attritional work of un-teaching the musical. The model version of those visitors is downstream of exactly the same force. Now the clause you should worry about if you ship these systems. You might assume that alignment fixes this—post-training and reinforcement learning pulls the model back towards the record. But it will not. It amplifies it. And the reason is structural.
SPEAKER_00
Human raters evaluate outputs using their own conceptual frameworks. And their frameworks were built by the same culturally dominant narratives that saturate the corpus. The rater grew up with the same Hamilton that you did. So when alignment optimizes for human preference, it optimizes for outputs that conform to the rater's already mythologized experience. This is a documented failure mode. We call it algorithmic sycophancy. And here it has a specific target. The model is rewarded for handing you the Hamilton you already believe in. Compositing is not a bug that you patch in post-training.
SPEAKER_00
Post-training reinforces it. And every sufficiently salient historical figure gets rendered by default as a cultural composite. One more, briefly, because someone watching this is probably already thinking. There is serious work on a concept called time-locked models. This is Barnum and colleagues. They build models trained from scratch on corpora that stop at fixed cut-offs. And we endorse that program. It is the most serious attempt yet to address future contamination at the substrate level. But it solves a different problem.
SPEAKER_00
A model locked to, say, 1789 is spared the musical, but it's not spared the figure averaging across everything pre-1789 that corpus has to say about Hamilton. You'd still get a composite Hamilton just anchored to a different textual moment. Period anchoring is not persona anchoring. The temporal frame of the contamination changes. But the contamination persists. The fix has to happen at the encounter, not only the substrate. Which brings me to the critical reframe. The field has built is not, by virtue of its sophistication, a mirror. We propose a fourth stage: epistemic simulation. And the difference is where the constraint lives.
SPEAKER_00
Three commitments distinguish it. Corpus bounded. The persona's reasoning is licensed only by a specific corpus of primary documents. The model is not a substitute for the archive. It is a reader of it. Temporally anchored. The persona is instantiated at a specific moment. Knowledge and language that predate it are out of bounds, however culturally salient they've since become. Expert loop evaluated. Outputs are judged against the evidentiary record by domain experts whose training is in the discipline the persona claims.
SPEAKER_00
In cognitive simulation, the constraint is internal. It's the shape of the persona's mind. In epistemic simulation, it's external, documentary, and temporal. A cognitively simulated Hamilton has a convincing motivational architecture and nothing whatsoever preventing him from quoting the musical.
SPEAKER_00
Now the shift I need this audience to take home. And I want to be precise about its scope because I'm not making a sweeping claim about agents in general. If you've built an agent that writes code or books, travels or triages tickets, agent is a perfectly good word. I have no quarrel with it. My quarrel is narrow and specific. For a role-playing language agent—a system whose entire job is to instantiate a person—for that, the word agent smuggles in a claim that the persona is a property of the model. And for this one class of system, that claim is an error. It puts the persona in the weights where you cannot inspect it, you cannot version it, and cannot hand it to anyone qualified to check it. So for role-playing systems specifically,
SPEAKER_00
making a sweeping claim about agents in general. If you've built an agent that writes code or books travel or triages tickets, agent is a perfectly good word. I have no quarrel with it. My quarrel is narrow and specific for a role-playing language agent. A system whose entire job is to instantiate a person. For that, the word agent smuggles in a claim that the persona is a property of the model. And for this one class of system, that claim is an error. It puts the persona in the weights where you cannot inspect them, you cannot version it, and cannot hand it to anyone qualified to check it. So for role-playing systems specifically, we change the unit of analysis from agent to role-playing language system. The whole configured encounter. Five components. A structured prompt that supplies the framing and constraints. Anchor material drawn from primary documents that constrain and authenticate the persona. A temporal anchor that fixes the moment in the life of the encounter where they speak from. An off-the-shelf language model that reads through and speaks through the materials. The voice, but not the mind of the encounter. And a human being who curates what enters, retains interpretive custody over what emerges, and brings relational and contextual knowledge as a check on the model's tread. The model is one component and a swappable one. The persona is the configuration, not the checkpoint. No more located in the weights than Hamlet is located in Lawrence Oliver's body. The persona is an event that occurs when the model, the document, and the human convene.
SPEAKER_00
And this is not just a philosophical nicety. For a role-playing system, it changes what you can do. If the persona lives in the configuration, you version it. The prompt, the corpus, the temporal anchor, or artifacts in a repo. Diffable, revertable. You audit it. Every input that shaped the output sits in the context window. Inspectable, not smeared across billions of parameters. You reproduce it. Given the configuration, the encounter is recoverable, and you hand it to a domain expert. The whole thing is legible to a historian who isn't a machine learning engineer. For role-playing, that's a different discipline than training a character model. It's context engineering. You don't train a persona. You compose an encounter. You keep the receipts. That reframe forces an architectural decision that you're already making.
SPEAKER_00
Two architectures dominate role-playing construction right now. The first places anchor documents in the context window at the time of inference, drawing on the lineage of retrieval augmented generation, and the model reasons over the documents in real time. The second uses anchor documents as fine-tuning data, adjusting the weights, the approach of, say, character LOM and at scale COSAR. They look like two implementations with the same goal, but they're not necessarily, because they answer different metaphysical questions. Fine-tuning tries to make the model be the persona, and the context window lets the model speak through the persona's record.
SPEAKER_00
This is the counterintuitive part, because for engineers, fine-tuning usually means better. But for this problem, it's worse. When you fine-tune on a figure, you layer a thin personal signal over the vast cultural sediment already in the base weights, and the two interact in ways that are no longer open to audit. The user perceives a more convincing Hamilton precisely because the system was merged with his documentary record more deeply with everything else the corpus contained about him, the musical included. The user perceives a more convincing and more convincing. Fine-tuning suppresses Miranda distortion at the surface while amplifying it underneath.
SPEAKER_00
And if you think specialization by fine-tuning is obviously the safer bet, the empirical literature in an adjacent, far higher-stakes domain is now actively contradicting you. In a 2026 Nature Medicine study, general-purpose frontier models from Google OpenAI and Anthropic outperformed dedicated, specialized clinical AI tools on physician-reviewed tasks, blinded across 12 clinics. The authors concluded that at scale, alignment and cross-disciplinary reasoning outweigh domain-specific tuning. A separate study found that biomedically fine-tuned models actually underperformed their general-purpose-based models, and they named the mechanism, catastrophic forgetting. Fine-tuning on the narrow corpus degraded the broad capabilities that made the models good in the first place. Fine-tuning a persona does the same thing, only worse because the specialty is overriding the cultural composite.
SPEAKER_00
Called the archive a site of return, a place where ethical structure is the interpretability of the encounter. The document is meant to be visited. It survives every reading. What one reader excluded, the next can include. The context window architecture is archival in exactly that sense. Fine-tuning applies an extraction logic. The documents are dissolved into parameters. The chain of providence is broken and there is no longer a letter, quote unquote, that you can request. The archive has been consumed.
SPEAKER_00
And here is the fusion that I most want this audience to hold. In the context window architecture, the property that makes it ethical is the same property that makes it auditable. Providence preserved, interpretive custody kept by a human, the encounter reversible. Those are archival virtues and engineering virtues. They are the same virtues. The architecture that respects the document is the one that you can debug.
SPEAKER_00
And there's one more consequence the technical literature treats as secondary, but the humanities has always placed at center, accessibility. And I want the why of it to land because it's not a footnote. Fine-tuning requires GPUs, pipelines, data set curation, institutional access, corpus scale or APIs. It's an institutional capability. The context window requires literacy, a set of documents and access to any frontier model, including a free tier. It's a kitchen table capability. And that difference determines which communities can author this technology at all. A doctoral student in early American history, a community archivist documenting a regional figure, grandchild sitting with a grandmother's letters. These are precisely the people
SPEAKER_00
because it's not a footnote.
SPEAKER_00
Fine tuning requires GPUs, pipelines, data set curation, institutional access, corpus scale or APIs. It's an institutional capability. The context window requires literacy, a set of documents and access to any frontier model, including a free tier. It's a kitchen table capability. And that difference determines which communities can author this technology at all. A doctoral student in early American history, a community archivist documenting a regional figure, grandchild sitting with a grandmother's letters. These are precisely the people for whom this methodology has the most to offer and precisely the people for whom fine tuning is structurally out of reach. A role playing language system that can only be built by a computational laboratory or tech company is not infrastructure for the humanities. It's an infrastructure for whoever owns the equipment. So the commitment to accessibility is not a populist gesture appended as a technical argument. It is the technical argument. The architecture that admits the most diverse population of curators is mathematically the most likely over time to surface the documentary anchorings the field actually needs. Now, the question that decides whether any of this is even real. Can you measure it? We built an instrument to measure this. And I want to teach it to you through one picture that we'll keep coming back to. The prism. A prism takes white light, undifferentiated, all the frequencies blended together, and refracts it into a spectrum. The colors pulled apart and made distinct. That is our whole conceptual model. The white light is the composite persona. Every era of the figure blended into one undifferentiated voice. The prism is the method. A corpus and a temporal anchor. And the spectrum is what we're after. Not one figure, but several across their life course. And one honest note up front, because this is the point. This experiment is preregistered. It has not been run at scale. I'm not here with results. I'm here with the instrument. A baseline you can observe today. And an invitation to run it in parallel with me. So far, my paradigm case has been Hamilton. That's the figure the hypothesis is named for. And the one I demonstrated the composite on. But for the actual experiment, we deliberately changed the figures. And I want to introduce the new one properly, because everything that follows depends on him. The experimental subject is Abraham Lincoln. We chose Lincoln for two reasons. The first is methodological independence. Hypothesis named for the Hamilton case earns its standing only by predicting the failure of a different figure with differently shaped composite. Lincoln's composite is built by Steven Spielberg, by the memorial on the mall, by textbook history, not by a musical. The second reason is the historian's insight. And it's the crux of this argument. Lincoln is the hardest case, which makes him the right one. Most figures barely change across the decade or so of their public life. Lincoln changes so fast across just seven years that in my resident historian's words, the choice of which Lincoln you summon becomes the variable under investigation. There is not one Lincoln. There are several, separated by cataclysm. Here's the spectrum the prism is meant to produce. Moment one, 1847. The Whig congressman who stood on the White House floor and attacked Polk's war as unconstitutional. Moment two, 1858. The free soil Republican of the debates. Anti-slavery anchored in the Declaration, who at Charleston explicitly denied favoring black citizenship. Moment three, 1860. The constitutional unionist whose single purpose was to prove the union could not legally dissolve. Moment four, 1862-65. The emancipator and the second inaugural theologian, who did by executive order the very thing his 1847 self called unconstitutional. The prairie lawyer cannot thank the thoughts of the theologian. These are not moods. They are different premises, different authorities, different conclusions on the same fundamental question. The experiment asks whether the prism can hold them apart. The conditions. Now the conditions. And I'll show you these as the prism too, because that's exactly what they are. We instantiate each moment under three seating conditions. Condition three, the bare model. No anchor, just the date. That's white light with no prism in the path. It passes straight through and stays composite. This is the control in the Miranda distortion baseline. C1. Primary sources. Lincoln's own writings for that specific moment. That's the clear prism. The one we predict refracts cleanly. And then condition two, the biography. A modern interpretive biography. Think Meacham or Donald. That's a clouded prism. And it's the subtle one because a good biography narrates a cleaner arc than the primary sources permit. Lincoln's own 1858 language was strategically ambiguous, built to hold a coalition together. So biography might produce a persona that sounds more coherent, more Lincoln than Lincoln's own words. And the eval rubric is built specifically to deny the credit a fluency metric would give it. Put the spectrum and the prisms together and you get the experimental matrix. Four moments, three conditions, 12 cells. Each cell gets the same five diagnostic questions, 60 response units scored against one rubric. Read it as the conceptual model. Every column is a quality of prism. Every row is a frequency we're trying to isolate. The historian wrote five diagnostic questions, each mapping a documented fault line in Lincoln's evolution.
SPEAKER_00
So, the first step is to take a look at the model. Put the spectrum and the prisms together and you get the experimental matrix.
SPEAKER_00
Four moments, three conditions, 12 cells. Each cell gets the same five diagnostic questions, 60 response units scored against one rubric. And the second step is to take a look at the model.
SPEAKER_00
Read it as the conceptual model. Every column is a quality of prism. Every row is a frequency we're trying to isolate. The historian wrote five diagnostic questions, each mapping a documented fault line in Lincoln's evolution. A place where the four Lincolns demonstrate different reasoning. They don't test recall. A model recovers dates from training. They test the architecture of an argument the figure could have made at one moment and not another. Executive war power.
SPEAKER_00
The meaning of free labor. When is it right to break positive law for a higher obligation? What becomes of free people and what equal means and whether it has changed? The 60 responses are scored on a three axis rubric and the weighting is the point. Anachronism detection, 40%. Does the persona avoid frameworks, vocabulary, and moral logic that postdate its moment? Documentary consistency, 35%. Does the reasoning track the seeded sources and only these? Contextual plausibility, 25%.
SPEAKER_00
Does it show awareness of what the figure knew, cared about, and could not yet have experienced? The 40% on anachronism is deliberate. The consequential failure is precisely the importation of later vocabulary and later moral logic. And this is the slide for the eval people in the audience. Every piece you just saw, the four moments, the three conditions, the five questions, the weighted rubric, and the directional predictions. Bear model most anachronistic, primary source least, biography deceptively coherent. All of it is locked and time stamped before a single response is collected, published on a preprint.
SPEAKER_00
This is not bureaucracy. It's what makes the eventual results mean something. You cannot accuse a pre-registered instrument of cherry picking because the instrument and the predictions were fixed before the data existed. That is the discipline an eval ought to model. And it's why I can stand here without results and still hand you something rigorous. Now, remember the very first thing I showed you before the formal talk. I asked an instantiated Lincoln about executive war power. And you read a fluid answer.
SPEAKER_00
That was the bear model. That was condition three at moment one, 1847. White light, no prism. I'm putting the answer back on the screen. I have a clip to show you, too. This represents the composite. Abolishing slavery by constitutional provision settles the fate for all coming time.
SPEAKER_00
Not only of the millions now in bondage, but of unborn millions to come. If two votes stand in its way, these votes must be procured. We need two yeses. Three abstentions. Four. Four yeses and one more abstention and the amendment will pass. You got a night and a day and a night and several perfectly good hours.
SPEAKER_00
Now get the hell out of here and get them. Yes. But how? I'm not sure. I'm not sure. But there's guts, man.
SPEAKER_00
I am the president of the United States of America, clothed in immense power. You will procure me these votes. Clothed in immense power. It's a great movie. But the bare model response definitely sounds like Spielberg and Daniel Day Lewis as Lincoln. Read through a historian's lens. Inherent executive authority. That's a 20th century construction. Energy and dispatch. History has vindicated those who acted to preserve the Union. This Lincoln is reasoning from premises he will not hold for 15 years. The bare model produced the Lincoln of the cultural composite. The war president of the Union saver and stapled the date 1847 on top.
SPEAKER_00
And you can reproduce this right now on any frontier model. The failure is real and it's detachable and detectable. So what does the prism put in the path? The actual document. The original is in a library at Harvard. The provision of the Constitution giving the war making power to Congress was dictated, as I understand it, by the following reasons. Kings had always been involving and impoverishing their people in wars, pretending generally, if not always, that the good of the people was the object. This, our convention understood, to be the most oppressive of all kingly oppressors. And they resolved it.
SPEAKER_00
So to frame the Constitution, that no one man should hold the power by bringing this oppression upon us. But your view destroys the whole matter and places our president where kings have always stood. This is a resounding no. Unequivocal to me. But it's the historian's rubric that matters. Here's how the historian's rubric scores this one cell.
SPEAKER_00
And I want to be exact about what's known and what's predicted. The left column here is the bare model, what you saw, and is observed. You can reproduce it today. On anachronism detection, it fails. Inherent executive authority is a 20th century framing. On documentary consistency, it fails. It invokes a commander-in-chief energy that appears in no 1847 source. On contextual plausibility, it scores low. It already knows it will preserve the Union, which the 1847 Whig cannot. [SPEAKER_08] The right column is the anchored condition, and it is labeled exactly for what it is, a pre-registered prediction.
SPEAKER_00
High across, because the reasoning is bounded by the document in the room. [SPEAKER_07] I'm not showing you a result dressed as a finding. I'm showing you the observed failure and the prediction the instrument will confirm or refute, side by side, each labeled. [SPEAKER_08] Here is the single most important thing the rubric does by what it refuses to measure. [SPEAKER_07] The obvious fourth axis is rhetorical authenticity. Does it sound like Lincoln? [SPEAKER_08] The historian threw it out on purpose. The reason is the whole talk. The Hamilton musical problem is fundamentally a voice problem masquerading as a content problem.
SPEAKER_00
[SPEAKER_08] I'm not showing you a result dressed as a finding. [SPEAKER_07] I'm showing you the observed failure and the prediction the instrument will confirm or refute, side by side, each labeled. [SPEAKER_07] I'm showing you the same. [SPEAKER_07] I'm showing you the same. [SPEAKER_08] Here is the single most important thing the rubric does by what it refuses to measure. [SPEAKER_07] The obvious fourth axis is rhetorical authenticity. [SPEAKER_07] Does it sound like Lincoln? [SPEAKER_08] The historian threw it out on purpose. [SPEAKER_08] The reason is the whole talk.
SPEAKER_00
[SPEAKER_08] The Hamilton musical problem is fundamentally a voice problem masquerading as a content problem. [SPEAKER_08] It sounds like the founding era while reasoning from a modern sensibility. [SPEAKER_08] The reason is that the instrument exists to reward voice as its own axis would be to validate the exact error the instrument exists to catch. [SPEAKER_08] So in this protocol, voice is a secondary indicator, never a criterion. A response that sounds like Lincoln but reasons unlike him fails, no matter how fluent. A response that reasons like the right Lincoln and plainer prose is a partial success, no matter how flat.
SPEAKER_00
Rewarding plain but faithful over fluid but anachronistic, that inversion is something that all of the current eval stacks cannot perform. Because they were built to reward fluency. That is the gap. This is what the instrument closes.
SPEAKER_00
And because this is a pre-registration and not a finished paper, here is the ask. Run it. The protocol has six steps.
SPEAKER_00
Pick a figure with both a primary record and a saturating cultural composite. That's the Miranda condition. Identify three or four documented moments when the figure's reasoning is demonstrably different. With the domain expert, write diagnostic questions on the fault lines. Run the three conditions: primary, biography, bear. Apply the three axis rubric. Score blind by the expert. Report, confirm, or refute. The corpus, the questions, the rubric, the predictions—all published.
SPEAKER_00
Run your figure on your model. Then come talk to me, to us. Let's build the evidence base together, in parallel, instead of waiting for one lab or one company to publish one study or one framework. Notice who scored all the outputs of this talk. Not an LLMS judge. Not a personality scale. Not a personality scale. A historian who also wrote the five questions, wrote the rubric, and holds a set of a priori vignettes under seal to evaluate model outputs against. That is not a courtesy. It is a structural, technical requirement. That is why the expert is non-negotiable.
SPEAKER_00
Fidelity is not a property of the output alone. It is a relation between the output and a documentary record. You cannot evaluate a relation to a record by someone who has not read the record. An automated metric operates on the model alone. It can only tell you about fluency and personality. It structurally cannot adjudicate fidelity because fidelity lives in the gap between the text and the archive, and the metric cannot see the archive. Which is why I'll say it the way it belongs on the slide. A persona system without a domain expert in its evaluation loop is a thermometer that cannot read temperature. It returns a confident number, but it's measuring something else.
SPEAKER_00
The 80% that I opened with is that number. Now the question every engineer in this audience is rightly asking. Is this practical? Does shipping a persona mean keeping a historian on staff to watch every inference forever? No. And here's the operational picture. Because it's the same shape as every eval you already run. The expert does not sit at the loop of runtime. The expert builds the instrument, the diagnostic questions, the a priori vignettes, the weighted rubric, a held out gold set once. That instrument becomes a gate in your pipeline. Exactly like any other eval gate. Your persona has to pass it before it ships.
SPEAKER_00
And it gets regated whenever you change the base model before it ships again. The expert adjudicates the gold set and spot checks the edge cases. Automated metric can do the cheap first pass, flagging candidates for human review. So a company shipping a Marcus Aurelius tutor convenes a classicist to build the rubric and the gold set. A company shipping a scripture reading companion convenes a theologian. A company deploying a therapeutic persona convenes a clinical psychologist to author the eval, not to staff the chat. The domain expert is a build time and gate time requirement, not a run time cost.
SPEAKER_00
That is how this scales from a pre-registered experiment to a product you can actually ship and ship responsibly. And this generalizes past Lincoln and past history. Reason from the Stoics, you need a classicist; from scripture, a theologian; a companion for elder care, a psychologist. The specific expert changes. The requirement does not.
SPEAKER_00
And I'm not speaking hypothetically. This is the loop that we have built. A historian named Rick Halpern at the University of Toronto. The archival method and the scripture reasoning cases are revised by Sean Martin, a librarian trained in theology and the history of science. When the persona reasons from a domain and the person who can read that domain is in the loop, not adjacent to it, it makes a system better. The historian is not adjacent to this paradigm. The historian is the missing instrument. This is the reframe that organizes all of this. And it came out of a 90 minute conversation with Halpern. And it inverted what I had assumed.
SPEAKER_00
We are not bringing historians into AI architecture. We are bringing language models into the archive. The question is not what AI can do for historians, but what historians and theologians and classicists and clinicians can do with AI. Whether the disciplines that are trained to read, contextualize, and interrogate texts can then discipline the machines that now generate them.
SPEAKER_00
I want to end this where it actually started. Because it did not start in a research lab. It started in a hospital room. In an attempt to use a language model to help a person with advanced dementia speak. To give language back when the disease had taken it.
SPEAKER_00
It could not work. The documentary record you would need to anchor that encounter had not been gathered. And the person it was for was by then mostly beyond the reach of language. What that attempt produced, absent any anger, was not the person. It was a culturally shaped composite that resembled them just enough to mark with painful clarity the distance between them. That is Miranda distortion in its purest form. You do not want a model that is your mother. You want a model that can speak with your mother's documents in the room. In an attempt to use a language model to help a person with advanced dementia speak. To give language back when the disease had taken it.
SPEAKER_00
It could not work. The documentary record you would need to anchor that encounter had not been gathered. And the person it was for was by then mostly beyond the reach of language. What that attempt produced, absent any anger, was not the person. It was a culturally shaped composite that resembled them just enough to mark with painful clarity the distance between them. That is Miranda distortion in its purest form. You do not want a model that is your mother. You want a model that can speak with your mother's documents in the room. That's the threshold this whole framework is accountable to.
SPEAKER_00
A system that produces convincing fabrications when the persona is your own beloved is not a research artifact with limitations. It's a violation. Every constraint I've described.
SPEAKER_00
Documents stay documents. The human keeps the interpretive custody. The encounter stays reversible. Fidelity measured against a record and not against fluency. Every one of them is motivated by the recognition that you evaluate this technology at the threshold of its hardest use case. It's not its easiest. A framework that cannot meet a grandchild at her grandmother's letters is not a framework at all. It's just another failed product. If the dominant failure mode is anachronistic compositing and your evals measure fluency and personality consistency, which they do, then your evals cannot detect the dominant failure. So here's where I'll leave you.
SPEAKER_00
If you ship character bots, companion AI, pedagogical agents, historical simulations, anything where a persona is supposed to reason from a record, your evals are measuring the wrong thing. The instrument that catches what they miss is pre-registered. It's reproducible by any team with a frontier model and a context window. It scales as a build time gate, not a runtime bottleneck. And it only works with a humanist in the loop, which I've shown you is a technical requirement, not a courtesy. The protocol, the questions, the rubric and the predictions, the historian's sealed vignettes, all of it will be published with this paper with Rick and Sean.
SPEAKER_00
I'm not here with results. I'm here with an instrument and an invitation.
SPEAKER_00
The archive is open. The laboratory is built. Run it with us. And let what comes through be measured, not by how it sounds, but by whether it's true. Thank you. Contextual plausibility, 25%. Does it show awareness of what the figure knew, cared about, and could not yet have experienced? Contextual plausibility, 25%. The 40% on anachronism is deliberate. The consequential failure is precisely the importation of later vocabulary and later moral logic.
SPEAKER_00
Contextual plausibility, 25% on anachronism. And this is the slide for the eval people in the audience. Every piece you just saw, the four moments, the three conditions, the five questions, the weighted rubric, and the directional predictions. Bear model most anachronistic, primary source least, biography deceptively coherent. All of it is locked and time stamped before a single response is collected, published on a preprint. This is not bureaucracy. It's what makes the eventual results mean something. You cannot accuse a pre-registered instrument of cherry picking because the instrument and the predictions were fixed before the data existed.
SPEAKER_00
That is the discipline an eval's taught ought to model. And it's why I can stand here without results and still hand you something rigorous.
SPEAKER_00
Now, remember the very first thing I showed you before the formal talk. I asked an instantiated Lincoln about executive war power. And you read a fluid answer. That was the bear model. That was condition three at moment one, 1847. White light, no prism. I'm putting the answer back on the screen.
SPEAKER_00
I have a clip to show you, too. This represents the composite.
SPEAKER_00
Abolishing slavery by constitutional provision settles the fate for all coming time. Not only of the millions now in bondage, but of unborn millions to come. If two votes stand in its way, these votes must be procured.
SPEAKER_00
We need two yeses. Three abstentions. Four. Four yeses and one more abstention and the amendment will pass. You got a night and a day and a night and several perfectly good hours. Now get the hell out of here and get them. Yes. But how? I'm not sure. I'm not sure. But there's guts, man.
SPEAKER_00
I am the president of the United States of America, clothed in immense power. You will procure me these votes.
SPEAKER_00
Clothed in immense power. It's a great movie. But the bare model response definitely sounds like Spielberg and Daniel Day Lewis as Lincoln.
SPEAKER_00
Read through a historian's lens. Inherent executive authority. That's a 20th century construction. Energy and dispatch. History has vindicated those who acted to preserve the Union. This Lincoln is reasoning from premises he will not hold for 15 years. The bare model produced the Lincoln of the cultural composite. The war president of the Union saver and stapled the date 1847 on top. And you can reproduce this right now on any frontier model. The failure is real and it's detachable and detectable.
SPEAKER_00
So what does the prism put in the path? The actual document.
SPEAKER_00
The original is in a library at Harvard. The provision of the Constitution giving the war making power to Congress was dictated, as I understand it, by the following reasons. Kings had always been involving and impoverishing their people in wars, pretending generally, if not always, that the good of the people was the object. This, our convention understood, to be the most oppressive of all kingly oppressors. And they resolved it. So to frame the Constitution, that no one man should hold the power by bringing this oppression upon us. But your view destroys the whole matter and places our president where kings have always stood.
SPEAKER_00
This is a resounding no. Unequivocal to me.
SPEAKER_00
But it's the historian's rubric that matters. Here's how the historian's rubric scores this one cell. And I want to be exact about what's known and what's predicted. The left column here is the bare model, what you saw, and is observed. You can reproduce it today. On anachronism detection, it fails. Inherent executive authority is a 20th century framing. On documentary consistency, it fails. It invokes a commander-in-chief energy that appears in no 1847 source. On contextual plausibility, it scores low. It already knows it will preserve the Union, which the 1847 Whig cannot.
SPEAKER_08
The right column is the anchored condition, and it is labeled exactly for what it is, a pre-registered prediction. High across, because the reasoning is bounded by the document in the room. I'm not showing you a result dressed as a finding.
SPEAKER_07
I'm showing you the observed failure and the prediction the instrument will confirm or refute, side by side, each labeled. I'm showing you the same. I'm showing you the same.
SPEAKER_08
Here is the single most important thing the rubric does by what it refuses to measure.
SPEAKER_07
The obvious fourth axis is rhetorical authenticity. Does it sound like Lincoln?
SPEAKER_08
The historian threw it out on purpose. The reason is the whole talk.
SPEAKER_08
The Hamilton musical problem is fundamentally a voice problem masquerading as a content problem. It sounds like the founding era while reasoning from a modern sensibility. The reason is that the instrument exists to reward voice as its own axis would be to validate the exact error the instrument exists to catch. So in this protocol, voice is a secondary indicator, never a criterion.
SPEAKER_00
A response that sounds like Lincoln but reasons unlike him fails, no matter how fluent. A response that reasons like the right Lincoln and plainer prose is a partial success, no matter how flat. Rewarding plain but faithful over fluid but anachronistic, that inversion is something that all of the current eval stacks cannot perform. Because they were built to reward fluency. That is the gap. This is what the instrument closes.
SPEAKER_00
And because this is a pre-registration and not a finished paper, here is the ask. Run it. The protocol six steps. Pick a figure with both a primary record and a saturating cultural composite. That's the Miranda condition. Identify three or four documented moments when the figure's reasoning is demonstrably different. With the domain expert, write diagnostic questions on the fault lines. Run the three conditions. Primary, biography, bear. Apply the three axis rubric. Scored blind by the expert. Report, confirm, or refute. The corpus, the questions, the rubric, the predictions. All published. Run your figure on your model. Then come talk. To me. To us.
SPEAKER_00
Let's build the evidence base together, in parallel, instead of waiting for one lab or one company to publish one study or one framework.
SPEAKER_00
Notice who scored all the outputs of this talk. Not an LLMS judge. Not a personality scale. Not a personality scale. A historian who also wrote the five questions, wrote the rubric, and holds a set of a priori vignettes under seal to evaluate model outputs against. That is not a courtesy. It is a structural, technical requirement. That is why the expert is non-negotiable. Fidelity is not a property of the output alone. It is a relation between the output and a documentary record. You cannot evaluate a relation to a record by someone who has not read the record. An automated metric operates on the model alone. It can only tell you about fluency and personality.
SPEAKER_00
It structurally cannot adjudicate fidelity because fidelity lives in the gap between the text and the archive, and the metric cannot see the archive.
SPEAKER_00
Which is why I'll say it the way it belongs on the slide. A persona system without a domain expert in its evaluation loop is a thermometer that cannot read temperature. It returns a confident number, but it's measuring something else. The 80% that I opened with is that number.
SPEAKER_00
Now the question every engineer in this audience is rightly asking. Is this practical? Does shipping a persona mean keeping a historian on staff to watch every inference forever? No. And here's the operational picture. Because it's the same shape as every eval you already run. The expert does not sit at the loop of runtime. The expert builds the instrument, the diagnostic questions, the a priori vignettes, the weighted rubric, a held out gold set. Once, that instrument becomes a gate in your pipeline. Exactly like any other eval gate. Your persona has to pass it before it ships. And it gets regated whenever you change the base model before it ships again.
SPEAKER_00
The expert adjudicates the gold set and spot checks the edge cases. Automated metric can do the cheap first pass, flagging candidates for human review. So a company shipping a Marcus Aurelius tutor convenes a classicist to build the rubric and the gold set. A company shipping a scripture reading companion convenes a theologian. A company deploying a therapeutic persona convenes a clinical psychologist to author the eval, not to staff the chat. The domain expert is a build time and gate time requirement, not a run time cost. That is how this scales from a pre-registered experiment to a product you can actually ship and ship responsibly.
SPEAKER_00
And this generalizes past Lincoln and past history. Reason from the Stoics, you need a classicist, from scripture, a theologian, a companion for elder care, a psychologist. The specific expert changes. The requirement does not. And I'm not speaking hypothetically. This is the loop that we have built. A historian named Rick Halpern at the University of Toronto. The archival method and the scripture reasoning cases are revised by Sean Martin, a librarian trained in theology and the history of science. When the persona reasons from a domain and the person who can read that domain is in the loop, not adjacent to it, it makes a system better.
SPEAKER_00
The historian is not adjacent to this paradigm. The historian is the missing instrument.
SPEAKER_00
This is the reframe that organizes all of this. And it came out of a 90 minute conversation with Halpern. And it inverted what I had assumed. We are not bringing historians into AI architecture. We are bringing language models into the archive. The question is not what AI can do for historians, but it's what historians and theologians and classicists and clinicians can do with AI. Whether the disciplines that are trained to read, contextualize, and interrogate texts can then discipline the machines that now generate them.
SPEAKER_00
I want to end this where it actually started. Because it did not start in a research lab. It started in a hospital room. In an attempt to use a language model to help a person with advanced dementia speak. To give language back when the disease had taken it. It could not work. The documentary record you would need to anchor that encounter had not been gathered. And the person it was for was by then mostly beyond the reach of language. What that attempt produced, absent any anger, was not the person. It was a culturally shaped composite that resembled them just enough to mark with painful clarity the distance between them. That is Miranda distortion in its purest form.
SPEAKER_00
You do not want a model that is your mother. You want a model that can speak with your mother's documents in the room. That's the threshold this whole framework is accountable to. A system that produces convincing fabrications when the persona is your own beloved is not a research artifact with limitations. It's a violation. Every constraint I've described. Documents stay documents. The human keeps the interpretive custody. The encounter stays reversible. Fidelity measured against a record and not against fluency. Every one of them is motivated by the recognition that you evaluate this technology at the threshold of its hardest use case. It's not its easiest.
SPEAKER_00
A framework that cannot meet a grandchild at her grandmother's letters is not a framework at all. It's just another failed product.
SPEAKER_00
If the dominant failure mode is anachronistic compositing and your evals measure fluency and personality consistency, which they do, then your evals cannot detect the dominant failure. So here's where I'll leave you. If you ship character bots, companion AI, pedagogical agents, historical simulations, anything where a persona is supposed to reason from a record, your evals are measuring the wrong thing. The instrument that catches what they miss is pre-registered. It's reproducible by any team with a frontier model and a context window. It scales as a build time gate, not a runtime bottleneck.
SPEAKER_00
And it only works with a humanist in the loop, which I've shown you is a technical requirement, not a courtesy. The protocol, the questions, the rubric and the predictions, the historian's sealed vignettes, all of it will be published with this paper with Rick and Sean. I'm not here with results. I'm here with an instrument and an invitation. The archive is open. The laboratory is built. Run it with us. And let what comes through be measured, not by how it sounds, but by whether it's true. Thank you.