AI Engineer

The Next Game Engine Won't Have a Manual — Arturo Nunez, Nereu

1922 summary words 9 min summary Watch video

Start with the signal

9 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: Nereu argues that the next generation of game engines should expose game-design intent through natural language and reusable asset tags, while hiding conventional engine boilerplate behind an AI assistant.
  • Why it matters: It is a concrete example of an AI-native control plane: instead of having an LLM generate arbitrary code, the model operates through constrained, domain-specific primitives with scene-aware context.
  • Best use: Watch for the product and agent-architecture patterns—intent abstraction, tool-constrained execution, asset metadata, and relevance-weighted context—not for a mature game-engine roadmap.

Executive Summary

Arturo Nunez presents Nereu, a browser-based, AI-native game-making tool intended to let people create playable scenes by stating game-design goals rather than learning engine internals. In the demo, users ask the assistant, Bibi, to add a robot, make it move with WASD and animate it, add buildings and rain, and configure a following camera. The premise is that users already understand concepts such as characters, jumping, scoring, and cameras; conventional engines force them to translate those concepts into meshes, renderers, colliders, animation systems, physics settings, and scripts.

The product’s core technical abstraction is an Asset Tag System (ATS), inspired by entity-component systems and data-oriented design. Rather than directly generating custom implementation code for each request, the assistant assigns or removes semantic tags and settings from assets. Engine systems then query those tags and apply reusable behaviors: an object tagged as player, vehicle, and drivable gets vehicle behavior, while an untagged building remains static. This shifts the LLM from inventing mechanics repeatedly to selecting from known engine capabilities.

Nunez frames this as a corrective to current LLM-plus-engine workflows, where users must still know the vocabulary of Unity, Unreal, or code and where models often recreate standard patterns such as character-follow cameras. Nereu constrains the assistant to built-in tools and tags, although the underlying browser-based system uses JavaScript and can be extended by advanced users. The harder unresolved problem is not tool calling but decomposing the enormous variety of game genres, mechanics, aesthetics, and moods into a coherent set of reusable tags and systems.

The presentation also contains a practical context-management pattern. Instead of sending the entire game scene to the LLM, Nereu uses a level-of-detail analogy: the selected object and nearby objects receive rich contextual descriptions, while distant or low-relevance objects are summarized minimally. Asset tags for a library of roughly 6,000–7,000 3D assets were generated with vision models. Nereu is still in closed alpha, and its stated goal is assisted iterative creation and learning—not one-shot generation of finished games.

Key Takeaways

  • Claim: The right AI interface for game creation is intent-level language, not exposure to the engine’s implementation vocabulary. | Evidence: The demo converts requests such as “add a robot,” “make this robot move with WASD and animate it,” “add rain,” and “make the camera follow my character” into a playable scene without requiring model imports, code, or manual component setup. | Implication: For AI-operated creative tools, the interface should accept concepts users naturally use in the domain, then translate them into platform primitives behind the scenes. | Caveat: The speaker is targeting creative accessibility and rapid iteration, not replacing professional game-development workflows or guaranteeing commercially viable games.
  • Claim: Nereu’s Asset Tag System is designed to prevent LLMs from repeatedly reinventing standard game-engine boilerplate. | Evidence: Nunez contrasts a conventional character setup—mesh, renderer, animator, rigid body, collider, audio source, movement logic, and game rules—with tags such as character, animated, and double jump. Systems query assets carrying relevant tag combinations and apply behavior. | Implication: A durable agent architecture should favor a constrained action schema and reusable deterministic systems over unconstrained code generation for common tasks. | Caveat: The abstraction only works where the engine has already implemented systems capable of interpreting the selected tags; novel mechanics may require new system design rather than prompting alone.
  • Claim: The AI assistant functions as an orchestrator over known engine tools rather than as an autonomous generator of arbitrary game logic. | Evidence: The user query is combined with scene context, asset-type context, and an optional description of the desired game; the agent then performs calls that append or remove tags. Nunez says all tags and systems are built into the engine. | Implication: This is a useful control-plane pattern for agent products: define the tool surface first, expose tool definitions to the model as capabilities expand, and reserve general-purpose coding for exceptions. | Caveat: Nereu deliberately has no primary scripting workflow, though JavaScript runs in the browser and advanced users can extend the system.
  • Claim: Context assembly should be relevance-weighted rather than based on dumping the entire working state into an LLM prompt. | Evidence: A scene may contain about 100 objects, many of which are irrelevant grass. Nereu gives detailed tag and setting information for the selected object and nearby objects, while distant objects may be represented only as a brief type-and-position summary. | Implication: For interactive agents operating on large graphs, scenes, or workspaces, spatial or task proximity can be a practical retrieval heuristic alongside semantic relevance. | Caveat: The talk does not provide quantitative evaluation of whether this context-selection method improves accuracy, latency, or cost.
  • Claim: Asset metadata is foundational to natural-language creation because it connects user intent to a searchable, usable asset library. | Evidence: Bibi finds robot and building candidates from asset tags or descriptions; Nunez used vision models on screenshots to label a library of approximately 6,000–7,000 3D assets because manual labeling was impractical. | Implication: Any agentic system that must act over unstructured inventories needs a scalable metadata-enrichment pipeline before natural-language retrieval becomes reliably useful. | Caveat: Vision-generated labels can be incomplete or inaccurate, and the presentation does not describe a validation, correction, or provenance workflow for asset metadata.
  • Claim: The intended product is an iterative creative partner that gets users unstuck, not a one-shot game-generation system. | Evidence: Nunez explicitly rejects the goal of generating games “that nobody is going to play” and emphasizes experimenting, sharing creations with friends or family, and learning the language of game design through use. | Implication: For generative creation products, optimize for a fast edit-feedback loop and user agency rather than maximizing apparent autonomy in a single prompt. | Caveat: This makes the product’s success metric less about instant output and more about whether users can sustain productive iteration and retain enough control to make meaningful design choices.
  • Claim: World-model-generated games are viewed as a distinct medium from deterministic, real-time game engines rather than an immediate replacement for them. | Evidence: Nunez notes that games must commonly render at 60 frames per second and 4K while simulating physics, which he believes remains far away for world-model approaches. | Implication: There may be room for both paradigms: structured engines for controllable interactive systems and generative world models for different forms of dynamic media. | Caveat: This is the speaker’s forward-looking technical judgment, not a benchmarked comparison of current systems.

Detailed Brief

Product-design philosophy and target user

  • Claims: Nunez argues that game-development friction is not only technical: small teams and solo developers must span programming, 3D modeling, rendering, audio, animation, real-time camera composition, and game design.; He believes the creative process should be enjoyable for developers as well as players, rather than being dominated by production pressure and commercialization.; The tool is intended to broaden participation without prohibiting professional or conventional development paths.
  • Evidence: Nunez worked at Unity for nearly 10 years, later worked at MongoDB, and also helped a startup dealing with game-asset version control.; He says repeated exposure to developers solving the same setup problems convinced him those problems should not consume creators’ energy.; He compares the desired experience to playing with toys or Legos: users can combine an astronaut, dinosaur, building, or vehicle without first mastering a formal pipeline.
  • Caveats: Lowering technical barriers does not solve the separate challenge of making a game enjoyable, polished, or commercially successful.; The talk does not offer user-retention, quality, or adoption data from the closed alpha.
  • Implications: The strongest initial market may be hobbyists, learners, families, and design-oriented creators rather than established studios with deeply customized production pipelines.; Nereu’s product positioning depends on preserving playfulness and discoverability while preventing the abstraction from becoming too limiting.

The unresolved ontology problem

  • Claims: Nunez identifies the central development challenge as composing games into tags and systems, not simply attaching an LLM to an editor.; A complete intent layer must account for genre-specific mechanics as well as aesthetic and emotional direction.
  • Evidence: He gives “a platformer that feels scary” as an example: the request reaches beyond player movement into lighting, post-processing, and other visual-system choices.; The team repeatedly gathers specialist input, decides which settings and features can be removed, builds a simpler tool, exposes that tool definition to the assistant, and iterates with alpha-user feedback.
  • Caveats: Overly generic tags may produce shallow results, while a rapidly growing tag taxonomy could recreate the complexity of conventional engine settings.; No governance model is described for versioning tags, resolving conflicting intents, or safely evolving system behavior as the engine expands.
  • Implications: The competitive asset is likely to be the evolving semantic ontology and its mappings to robust systems, not merely the chat interface or base model.; A scalable implementation will need explicit mechanisms for capability discovery, composability, conflict resolution, and graceful fallback when user intent exceeds built-in primitives.

Notable Concepts & Terms

  • Nereu: A closed-alpha, browser-based game-creation tool built around natural-language interaction and structured engine capabilities.
  • Bibi: Nereu’s AI assistant, which interprets user requests using scene and game context and applies engine-supported changes.
  • Asset Tag System (ATS): Nereu’s semantic layer: assets receive intent-oriented tags and settings, while systems query those tags to enact reusable behaviors.
  • Entity Component System (ECS): The game-development architectural inspiration for ATS; behavior emerges from systems operating over objects described by composable data.
  • Data-oriented design: A design approach centered on describing and processing data consistently, used here to justify tags plus systems rather than bespoke object logic.
  • Level of detail (LOD) context assembly: An adaptation of rendering LOD for prompts: provide rich state for the active and nearby scene elements while compressing peripheral information.
  • World models: Generative systems that create worlds on the fly; Nunez treats them as promising but currently distinct from high-performance, physics-driven real-time engines.

Operator Notes / Why Ken Should Care

  • Evaluate the ATS pattern as a reusable agent-control-plane design: user intent → constrained semantic primitives → deterministic systems, rather than prompt → unrestricted generated code.
  • For any agent operating over a large workspace, test locality-aware context construction against full-state prompting; measure task success, token cost, latency, and failure modes when relevant dependencies are distant.
  • Treat ontology development as a first-class product and engineering function: define capability schemas, composition rules, conflict handling, versioning, and an escalation path for unsupported user requests.
  • If assessing Nereu as a product or investment, request closed-alpha evidence on completion rates, iteration behavior, generated-game quality, asset-tag accuracy, and the proportion of requests that require manual or scripted escape hatches.
  • Monitor the tradeoff between simplification and expressive power: removing settings can improve usability but may concentrate advanced users’ needs into an increasingly important extension layer.

Source/Metadata

  • Title: The Next Game Engine Won't Have a Manual — Arturo Nunez, Nereu
  • Transcript words: 2842
  • Duration seconds: 1172
  • Timestamp note: No timestamps or chapter markers were present in the provided transcript.
Search this video Resume transcript at 0%
Full transcript 2813 words · 13 min read
0:00

Hello. Hi, everyone.

0:12

Thank you for being here. Let's start the presentation. I really appreciate your interest in learning more about game development. So today I'm going to talk about how I think the next game engine won't have a manual. And at the end, you'll see what I mean by this. My name is Arturo. I'm working on this tool called Nereu. And first, I want to show you how it works today so you get a glimpse of what I mean by all this. So we can start asking for what we want and describing something like, OK, I want to add a robot. OK, so I ask my assistant to help me add a robot. It found some assets that are robots or have been tagged as robots, or the descriptions are a robot.

0:43

In this case, I want to have a character.

0:50

It's going to be the character that I will be controlling. If I try to play right now, it won't do anything. I still need to tell it what I want. I need to describe, OK, I want this to move with WASD, and I want it to be animated, have some animation. So I will describe that to my assistant, which is called Bibi, by the way. And I'm saying, make this robot move with WASD and animate it, as I mentioned. These descriptions are pretty common if you know or if you've played games. You just describe what you want the thing to do. And it's language that people who play games understand.

1:24

You don't need to code, or you don't need to learn about importing models or anything like that. So I want to add some buildings. In this case, we have a library of assets that might match the description of a building. We could have futuristic buildings, something like this warehouse or store. Maybe we can use a castle. The idea here is that we can let our imagination go wild. It doesn't need to be the perfect game or anything. It's playing a game, playing with toys, playing with Legos, or something like that. I'm saying I want it to have some rain. So I just described the rain, and it will spawn a particle system that's set up or configured as rain.

2:24

The last thing is I want a camera to follow my character. Because if I move around, it will go out of view, and I won't be able to see it. So it added a camera. It configured it. It's looking at my main character. And I have a game without knowing how to code or import assets or anything besides just knowing what I want. Because I know the language of building games. That's the idea. So making games or getting to this point with a regular tool is very difficult. I recorded this demo. But you could do this in just a couple of minutes without knowing anything, just describing. All the assets are there for you.

3:18

But if you were to follow the traditional workflow, you would need to know other things. Sorry. Before continuing, I want to mention why I am speaking here. And you can decide if you want to pay more attention or less attention to me. I worked at Unity, a game engine company, for almost 10 years. So I saw a lot of people building games and struggling with the same things over and over. At first, it was fun. The challenges were fun. But after seeing them thousands of times, it was like, OK, this is not fun. I don't think people should spend their energy and time reinventing the same world. So then I was part of MongoDB.

4:09

Then I was learning about more AI topics and how to manage data and all that stuff. And I was helping a startup to manage and handle version control of assets of games. So as I said, I've seen people make games. I've made games. And I think I want to let more people have fun and experience the joy of making games without getting frustrated and without spending two or three years releasing something, realizing that it's difficult to sell a game. There's a lot of competition. And, of course, if you want to do that, I'm not preventing you from doing that. But for a lot of people, I think it's just a creative outlet, making games.

4:42

And the game itself should be the thing, right? Enjoying the process rather than the end product. Okay. So as I was saying before, making a game requires a lot of skills, a lot of knowledge. It's very difficult. You need to know programming, 3D modeling, rendering, music, animation, and so many other things. You either have a huge team that complements each other, or if it's just you or you have a small team, people need to wear multiple hats. And honestly, it's very difficult to find someone who's great at game design and also great at rendering and great at composition of cameras in real time. So on top of that, the game has to be fun, right?

5:33

If it works and technically works, if it's not fun, honestly, people won't play it. And I think fun should be for the player, but also for the developers. In recent years, I think it's become more about producing more and trying to sell more rather than the craft of making games. And I don't think you need to be a professional game developer to be able to make games. The same as with AI assistance to write code, now you don't need to be a programmer to build whatever you want. The same for games. There are other challenges, but that's my thought. Now, there are powerful engines, Unreal, Unity, and there are powerful LLMs and agents. But I think still it's hard.

6:25

It's hard because I think we're just building a bridge between two worlds, and it's not optimal. And you still need to know what to ask in the vocabulary of an engine, in the vocabulary of code. Otherwise, the LLM goes rogue and does stuff and reinvents the wheel over and over. This is the point, right? If I say I want a camera that follows this character, I've seen the demos, and the LLM reinvents the wheel every single time, when the result is going to be essentially the same. By default, the context is on the game engine rather than on the game design part. And I think we should flip that idea.

7:04

So, in a current engine, if you want to control a character, you need to have a mesh, import a mesh, and think about so many things: a renderer, an animator, a rigid body and collider for physics, an audio source. And then you add your movement logic and your game rules. Most of that is just boilerplate that every single game out there has, or every character in every single game out there has. But somehow developers need to understand and read the descriptions of those components and what the hundreds of sliders do in a game. So, I think the goal should be just to think, OK, everything is just an asset. Everything has to be rendered on screen.

7:48

Everything has physics most of the time. And you just add tags to describe the intent of those assets in your game. So, I want this to be a character, to be animated, to double jump. And this is the language that we use when we play games, right? The tutorial tells you, oh, press A to jump and press A again while you're in the air to do a double jump. That's the language that we should be using. And also defining what happens when an event occurs. In this case, oh, when you collect a coin, I want you to increase the score. And, of course, the physics and the rendering still happen.

8:41

But we're building on a layer on top to make it more accessible for more people to build games. And the engine knows what to do with these tags because these systems. And this system comes from something that we're calling the ATS, or asset tag system. It comes from the idea in game development called the entity component system, data-oriented design, which means we just describe objects with components or, in this case, with tags. And then there are systems that query for all the assets in the world and say, OK, I'm going to move all the objects that have the vehicle, the player, and the drivable tag. So that's how it works, and all the games can recycle this.

9:14

And then there's a building that doesn't have tags, so it's not going to move. But nothing prevents you from adding the vehicle and drivable tag to your building, and then you have a building that you can put in a Mario Kart style of game. Nothing prevents you from doing that. And, as I said, letting your imagination go wild. The assistant, the AI assistant, essentially helps you get unstuck. If I don't know how to make the car move, I just ask it, and it knows, understands, what are the tools available, what are the tags available, and then it applies them to the asset that I'm talking about. So how is it driven? This is a pretty common concept from agents.

10:14

You just type your query. We build the prompt using context from the scene, but also some extra context based on, for example, what type of assets are in your game. If you're using robots, if you're using something medieval, et cetera, we append that to the context. We also allow the user to describe the type of game that they want to make. So that's also part of what is appended into the context. And then the agent just performs calls and appends or removes the tags. All these tags and all these systems are built into the engine. We don't have a scripting system in there. That's on purpose, but it's just JavaScript and runs on the browser.

11:03

So whoever wants to extend, they can. Nothing prevents them from doing that. But for most users, that shouldn't be the case. And, yeah, all the updates to the scene happen, and that's how it works. Honestly, the challenging part is composing games into these tags and into these systems. Because there are a lot of genres. There are a lot of ways to describe the same game. There are things like the mood of the game. I want to make a platformer, but that feels scary. Well, that also touches on things like the post-processing effects and things like the lighting, et cetera. So those things are what we're decomposing to drive the engine. How am I building this?

12:09

It's the first time that I build something of this scale at this speed and using, mostly, working with an AI daily. And I'm sure everyone uses Cloud Code or something similar. The way it works is I bring all this stuff that is on my mind, and I talk to Cloud, and we discuss that part. Then I go to, let's say, if I'm focusing on something related to lighting, I talk to some friend in the industry who focuses on that, and I ask questions. And then I bring that in, and then we build those tools. And I say, I don't think we need this setting. I don't think we need this feature.

12:45

I haven't seen a lot of users using those things, so let's get rid of them in order to simplify the engine. We build it. Then, as we build, we give the definition of the tools to the assistant. So every time we add new tools, the assistant knows the tools that we're adding. And, yeah, right now it's still in closed alpha. I don't even know how to call it at this point. But some people are using it, giving feedback, and then we go back to square one. So one thing that I think is worth sharing also in this presentation is the way of assembling these contexts. Because, yeah, we're using an LLM mostly.

13:37

I've also used vision models mostly to tag the assets that we have because it's 6,000 or 7,000 assets. I could not manually tag them all and explain, oh, this is an astronaut, and this is a knight, and this is a castle, and this is whatever. I just have the names and the 3D file. So I took a screenshot and ran a vision model to describe those. But it's mostly an LLM, what we're using. And if we feed the entire scene to the LLM, the context grows a lot. Let's say in this scene, I think I had 100 assets, 100 things there. But most of them are grass that we could ignore. It doesn't really make sense. But this is a simple scene.

14:34

What we're using here to assemble the context is something that's very well known in the game dev world, which is called level of detail, which means if I'm close to an object or it is close to me, I'm going to render it with a higher quality texture, with a higher quality material, with a higher quality model in many cases. Something that's too far away from the camera, I'm just going to maybe put a cube, and the user won't be able to tell because it's so far away. So we're using something similar to assemble the context. In this case, if the user is editing the game, you can see that maybe it's clicking on the knight, right?

15:04

So the things that are around it might have a higher priority. So we feed information about the tags that they have and the values of the settings on those tags. Things that are nearby, we just say, OK, there's something that's a player here at this position, but I'm not going to send you the entire context in there. Okay? And as the user keeps moving around and modifies things, then we update that and feed the assistant with more relevant information about the game. Okay. So just to start wrapping up, I think this assistant called Bibi should be something that gets people unstuck from making the game.

15:45

I don't want us to one-shot games that nobody is going to play, and I don't see the point in that. There's, in the industry, of course, the need for games that are one-shotted, but here the idea is that we allow people to make games and experience that and have fun and share those games with their families and friends and that. And, of course, that they learn along the way. They learn along the way the language of making games, the language of game design, not necessarily the coding or programming, although people have used this and say, oh, yeah, I can understand concepts that I didn't understand before of programming, but that was not the initial goal.

16:33

There's another branch of engines or tools that are being used called world models, which are being generated on the fly, and some people, I think there's another session later talking about this and how games can be achieved with world models. I think that's going to be a different medium. Even if we call them video games, there are many challenges. Games, for instance, these days, have to render 60 frames per second, and doing that at 4K resolutions in real time with a world model, I think is still far away, and on top of that, rendering physics and, sorry, simulating physics and stuff, is very difficult, but I think it's exciting what's going on. So that's it.

17:32

If you want to play around, if you want to share this with your kids or friends or whoever, I can add you to the wait list. Honestly, it's been very fun to make, but also whenever I want to test something and I just play around and bring in an astronaut and bring in a dinosaur and make them just walk around, it's so much fun. It's what got me into game development, I think, 20 years ago, and I progressed throughout my career. So this year, I started getting disappointed, like, oh, this is not fun anymore, so I'm having fun again, and I want to share that with more people.

18:15

So thank you so much, and I'll be outside if you have questions, want to chat more about games or whatever. Thank you so much for your time. Bye. Bye. Bye. Bye. Bye. Bye. Bye. So thank you so much, and I'll be outside if you have questions, want to chat more about games or whatever. Thank you so much for your time.

19:28

Bye. Bye. Bye. Bye. Bye. Bye.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note