A Piece of Pi: Embedding The OpenClaw Coding Agent In Your Product — Matthias Luebken, Tavon

Transcript

[SPEAKER_00] All right. I was introduced to Pi by looking into OpenClaw. There was a conference, a meetup, and I said, okay, we're doing OpenClaw. And I wasn't so much interested in all the crazy things that people are doing, but I was more interested in understanding how these things work. So I was looking into Pi and understanding the whole world of what Pi is able to do. This is the one picture you need to take. Please feel free to take more pictures, but all the slides and the examples are there. So that's the one slide. All right. Very quick about myself. We're creating a small company, Taven AI. We're building agents for organizations, small out of Europe, but getting started. And what I really like about Mario's talk is this quote. You probably have seen this this morning. We are in the fuck around and find our own phase for coding agents. Right. So everything that I'm going to show you is what I know today. Right. And I'm going to do the talk again in a couple of weeks, and it's going to most likely be different. But as Mario was showing this morning, he has created this minimal set. Right. This coding agent that is available for you guys to fool around with. And that's what I'd like to encourage you. So coding agents and why is it so exciting for us to build more products? This is Ken Thompson, inventor of Unix. And this is the famous quote by him. One of the quotes: write programs that do one thing and one thing well. And I really like that because that works to our advantage with agents. And the best part where I show this is with CoWork. So this is CoWork, Cloud's desktop. And they're bundling their coding agent into something where they feel is more applicable. And to be honest, I've seen very good receptions around this. And when you use it with financing tools, with their finance tools, you always need to work with Excel. Right. So they have this Excel skill down there. And it talks to Excel, right? Well, it doesn't. Instead, it uses a set of small tools, small CLIs, Pandas, OpenPyExcel, stuff from LibreOffice, and package this into their own skill to make it up and running. And I think this is a great example to get your thoughts going of what is doable. I haven't written a book. And nobody can write a book about this, right? Because there are no patterns, right? We need to figure this out. We're seeing some emerging patterns in the coding space, right? There's obviously tons of different coding agents. And we're seeing this. But there is no authoritative resource around this, right? So get going. One thing, when I was talking to Ivan yesterday, we realized is one architectural pattern that we're seeing is that make it easy for coding agents, right? Now that is very broad, but think about it, right? Don't try to be very complex and things, but think about the coding agent. What is it good at? And how do I build my system so that the agent is easy, make it accessible? And I have some examples. All right. This is the rough agenda for the next 10 minutes or so. I'm not going to talk too much about Pi in OpenClaw. I have two slides. Slides are online. So we'll take it from there. So again, very brief introduction of Pi. Mario, great work. Something he didn't mention is that he's joining Arendelle, which I think is awesome. It seems like great folks working together. And yes, it's open source. It's minimal. So it's just perfect to get started. And the other part that I do want to re-emphasize on is give it a try, right? We're going to talk about a little bit different, but open up Pi and ask it to build what you want, right? It's amazing what it actually is able to do by the system prompt that Mario has shown. All right. These are the extensions. So again, all the extensions you can download, build yourself or download. And tons to explore. All right. So let's go in. This talk is not about the coding agent itself. So using it for your daily dev works. But what can we potentially do with this? And the starting point are actually not coding agents, right? The starting point is, and I encourage you to do the same, is looking at the core agent itself. And there's other SDKs, but we're talking about Pi. So let's use Pi. And what is an agent? An agent is actually just an LLM agent that runs tools in a loop, right? So you have some goals. You have some context information, agents.md in many cases. And then you go to code tool calls, right? And you get some results. And you basically do it in a loop, right? That's it. There's not much more. The rest is magic trying to put it in your use case a little bit more, in the other use case a little bit in that direction. So that's really it, right? So please, open the curtain and play around with it. Now with agent core, this looks a little bit something like this. You have an agent class. This is all TypeScript. You can address all sorts of information. You can prompt it with different information. And also, you have an event system, so you know a lot of things that are going on. So, small example, this is a CRM lead qualifier. I've started the CRM use case personally and it just sticks around. So, terminal interface, obviously, small TypeScript application, three files, really easy. And you can see this, right? You have a couple of commands that you can execute and show me all leads and score them, right? So, that's what we do, show all leads and score them. And here you see all these things that are going on under the hood, right? And also, you have an event system, so you know a lot of things that are going on. So, small example, this is a CRM lead qualifier. I've started the CRM use case for myself and it just sticks around. So, terminal interface, obviously, small TypeScript application, three files, really easy. And you can see this, right? You have a couple of commands that you can execute and show me all leads and score them, right? So, that's what we do, show all leads and score them. And here you see all these things that are going on under the hood, right? You see that the assistant is calling tools, that you get some results, and eventually, you get some input. Now, obviously, there's tons of things to do, but I've just coded this away. And it's a good learning exercise. The system prompt, as you could imagine, right? You know, calling out the different tools, that's what you do, right? So, all pretty straightforward if you are building an agent. This is an example of how you inject here, right? So, we said we do tool calling, right? We reach out to this and call a specific tool. But for the agent for steering it more, right, a typical hook would be before the tool call do something, right? And in this case, we don't want to update a contact without checking something or, you know, you can imagine any types of authoritative role-based access, whatever enterprise feature in here, but basically, just before the tool call. There's another one, events. So, we've seen these, the stream and you might have seen a little check mark there. Okay, the tool call was fine and returned some result. So, again, we're subscribing to events, all pretty straightforward. And, again, please give it a try. All right. So, this is simple agents. Other agents SDK are available. And now we're moving through the coding agent. Now, what's the coding agent? At the end of the day, it's really the same thing as we've seen before. It's a normal agent, right? It runs tools in the loop. But now we have a runtime and some type of shell, right? Bash seems to be the shell that everyone is using. But we have a shell and a runtime to start executing. And now things are getting interesting. And now the magic of what you've seen with OpenClaw suddenly shines. Peter shared this example on some presentation where he sent a message to his OpenClaw and sent a voice message. Now, at that time, OpenClaw didn't know anything about voice, about voice message. So, what it did is it created and used different tools. And, in the end, one of the tools was FFmpeg, right, on the local machine. And it started this. And this was one of the tools, right? So, from the outside, it looks learning. But on the inside, it's actually just another tool call that is available to the agent. And that's why these things make it so interesting. So, again, the example here, now, this is a little bit more sophisticated. But the important part, and this is the extension API, and please look it up online. We're going to do two things, or the things that I'm mostly interested in is in session events and UI interaction. And, yeah, look it up online. But here's the actual extension. Now, again, this is what you would, in a coding agent, you probably just generate by asking it. But here, if we have a look, this is a CRM TypeScript, a small snippet of it. And basically, what we're now doing is we're doing the same example as before, right? And we have a new command called pipeline, right? So, if you have the slash commands, and you have a new command called pipeline, and now we are able to, we're loading all the context. And you see this little in, don't have the lines, just below step one, you can see context UI select, right? So, all of a sudden, we're not only interacting with the backend systems and sessions and of those sorts, but we're also interacting with the UI. And we're able to select, right? And that's got me thinking. So, right? So, you have this command. And, again, this is now just the coding agent, right? We're not talking about the core agent class, but this is how you would load up Pi if you just don't download the coding agent. And now, with this new extension, we have Pi, right? And we can start selecting things, right? So, this is a simple select here. And you even have dropdowns. Now, the important part here is these are extensions, and the framework that currently Pi has included is catered towards the use cases of a coding agent, right? So, there's lots of work and other things to do to make this ready for others, for other types of applications. But I hope you can see and understand the vision, where this is heading. And, yeah, this is all terminal, right? So, you wonder how would this look like in the web? It currently is not possible if you ask Pi to build something. So, I asked Pi to build something, right? And this is the web UI. It would be a web UI, same command, same selection, all based on the same extension mechanism. Now, there's a refactoring going on to make this better accessible and make it more clean. But I hope, again, it shows you a little bit of where the things are going. All right. Now, Pi and OpenClaw is a special setup, right? So, Pi and OpenClaw, what we have there is that now we're not only talking about a single agent in a single session in a coding environment, but now we have a multi-channel environment where we have multiple threads going on, multiple agents going on. So, there's a little bit more to it. This is, and the interesting part, right? That's where I got started, is if you look into the packages, the core packages of Pi, all of them are used in OpenClaw, right? All right. Now, Pi and OpenClaw is a special setup, right? So, Pi and OpenClaw, what we have there is that now we're not only talking about a single agent in a single session in a coding environment, but now we have a multi-channel environment where we have multiple threads going on, multiple agents going on. So, there's a little bit more to it. This is, and the interesting part, right? That's where I got started, is if you look into the packages, the core packages of Pi, all of them are used in OpenClaw, right? So, OpenClaw has this function, run, embed, a Pi agent, and it creates a session, right? So, sessions, Pi itself has a great session support, and it creates a session agent and streams all the information back. We have the coding agent, which we just talked about. We have agent core as the other part that we talked about. And there's two other minor or major packages, a Pi AI for the unified LLM abstraction and a terminal UI interface. There's, OpenClaw has built its own plugin mechanism, and that's because it's a different use case, right, and it has different requirements. So, you have a plugin support for a multi-channel routing, different provider orchestration, sub-agents, gateway support, all those things that you know by OpenClaw, but it's based around the core mechanics of Pi and leverages it. Cool. But one thing, and that's the major gist I would like to bring across, is what do we do now with this? What are other options for us to do? And this is one of the applications we've been building for a client. And the use case is a sales process. They get requests for proposals of ordering another system, right? Parts being sold by that company. And we're taking all this coding agent, all of that we're taking away, right? We're new fresh, new thinking, right? And look at the process from the get-go. So, an email comes in, right? We monitor basically that inbox. Then we have some gateway, because what we want to do is we want to forward this to different agents. Right? So here, I have multiple agents, right? The way it's structured is we have one agent per customer. And that agent has a general harness, right? Agent MDE, agents MDE as an example, but you can obviously also use different ones. And that helps understanding the role of that agent. In this specific case, it tells how to use the system and how to react to certain inputs, outputs, etc. Now, the other one is customer MDE, where we basically explain the agent, the specific customer might have specific quirks, right? Specific access, specific discounts, and all of that. And then, right, and that's what I said earlier, I like using sessions. Then, for each case, right, we're creating and reusing existing sessions, so we can back and forth know what was previously talked about. All right. So email comes in, we're looking at the inbox, and we route this to different agents. And now we have tools, right? So we have these different tools to talk to the CRM, to talk to the ERP, and get the right information out of the system for this agent to behave. Maybe it has new contacts information or that sort. And again, we make this available, we make it easy for the agents to access, right? And our way currently is doing this with CLIs, right? So CLIs, our agents are really good at using CLIs, so we make it available as a CLI. We put, we make sure that the data is secure, we have our own sandbox, and then we're creating the drafts again, right? So that's the system, and I hope by this point you basically understand logically where these things fit together, but how would this look like? Oh, one final thing, right? There is always the question around sandboxing, et cetera. And to be honest, we're on the steps of getting there. But if you've seen NVIDIA's announcement around OpenClaw, their policy, their OpenShell is really interesting. And it's a way of securing an agent. We're looking into this, please do as well. All right, so how does this look like? To get you an understanding of how these things, right? So here's the dashboard, rather boring. But here's the email, the inbox, right? So again, we see the email coming in. And yeah, we, it's one of many emails, most of them are ignored. But this one is the DLM call said, okay, I'm interested in this. And it is associated to a case, right? We see the case up there. Now this case is an agent session, right? So we find the session and associate it to it. We then create a draft. So there's tons of calls, which I'm going to show you in a second. But basically the output of all that is a draft email that the user will be able to use, right? So our thinking is let the user stay in an email, let them stay in the inbox and drafts. And they don't even need to do a lot. So this is more like an admin interface. They can stay in email. But basically the output would be a draft generated. And how does that look behind, right? We had the different sessions before the threads. And this is the same thing, right? The assistant says, well, apologies, it's German. But now I'm looking at the articles. It does different tool calls, right? It gets results and does this in a loop to result, right? The end effect for the user is I'm looking at my inbox. There's a new email. It's associated to a case. And I get a new draft, which they can freely edit. But under the hood, we have all these agents working. All right. That's it for me. Again, here you find the slides. Key takeaways, please. Coding agents are and will be a core building block for your software systems. I'm betting on it. A lot of people are betting on it. It gets results and does this in a loop to result, right? The end effect for the user is I'm looking at my inbox. There's a new email. It's associated to a case. And I get a new draft, which they can freely edit. But under the hood, we have all these agents working. That's it for me. Again, here you find the slides. Key takeaways, please. Coding agents are and will be a core building block for your software systems. I'm betting on it. A lot of people are betting on it. So please give it a try. Pi is perfect for tinkering. Whether you like it or not, it's minimal.