Software Engineering Is Becoming Factory Engineering — Zach Lloyd, Warp

Transcript

Zach Lloyd Okay. Hello, everyone. I'm excited to be here. My name is Zach Lloyd. Today I'm going to be talking about self-improving software factories, the new open source model, and what I think is happening to development. A little bit about me just to begin. So I am a former principal engineer from Google. I used to lead engineering on the Google Docs suite. I've been an engineer now for over 20 years, a long time. I am still shipping frequently, but I haven't written a line of code in the last six months. And I'm the founder of a company called Warp. Warp, if you're not familiar, is an open source agentic development environment. You may know us as a terminal. That is how the company started. We're a terminal that has agents built in. We open sourced it a couple months ago, and I'm going to talk a little bit about that experience and the motivation for it. It's a popular open source project, over 60,000 GitHub stars. We've had a couple hundred people contributing. We have over 800,000 active developers who are using Warp. And increasingly, we are focused not just on the terminal aspect and the interactive aspect of development, but more so on how do you automate development. I'm going to talk mostly about that. So the thesis that I have is that the discipline of software engineering is going to become something more like factory engineering. And I'll explain what I mean by this in a minute, but just keep that in mind. That's what I think is going to happen. If you look at development over the past couple years, it's crazy how it's changed. We've gone from a world of chat and AI autocomplete, cursor, co-pilot, to the phase that we're in now, which I consider to be mostly interactive agents. So you're sitting at your computer and you are telling Cloud Code to do something. You're telling Warp to do something. And I believe what's going to happen over the next six months, a year, hard to predict the pace, is that we're going to move much more towards a world of automation. But before I get into that, just a quick show of hands, how many folks in here are building with agents? 100% makes sense. How many folks are building typically with multiple agents at one time? So again, almost everyone. How many people are running agents in the cloud out of curiosity? So that looks like less than half, but still significant. And how many folks have set up a system internally to automate the whole software development life cycle? So everything from triaging, spec-ing, implementing, reviewing. So I see some hands. So some people are doing this. So this is what's going to happen. Every project of significant size, I believe, is going to have something like this. And it's going to look like this big loop. And everyone is talking about loops. There's nothing that complicated about loops. This loop says the cloud software factory. This loop could literally just say the software development life cycle. It's the same thing. But just to go through this loop, ideas are going to come in at the top. Agents are going to do triage. If something is complicated, they will write a spec. These little blue boxes are where humans step in. Humans will review the spec. Agents will do the implementation. A human and agent will review the code. Agents will verify. Human will review the product. You ship, and then you monitor, and round and round you go. And this is what software development, for better or worse, I think is going to end up looking like. So I repeat the thesis, which is that if this is what software engineering is going to look like, software engineers are going to be the ones who end up building and managing these factories. Now, I promised at the beginning, and I put in the title of the talk that I was going to talk about open source. And so I want to do that for a few minutes. I'm going to take a quick digression. I bring up open source because one of the main reasons that Warp open sourced was to build a public factory. And so this is a picture of this website we've built called build.warp.dev, which shows all of the issues that are flowing through our system, and what state they're in, what agents are working on them, what contributors are working on them. And it's a proto factory done at scale. It's not working perfectly, but it is working. And one of the reasons we open sourced was to try to build this. Just in general, I think it's interesting to talk about open source in the time of agentic development. This is a really stupid graph, but it's that it's becoming much cheaper to build software. A corollary of that is that it's becoming trivial to clone software. And so if you are in the software business, I don't know how many folks in this room are in the software business per se, but it's very hard to build a software business if it's free to build software. It's hard to capture the value, especially if a competitor can clone. And so my big takeaway or a big tip for everyone in here is that the first thing you should do is patent your code. I'm kidding. This is a complete joke. Don't do this. My first tip is obviously you need to have a great product. This has always been the case, but I would say a great product probably was never enough, but even now more than ever, if you think that you're going to build a great software business just by building and shipping a great product, you're probably not going to succeed. You need advantages beyond the product. And so those advantages could look like distribution, ecosystem, it could be that you have a great brand or data moat, you might have capital. But if you're a startup, again I'm coming from the startup world here, you just don't have these advantages. And so you still want to break through. And one of the ways that I suggest doing this is by building in the open. And so to be clear, it took Warp five years of building closed to sort of make the leap into building in the open, and I'll explain why. But if you build in the open, it helps build your ecosystem. It can take you from being hated on Hacker News to tolerated. It can burnish your brand, it creates community, and so there's all these advantages to it. And some of the things that I think have traditionally been a pain can now be managed. And so the traditional pain of open source might be something like you get a lot of noisy issues. You get sloppy PRs, you can end up in code review hell, you can end up having to spend a lot of time verifying changes. And so the solution, it's a long-winded way of getting to this, for open source, or at least for Warp in the case of open source, the thing that made us finally decide to do this was that we built a whole set of automations, really a software factory, around managing the open source project. And so this is what I think the future is going to look like. I'm going to drill into it a bit just to get a little bit more technical for folks who want to try to build something like this for their own projects. So what are the components of an effective software factory? It's really not that complicated to start or at a high level. You need a set of automations, you need a way of providing context and skills, you need a way of bringing humans in at the correct time, so when things get stuck on the factory. And then a really important thing is you need some set of self-improvement capabilities. So think of this as loops. And if you do this right in the open source world, you can get a world where agents are helping contributors contribute, they're helping maintainers maintain. And I want to emphasize there's nothing special about open source here. I think every sizable project can benefit from this approach, and I predict that every company, every open source project will have, at its core, a software factory, the way that CI CD became just, of course you have that. Maybe, I don't know when that happened 10 years ago. Let's tour the factory floor for a second here. So you're not going to look at this. This is too much. The point of this slide is not to have you read the workflow. It's that the factory floor is basically a graph of steps where you are defining how software gets built for my product. And it looks pretty similar for every product. Things come in, they flow through, they get stuck at certain points. And broadly speaking, just to back out a second, so there's the inputs. The inputs are really ideas. The inputs could be coming from your team, they could be coming from your users. The inputs themselves tend to come in through certain channels that you should think about as your task tracker is an obvious one, or Slack, your teams, your communication channels. They could come directly from your terminal or IDE. They could come from your monitoring systems. But there's some set of inputs. here. So you're not going to look at this. This is too much. The point of this slide is not to have you read the workflow. It's that the factory floor is basically a graph of steps where you are defining, okay, how does software get built for my product? And it looks pretty similar for every product. Things come in, they flow through, they get stuck at certain points. And broadly speaking, just to back out a second, so there's the inputs. The inputs are really ideas. The inputs could be coming from your team, they could be coming from your users. The inputs themselves tend to come in through certain channels that you should think about as your task tracker is an obvious one, or Slack, your teams, your communication channels. They could come directly from your terminal or IDE. They could come from your monitoring systems. But there's some set of inputs that bring work into the factory. There's triage. This is a really important step. So, again, I boil it down to something very simple. But you want an agent that is looking at issues as they come in, and just saying, you know, if this is easy and this is unambiguous, just implement it. And this is how you can actually get going with a factory. If an issue is hard, I recommend having an agent that produces specs. Folks in here using spec-driven development, show of hands. People follow this? Okay. You can do this many different ways, but I think it's very effective. The way that we do it at Warp that I recommend is having an agent write what we call a product spec and a tech spec. Product spec describes the product invariance that you're building towards. Tech spec describes the architecture and the shape of the code. Then you have an implementation agent. This is basically a coding agent that runs somewhere in the cloud. It makes a diff. You can use all sorts of coding agents for this. You have review. This is, in many ways, the most painful part. I expect that people are a little bit tired of reviewing agentic slop. I would have an agent do code review first. And then it becomes, over time, a risk management exercise of, when do you bring in humans to do code review? But you want to have a step in here where humans can do it. This is a very important step for certain types of apps. The verification step. So this would be things like computer use. If you're building a sort of UI, having the computer actually use the code that the agent produced in producing videos and screenshots. CICDs still use it, obviously. And then monitoring. So agents don't stop in your factory when code is shipped. They should observe what's been shipped. Is it crashing? Is it being used? And round and round you go, because you take the output of this monitoring step and you feed it back into the top of the factory. Now you could try and build this. And I went to a talk earlier that my friend Adam gave where Uber has built an internal version of this. And it's pretty cool. I would say for most organizations, it really depends where you are, you'll be able to build a simple version of this easily. But to build a thing that actually scales, it's probably you should probably be focusing on your own product, not building this infrastructure, because there's a lot of stuff that you end up wanting. You're not supposed to read this. It's just a lot of stuff. If you do build it, or if you buy it, you'll end up with something that looks like this, which is you're going to have a bunch of ways of getting work into your factory. That's what's at the top here. You're going to have a sort of control plane for figuring out how work gets distributed across your factory floor. You're going to have the actual place where the work happens. And so that's going to be cloud sandboxes. It's going to be figuring out what agent to run. So what's the harness? What's the model? And then finally, I think this is a really important thing. You're going to want to set up some kind of data plane that sits below your factory. And so that's something that lets agents remember what they've done, learn, improve over time. The factory is not just a product. It's also a mindset. And this brings me back to the thesis I had at the beginning. You need to measure and improve. So factory, this is where the, I don't know, you can stretch this metaphor as far as you want, but you should be thinking of efficiency. And so that means how much software did you ship? How much did it cost in terms of human time and token time? And you're going to want to measure this and try to improve it over time. A key part of this is creating loops. So loops are, again, they sound complicated. They're not that complicated. Loops are basically ways of having agents improve by observing what they're doing, where they're failing. And so a common kind of loop that you're going to want to put in your factory is a skill loop. That means you're going to have your factory agents that are running skills. And then you'll have observer agents that are seeing how those skills are being applied, looking for issues and trying to improve the skills. So for instance, if you had a code review agent and it was leaving comments and a senior engineer on your team is going and correcting those comments, you'd want an observer agent that would look at that and basically improve the code review agent for the next run. This is one thought just to leave folks with, where does this leave engineers? I think you're going to have to get into this mindset, and I'm trying very hard to get our team into this mindset, it's not always easy, that you're not just building the product, but you're building the thing that builds the product. And that's different. It's more like process engineering or manufacturing or something like that. And you could think, okay, maybe that's a bummer. Is that a bummer? And it depends. It depends what joy you get out of software engineering. If your joy is in writing the code, I think everyone in here who is a software engineer is going to be writing less code. But if your joy is in shipping product, it's never been a better time. And this is actually where I find my joy. I like building and shipping the thing. So everyone in here is going to code less, but they're going to ship more. And that's going to be a trade-off. But if you approach it like you're a factory engineer, I think you can see that there's still a really cool set of engineering challenges. You can almost think of it as meta engineering. How do you engineer your system of agents to be the best possible engineering? That I think is a very compelling and interesting set of challenges to solve. So that's it. For folks who are interested, if you follow the link on this QR code, I've set up an open source GitHub repo where anyone who wants to try building their own factory agents can do it. This uses Warp's agent platform as part of it. But you honestly don't have to use it. I'm not trying to push into our product. But this should give you a good sense of, okay, if you want to set up an agent that does triage or an agent that does spec writing, how do you actually do that? How do you get from the theory of working with a factory to actually putting it into practice? I don't know if we have the capability to do questions in here. I saved a few minutes for questions if anyone has questions. Otherwise, I will wrap up. Yes? Yes. It's a great question. So I said something that's almost contradictory. I think you should think of it like, everyone's going to deploy some sort of factory. But then the tuning of the factory, the are these the right skills for my domain? Is this factory building my product in the right way? I still think there's a bunch of interesting engineering challenges. And for some places, you can build this. But again, I think you should probably be focusing on building the core product for your company for the most part. But there's a bunch of tuning and figuring out how to make the factory work for your product that matters. That's a great question. Yes? Yes. Yes. Yes. So the question in case people couldn't hear was if I was a college student graduating and entering the workforce right now. So I think that the most important skills in this new world are adaptability. I think that's critical thinking. It's the speed at which you can learn. I do think, I don't know if you're a computer science student, but I still think there's a ton of value in understanding the underlying systems and architecture and being able to reason and product in the right way? I still think there's a bunch of interesting engineering challenges. And for some places, you can build this. But again, I think you should probably be focusing on building the core product for your company for the most part. But there's a bunch of tuning and figuring out how to make the factory work for your product that matters. That's a great question. Yes? Yes. Yes. Yes. So the question in case people couldn't hear was like, if I was a college student graduating and entering the workforce right now. So I think that the most important skills in this new world are adaptability. I think that critical thinking is key. It's the speed at which you can learn. I do think, I don't know if you're a computer science student, but I still think there's a ton of value in understanding the underlying systems and architecture and being able to reason and understand the code, understand the specs that agents are writing. So I would focus on those skills. We're hiring more people than we've ever hired. There's a lot of misdirection around people not being hired because of AI. That's not the experience we've had so far. And what I'm looking for are really adaptable product focused thinkers who can be great problem solvers, even as the underlying technology changes. Yeah, I think I have time for one more question. Yes? What about the product structure, discovery, ideation, design, things? What about those? Like, so the question was, what about the product taste, how do you actually build something useful? I think that's probably the right framing, and where do the ideas come from? And so I think that the problem with the factory metaphor, even though I'm leaning into it because I think there's something to it, is that it can sound like mechanizing or dehumanizing. And I still think underlying all of this, the only thing that matters is are you building something useful? And if you have a factory that is churning out stuff that no one cares about, what's the point? And I think that human taste, human input, human product sense, humans guiding at those touch points where you can't automate stuff is absolutely essential. And that's what I do. I'm trying to figure out what do customers want? What do people want? What's going to be valuable to them? So I think that's an absolutely key point to it. I think I'm at time. So I have to go. I hope folks enjoyed this chat. I'm really grateful for being invited to speak. So thank you all very much. Thank you. telling Cloud Code to do something. You're telling Warp to do something. And I believe what's going to happen over the next six months, a year, hard to predict the pace, is that we're going to move much more towards a world of automation. But before I get into that, just a quick show of hands, how many folks in here are building with agents? 100% makes sense. How many folks are building typically with multiple agents at one time? So again, almost everyone. How many people are running agents in the cloud out of curiosity? So that looks like less than half, but still significant. And how many folks have set up a system internally to automate the whole software development life cycle? So everything from triaging, spec-ing, implementing, reviewing. So I see some hands. So some people are doing this. So this is what's going to happen. Every project of significant size, I believe, is going to have something like this. And it's going to look kind of like this big loop. And everyone is talking about loops. There's nothing that complicated about loops. This loop says the cloud software factory. This loop could literally just say, like, the software development life cycle. It's the same thing. But just to go through this loop, it's like ideas are going to come in at the top. Agents are going to do triage. If something is complicated, they will write a spec. These little blue boxes are where humans step in. Humans will review the spec. Agents will do the implementation. A human and agent will review the code. Agents will verify. Human will review the product. You ship, and then you monitor, and round and round you go. And this is what software development, for better or worse, I think is going to end up looking like. So I repeat the thesis, which is that if this is what software engineering is going to look like, software engineers are going to be the ones who end up building and managing these factories. Now, I promised at the beginning, and I put in the title of the talk that I was going to talk about open source. And so I want to do that for a few minutes. I'm going to take a quick digression. I bring up open source because one of the main reasons that warp open source was to build a public factory. And so this is a picture of this website we've built called build.warp.dev, which shows all of the issues that are flowing through our system, and what state they're in, what agents are working on them, what contributors are working on them. And it's kind of like a proto factory done at scale. It's not working perfectly, but it is working. And one of the reasons we open source was to try to build this. Just in general, I think it's interesting to talk about open source in the time of agentic development. This is a really stupid graph, but it's like, you get it? It's becoming much cheaper to build software. A corollary of that is that it's becoming trivial to clone software. And so if you are in the software business, I don't know how many folks in this room are in the software business per se, but it's very hard to build a software business if it's free to build software. It's hard to capture the value, especially if a competitor can clone. And so my big takeaway or a big tip for everyone in here is that the first thing you should do is patent your code. I'm kidding. This is a complete joke. Don't do this. My first tip is obviously you need to have a great product. This has always been the case, but I would say a great product probably was never enough, but even now more than ever, if you think that you're going to build a great software business just by building and shipping a great product, you're probably not going to succeed. You need advantages beyond the product. And so those advantages could look like distribution, ecosystem, it could be that you have a great brand or data moat, you might have capital. But if you're a startup, again I'm coming from the startup world here, you just don't have these advantages. And so you still want to break through. And one of the ways that I suggest doing this is by building in the open. And so to be clear, it took Warp five years of building closed to sort of make the leap into building in the open, and I'll explain why. But if you build in the open, it helps build your ecosystem. It can take you from being like hated on Hacker News to like tolerated. It can burnish your brand, it creates community, and so there's all these advantages to it. And some of the things that I think have traditionally been a pain can now be managed. And so like the traditional, you know, pain of open source might be something like you get a lot of noisy issues. You get sloppy PRs, you can end up in code review hell, you can end up having to spend a lot of time verifying changes. And so the solution, it's kind of a long-winded way of getting to this, for open source, or at least for Warp in the case of open source, the thing that made us finally decide to do this was that we built a whole set of automations, really a software factory, around managing the open source project. And so like I said, this is what I think the future is going to look like. I'm going to drill into it a bit just to get a little bit more technical for folks who want to try to build something like this for their own projects. So what are the components of an effective software factory? It's really not that complicated to start or at a high level. You need a set of automations, you need a way of providing context and skills, you need a way of bringing humans in at the correct time, so sort of like when things get stuck on the factory. And then a really important thing is you need some set of self-improvement capabilities. So think of this as loops. And if you do this right in the open source world, you can get a, you know, world where agents are helping contributors contribute, they're helping maintainers maintain. And I want to emphasize there's nothing special about open source here. I think every sizable project can benefit from this approach, and I predict that every company, every open source project will have, at its core, a software factory, kind of like the way that CI CD became just like, oh, of course you have that. Maybe, I don't know when that happened 10 years ago. Let's tour the factory floor for a second here. So you're not going to look at this. This is too much. The point of this slide is not to have you read the workflow. It's that the factory floor is basically a graph of steps where you are defining, like, okay, how does software get built for my product? And it looks pretty similar for every product. Things come in, they flow through, they get stuck at certain points. And, you know, broadly speaking, just to back out a second, so there's the inputs. The inputs are really ideas. The inputs could be coming from your team, they could be coming from your users. The inputs themselves tend to come in through certain channels that you should think about as, like, your task tracker is an obvious one, or Slack, your teams, like your communication channels. They could come directly from, like, your terminal or IDE. They could come from your monitoring systems. But there's some set of inputs that bring work into the factory. There's triage. This is a really important step. So, again, I boil it down to something very simple. But, like, you want an agent that is looking at issues as they come in, and just saying, you know, if this is easy and this is unambiguous, just implement it. And this is how you can actually get going with a factory. If an issue is hard, I recommend having an agent that produces specs. Folks in here using spec-driven development, show of hands. People follow this? Okay. You can do this many different ways, but I think it's very effective. The way that we do it at Warp that I recommend is having an agent write what we call a product spec and a tech spec. Product spec describes the product invariance that you're building towards. Tech spec describes the architecture and the shape of the code. Then you have an implementation agent. This is basically a coding agent that runs somewhere in the cloud. It makes a diff. You can use all sorts of coding agents for this. You have review. This is, in many ways, the most painful part. Like, I expect that people are a little bit tired of reviewing agentic slop. I would have an agent do code review first. And then it becomes, over time, like a risk management exercise of, like, when do you bring in humans to do code review? But you want to have a step in here where humans can do it. This is a very important step for certain types of apps. The verification step. So this would be things like computer use. If you're building a sort of UI, having the computer actually use the code that the agent produced in producing videos and screenshots. CICDs still use it, obviously. And then monitoring. So agents don't stop in your factory when code is shipped. They should observe what's been shipped. Is it crashing? Is it being used? And round and round you go, because you take the output of this monitoring step and you feed it back into the top of the factory. Now you could try and build this. And I went to a talk earlier that my friend Adam gave where Uber has built an internal version of this. And it's pretty cool. I would say for most, you know, most organizations, it really depends where you are, you'll be able to build a simple version of this easily. But to build a thing that actually scales, it's probably like you should probably be focusing on your own product, not building this infrastructure, because there's a lot of stuff that you end up wanting. You're not, again, not supposed to read this. It's just a lot of stuff. If you do build it, or if you buy it, you'll end up with something that looks kind of like this, which is you're going to have a bunch of ways of getting work into your factory. That's what's at the top here. You're going to have a sort of control plane for figuring out how work gets distributed across your factory floor. You're going to have the actual place where the work happens. And so that's going to be cloud sandboxes. It's going to be figuring out what agent to run. So what's the harness? What's the model? And then finally, I think this is a really important thing. You're going to want to set up some kind of data plane that sits below your factory. And so that's something that lets agents remember what they've done, learn, improve over time. The factory is not just like a product. It's also a mindset. And this brings me back to the thesis I had at the beginning. You need to measure and improve. So factory, this is where the, I don't know, you can stretch this metaphor as far as you want, but like, you should be thinking of efficiency. And so that means like, how much software did you ship? How much did it cost in terms of human time and token time? And you're going to want to measure this and try to improve it over time. A key part of this is creating loops. So, loops are, again, they sound complicated. They're not that complicated. Loops are basically ways of having agents improve by like observing what they're doing, where they're failing. And so a common kind of loop that you're going to want to put in your factory is like a skill loop. That means you're going to have your factory agents that are running skills. And then you'll have observer agents that are seeing how those skills are being applied, looking for issues and trying to improve the skills. So for instance, if you had a code review agent and it was leaving comments and a senior engineer on your team is going and correcting those comments, you'd want an observer agent that would look at that and basically improve the code review agent for the next run. This is one thought just to leave folks with, like, where does this leave engineers? I think you're going to have to get into this mindset, mindset, and I'm trying very hard to get our team into this mindset, it's not always easy, that you're not just building the product, but you're building the thing that builds the product. And that's like, it's just different. It's more like process engineering or manufacturing or something like that. And you could think like, okay, maybe that's a bummer. Like, is that a bummer? Is that, you know, and it depends. Like, it depends what joy you get out of software engineering. If your joy is in writing the code, I think everyone in here who is a software engineer is going to be writing less code. But if your joy is in shipping product, like, it's never been a better time. And this is actually where I find my joy. It's like I like building and shipping the thing. So everyone in here is going to code less, but they're going to ship more. And that's going to be a trade-off. And that's going to be a trade-off. But if you approach it like you're a factory engineer, I think you can see like that there's still a really cool set of engineering challenges. You can almost think of it as like meta engineering. Like, how do you engineer your system of agents to be the best possible of engineering? That I think is a very compelling and interesting set of challenges to solve. So that's it. For folks who are interested, this, if you follow the link on this QR code, I've set up a open source GitHub repo where anyone who wants to try building their own factory agents can do it. This uses Warp's agent platform as part of it. But you honestly, you don't have to use it. I'm not trying to like push into our product. But this should give you a good sense of like, okay, if you want to set up an agent that does triage or an agent that does spec writing, how do you actually do that? How do you get from like the theory of working with a factory to actually putting it into practice? I don't know if we have the capability to do questions in here. I saved a few minutes for questions if anyone has questions. Otherwise, I will wrap up. Yes? Yes. It's a great question. So I said something that's almost contradictory. I think you should, the way you should think of it is like, everyone's going to deploy some sort of factory. But then the tuning of the factory, the like, are these the right skills for my domain? Is this factory building my product in the right way? I still think there's a bunch of interesting engineering challenges. And for some places, you can build this. But again, I think, I think that the you should probably be focusing on building the core product for your company for the most part. But there's a bunch of like, tuning and like, figuring out how to make the factory work for your product that matters. That's a great question. Yes? Yes. Yes. Yes. So the question in case people couldn't hear was like, if I was a college student graduating and entering the workforce right now. So I think that the most important skills in this new world are adaptability. I think that that's critical thinking. It's like the speed at which you can learn. I do think, I don't know if you're a computer science student, but I still think there's a ton of value in understanding like, the underlying systems and architecture and being able to reason and understand the code, understand the specs that agents are written, sorry, are writing. So I would focus on those skills. And like, I don't know, we're hiring more people than we've ever hired. There's a lot of kind of like, misdirection around like, you know, people not being hired because of AI. That's not the experience we've had so far. And what I'm looking for are like, really adaptable product focused thinkers who can like, basically be great problem solvers, even as the underlying technology changes. Yeah, I think I have time for one more question. Yes? What about the product structure, discovery, ideation, design, things, and things? What about those? What about those? Like, so the question was, what about the product taste in like, the, how do you actually build something useful? I think is probably the right, like maybe the framing and like, where do the ideas come from? And so I think that the problem with the factory metaphor, even though I'm like leaning into it because I think that's like, that there's something to it, is that it can kind of sound like mechanizing or dehumanizing. And I still think underlying all of this, the only thing that matters is like, are you building something useful? And if you have like, a factory that is like, churning out shit that no one cares about, it's like, what's the point? And I think that human taste, human input, human product sense, humans like guiding at those touch points where you can't automate stuff is absolutely like essential. And like, that's like what I do. Like, I'm trying to figure out what do customers want? What do people want? What's going to be valuable to them? So I think that's an absolutely key point to it. I think I'm at time. So I have to go. I hope folks enjoyed this chat. I'm really grateful for being invited to speak. So thank you all very much. Thank you.