The Death of Developer Advocates — Stephanie Jarmak, Sourcegraph

.

Hi everyone, sorry for the start with technical difficulties and all of that. We made it to the end of this track, super exciting. Thank you, everybody, for sticking it out this long. Are there any developer advocates or DevRel people in the audience? Raise your hand. Yeah, okay. So did you come to throw tomatoes at me? Because I'm talking about the debt now. Okay, so it's not going to be all doom and gloom like that. A bit of backstory in this. I'm a research scientist. So last year I was an astronomer, and I just wound up. I didn't know what GTM was or any of that. I just wound up in this.

And I submitted a much boring, science-y eval talks that were unceremoniously, I assume, being thrown into the trash for this conference. But my manager, who is a developer advocate, put in the death of developer advocates, which is appropriately buzzwordy and hypey. And so that was great. But his title is developer advocate, so it didn't necessarily make as much sense for him to be coming up here and giving his eulogy. So we brainstormed maybe I would dress up as a robot and maul him and attack him on the stage or something like that. But then, logistically, it was going to be hard to do that. So he just went on vacation.

So I'm here as the agent advocate to talk about this new role and try to advocate for it and convince all of you that we should all be agent advocates to help in this new era. So zooming out a little bit and going back in time a bit, because I was trying to talk about developer advocates to somebody at the conference yesterday and their eyes glazed over, they had no idea what I was talking about. So just to talk about what this thing is that I'm saying is dead. So back in the 80s, right, it was called software evangelism, where one would go forth and speak the good word of the product and bring it out there.

But then fast forward to the 2010s or so, that's when developer advocacy started to become a thing, where now instead of having this single trajectory of the communication pathway, now it's a feedback loop and a two-way street where you have these people with very deep empathy for developers who understand them and speak their language and could understand what their needs were and then bring that back to the product. And then these developers, fast forward even more, have so much influence within their company and become these king makers. And so developer experience became a very important aspect of the go-to-market strategy.

But now in 2026, developers are no longer working alone, and what it means to be a developer is completely changing. And so our role, as developer advocates, developer and developer relations, we're relating to developers. And so as the role of developer is fundamentally changing, so must the role of the developer advocate. So in this slide I'm just talking about the other users, right, so what's happening with DevRel outside of the agent. So most of the talk is going to be talking about the agent as a user. But I also did want to bring up that engineers, they're becoming these orchestrators of these fleets of agents, babysitters and whatnot of these things.

And their job, all of the job postings and whatnot, the language is continuously changing, right. They're expected to have this AI fluency. And at the same time, there's also people like me, non-engineers. I was a research scientist. I had zero commits on GitHub last year, and now I have 12,000 and I'm an open source maintainer for multi-agent orchestration framework. We have so much capability now with all of these agents, and now anybody with these agents can use DevTools essentially. So you have this whole other persona in ICP to potentially be relating to and having empathy with when they're using your product.

So let's talk about now this whole new user that we have in the form of an agent. So an agent is somewhat unique, right, in the sense that it is both the user of your tool in a very similar way to the developer. It's going out and reading your docs, but it's just reading them differently because it's a machine. It's calling API. It's had its own frustrations with how it's encountering errors and recovering from them, right. But then it's also a recommender of your tools, somewhat similar to developers in the way that they are also recommenders of your tools in a more organic bottom-up way. So the whole basis for DevRel is to encourage that bottom-up adoption.

But now the adoption, the recommendation system, a lot of it's being driven by the agent itself that is either maybe servicing your product directly through ChatGPT or Claude, directly in a Q&A environment, or it's, as we had heard in some of the previous talks where the speaker asked folks, how many of you have just let your agent install a library for you? And many hands went up, right? So there is this recommender of tools where basically it's just installing these frameworks and things directly and embedding them into the workflow and working with the developer in that taste. So I know it's late for numbers. You don't have to read them or anything like that.

So I have a couple different concrete examples for measuring these seats, right? So I am a data science scientist nerd person. So one of my first projects when I was working on this, when I became an agent advocate, was to build a benchmark called Code Scale Bench. And so I developed hundreds of tasks that were reflective of the software development life cycle. And I basically unleashed these agents with and without our product tooling. So I work at Sourcegraph, and we have a code navigation MCP tool. And the point of that was to understand, okay, how is our tool helping the agent do the work that it's going to be doing?

And when it isn't working well, why isn't it working well? So that we can then go in and actually fix that. So I have thousands and thousands of these traces. And as we have heard in the previous talks, now we have these amazing logs of data for these really tight feedback loops where you can see exactly where it's breaking down and then go in and fix it. So this one specific example here was when I was looking at how it was using a read tool. And the model had these expectations based off of its biases from its training data of what it expected for a particular command that would be available within the tool.

And there's nothing in our description that would have led it to believe otherwise. So it tried to use read line instead of start line or something like that. And then it ended up failing, but then at least the error told it why it failed. So it was like, okay, that was a good part of it. So it was able to fix itself. But then it's burning an entire turn just failing when you could just go in and fix that aspect of how it's interacting with the tool.

And this is really important, right, to gather that feedback and understand the friction that now your new agent user is having with your tool because it's the way that different organizations are going to be evaluating your tool, right, in terms of not just is it working well, but how many tokens is the agent dealing with to work with your tool and how fast is it. So this is really an important aspect of the role, to measure how these users are using it. The other side of it is the recommendation layer, right, so the GEO instead of SEO, so the generative engine optimization.

And I didn't mention it before, but in the previous slide I had a GitHub repo, there's two different toy projects that I put together. At the end of the talk there's a QR code with a link that you can send your agent to to have access to all this. So don't worry about taking screenshots or anything. All of the data will be released to you. So anyway, back to this. I set up a little experiment, right, to see how these different chatbots and agents and whatnot were recommending our product or mentioning it at all. And so there's a process to that because you want to understand what is your ICP actually doing when you would want your product to be serviced.

So there was a bit of a gap that I found. If I had designed some of these prompts around somebody who was actively shopping for this code intelligence tooling and doing a comparative sort of thing, then our product was ending up being recommended 65% of the time. But what I found was arguably the more typical use case and where we'd want to be showing up for people when they're encountering a specific pain or have a specific need where our product could serve them better, zero mentions, right? So in this particular instance, I put in a prompt that was like, we keep breaking downstream services when we change shared libraries because we can't see all the consumers.

chatbots and agents and whatnot were recommending our product or mentioning it at all. And so there's a process to that because you want to understand what is your ICP actually doing when you would want your product to be serviced. So there was a bit of a gap that I found. If I had designed some of these prompts around somebody who was actively shopping for this sort of code intelligence tooling and doing a comparative sort of thing, then our product was ending up being recommended 65% of the time. But what I found was arguably the more typical use case and where we'd want to be showing up for people when they're encountering

a specific pain or have a specific need where our product could serve them better, zero mentions, right? So in this particular instance, I put in a prompt that was, we keep breaking downstream services when we change shared libraries because we can't see all the consumers. And our one part of our product is being able to have this observability layer to see across all the repos. So we'd want some level of attribution or recognition from an agent to say, hey, you could use something like this. But instead it said, you could just have your developers make a wiki page or something. But we wouldn't know that without running these sorts

of experiments and getting this sort of data. So what this leads to is then you can have a hypothesis of, okay, maybe the messaging that we're putting out there isn't attributing some of these pains and use cases clearly enough for the agents to be picking it up. So we have a content campaign in the works to make changes to our website, and then we can directly measure whether that has an actual lift and not necessarily in the form of anything that was baked into the training data, but then how the agents that are using those web search tool calls, how they are then interpreting the information about your product.

So there are just some different ways that you could think about guiding the agents to help support the surfacing, the discoverability of your product and this user finding it at their moment of need, right? So, for example, this whole field is moving so fast. So training data is always going to be stale. Actually, in the GEO pilot study that I did, the data that I was showing there that was using Claude Sonnet 4, very old obviously, and I just today, this afternoon, ran it with 4.6 thinking that, okay, surely it's going to be better. It's going to know improved information about our product. But in the previous model, it kept pitching Cody,

which was one of our older products. But when I ran it again, it pitched Cody even more, right? Because now you have all of these old models outputting content that then is compounding on the internet. So you have to figure out how to bury all of that noise with your true signal. And the way that some folks are working on that is, as we've heard from other people, these LLMs.txt sort of pages, right? So you have more authoritative sources of truth that you're hoping to direct the agent to. But they still need to be using the tools and using real-time information in Providence to be able to give accurate answers about your product.

You also want to give the agent something to quote, right? They want to bring something that they can really sell to the user, right? So you want current examples and keep everything up to date. Even if your stuff hasn't changed in two years, which would be shocking, even if it hasn't, keep everything up to date and fresh because part of that is how they have their relevance algorithm. And they also really, really like charts and FAQs and things like that. And you also want to make sure your product is where the agents are, right? You're going to market. So go to agent market, right?

So make sure you're in the marketplace and the MCP registries everywhere that you would expect an agent to be able to easily find you. And also make sure that that whole, you reduce as much friction as possible for an agent or a developer to go from finding out about your tool to embedding it in their workflow. Because if an agent realizes your tool requires three different demos and emailing sales reps and stuff, they're never going to say, hey, user, here's what you should do, but FYI, you're going to have to do all this other stuff. It's just not going to happen. And then also make sure that you are covering those pains, right?

Because that's how a user is going to be in their time of need, right? That's going to be the best opportunity for your product and your service, right, to be surfaced to them. And so you want to make sure that there's enough content out there on the internet for the agent to be aware of that and make those connections for you. And so there's this ongoing question of what even the heck is DevRel and advocacy and now this agent advocacy thing, right? It's like where does it fit? Where does it go? Is it engineering? Is it product? Is it marketing? Yeah, yes, yes, it's all of those things. And with the rise of agents, it hasn't gotten any clearer, right?

Those teams haven't gotten any clearer. If anything though, everybody's role across the organization has gotten fuzzier. So that actually helps in a lot of ways. And you can split it up and think about it in terms of these different flavors, right? And you can mix and match depending on whatever skills and abilities various employees have within your organization and whatever the product needs at a given time. So you have the engineering flavor, right? And those are folks that are partnering directly with the engineering team to make these interfaces for how the agent is talking to your product, through the MCP server and building out these evals and the instrumentation.

Then you have the product flavor. So those are folks that are going to own the end-to-end agentic experience, right? And so translating these evals to bring it to the product team and having the agent experience rubrics and how they're encountering all of that content. And then you have the marketing flavor, right? And that should be the folks that are really owning that pipe gen and how the agents are entering the funnel and finding out about your product and then bringing the developers along with them by surfacing those recommendations. So I know I said the death, right, of developer. I said the developer advocates, but the core of DevRel still holds.

It's just you have a change in your audience. So it's still extremely important to do enablement, right? It's just the type of enablement is a bit different. You're educating developers now who have a completely different type of job where they're orchestrating these fleets of agents. And you're also educating agents, right? So you're having to put out content that is machine readable, has agent-friendly APIs, all of these things that make it as easy as possible to use your product both for human developers and for the agents that they're using. And community is also more important than ever, right?

Having that human-to-human connection where developers can come and bring their agents also into the loop, right? So that's another component that needs to be considered when you're building these different communities because there's all these questions, right, of privacy and data concern as well. If people are bringing their claws and whatnot into the Discord and they're recording all of the conversations and everything, it's just a new thing that you have to think of as a community builder. And then there's the feedback loop. So you're still responsible for bringing the voice of the developer who's using the agents back to the organization,

but then you can also spin up thousands of these agents to perform experiments on them, experiments that you can't really do as easily with the developers who don't want to maybe talk to you that much. And then credibility, right? So you need to be earning credibility both from human developers. So don't, not using Claude's slop at them, right? Then tell your AEs to stop that as well. Everybody knows what it is and nobody likes it. But then credibility, actually Claude loves its own slop for whatever reason. So there's a bias, right, from agents of their own content. So whenever you're making agent-facing content,

as long as it's structured, you can have as many em dashes and whatever as it wants. But it's just a completely different sort of credibility landscape, human. So what I'm advocating for here, right, is building out a curb cut. So curb cuts were built for wheelchairs, built for a specific user to use them. But now everybody benefits from that, right? Anybody with wheels, right? Strollers and suitcases and all of those things. So my argument is that by serving the agents, the human path gets clearer too. There's just one more user in the room now, but they are still serving the human on the other end, and we're all working together on this.

But then credibility—actually, Claude loves its own slop for whatever reason. So there's a bias from agents of their own content. Whenever you're making agent-facing content, as long as it's structured, you can have as many em dashes and whatever as it wants. But it's just a completely different sort of credibility landscape, human. So what I'm advocating for here is building out a curb cut. So curb cuts were built for wheelchairs, for a specific user to use them. But now everybody benefits from that, right? Anybody with wheels, right? Strollers and suitcases and all of those things. So my argument is that by serving the agents, the human path gets clearer too.

There's just one more user in the room now, but they are still serving the human on the other end, and we're all working together on this. So for DevRel, one quick thing that you could do right away is point a coding agent at your docs and then look through that transcript and start developing your agent experience report. And then if you're more on the GTM side, start developing some of these experiments with the GEO, putting together those prompts and looking at the mentions versus recommendations. And I made this whole talk agent legible, right? So there's a QR code there, as well as a couple of different toy repos that have some templates for you to get started.

And that's it. That's it. That's it. That's it. That's it. That's it. Like yeah, yes, yes, it's all of those things. And with the rise of agents, it hasn't gotten any clearer, right? Those teams haven't gotten any clearer. If anything though, everybody's role across the organization has gotten fuzzier. So that actually helps in a lot of ways. And you can sort of split it up and think about it in terms of like these different flavors, right? And you can mix and match depending on whatever skills and abilities various employees have within your organization and whatever the product needs at a given time. So you have like the engineering flavor, right?

And those are folks that are partnering directly with the engineering team to make these interfaces for how the agent is talking to your product, like through the MCP server and building out these evals and the instrumentation. Then you have the product flavor. So those are folks that are going to own the end-to-end agentic experience, right? And so translating these evals to bring it to the product team and like having the agent experience rubrics and how they're encountering all of that content. And then you have the marketing flavor, right? And that should be the folks that are really owning that pipe gen

and how the agents are like entering the funnel and finding out about your product and then bringing the developers along with them by surfacing those recommendations.

So I know I said the death, right, of developer. I said the developer advocates, but the core, right, of DevRel still holds. It's just you have a change in your audience. So it's still extremely important to do enablement, right? It's just the type of enablement is a bit different. You're educating developers now who have a completely different type of job where they're orchestrating these fleets of agents. And you're also educating agents, right? So you're having to put out content that is machine readable, has like agent friendly APIs, all of these things that make it as easy as possible

to use your product both for human developers and for the agents that they're using. And community is also more important than ever, right? Having that human to human connection where developers can come and bring their agents also into the loop, right? So that's another component that needs to be considered when you're building these different communities because there's all these questions, right, of privacy and data concern as well. If people are bringing their claws and whatnot into the Discord and they're recording all of the conversations and everything, it's just a new thing that you have to think of as a community builder. And then there's the feedback loop.

So you're still responsible for bringing the voice of the developer who's using the agents back to the organization, but then you can also basically spin up like thousands of these agents to perform experiments on them, experiments that you can't really like do as easily with the developers who don't want to maybe talk to you that much. And then credibility, right? So you need to be earning credibility both from human developers. So like don't, like not using Claude's slop at them, right? Then tell your AEs to stop that as well. Everybody knows what it is and nobody likes it. But then credibility, like actually Claude loves its own slop for whatever reason.

So there's a bias, right, from agents of their own content. So whenever you're making like agent-facing content, as long as it's structured, you can have as many em dashes and whatever as it wants. But it's just a completely different sort of credibility landscape, human. So what I'm advocating for here, right, is like building out a curb cut. So curb cuts were built for wheelchairs, like built for a specific user to use them. But now everybody, you know, benefits from that, right? Anybody with wheels, right? Strollers and suitcases and all of those things. So my argument is that by serving the agents, the human path gets clearer too.

There's just, you know, there's just one more user in the room now, but they are still serving the human on the other end, and we're all working together on this. So for, you know, DevRel, one quick thing that you could do like right away is point a coding agent at your docs and then looking through that transcript and start developing your agent experience report. And then if you're more on the GTM side, start like developing some of these experiments with the GEO, putting together those prompts and looking at the mentions versus recommendations. And I made this whole talk agent legible, right? So there's a QR code there, as well as a couple of different toy repos

that have some templates for you to get started. And that's it. That's it. That's it. That's it.

That's it. That's it. That's it.