500 Skills, Zero Fine-Tuning: LinkedIn's Playbook for AI Agents — Ajay Prakash, LinkedIn

Transcript

Ajay Prasad- Hey everyone, good morning. Thanks for being here. I see people are still coming, but my name is Ajay and I am a software engineer at LinkedIn. Today I'm going to be talking about how we are doing context engineering to improve the performance of coding agents at LinkedIn. Imagine you are a software engineer in a big tech company and your products are being used by millions of users on a daily basis. And you happen to be on a team which owns a set of very critical services and you are on call. And you get an alert saying that there is an error spike in one of your services. By the time you are trying to figure out how to deal with this issue, you take the link to the alert and give it to a coding agent like Cloud Code or GitHub Copilot. While you are trying to figure out how to deal with the issue, the coding agent is working in the background. It will fetch the instructions on how to debug such issues in your company and identifies that based on that instruction, this alert is happening in a specific service. Then it fetches instruction and context on how to debug that particular service. And based on those instructions, it will take actions like fetching logs and metrics. Then it uses those logs to identify the root cause of the issue. It identifies based on the error logs where the issue is happening. And it doesn't just find the root cause, it also figures out the steps to mitigate the issue. And once it finds all the details, it summarizes and gives it to you, saying this is the error and this is the issue and these are the actions that you need to take to mitigate. And once you confirm, it also goes ahead and takes those actions on your behalf to mitigate the issue. And it doesn't just stop there. It updates your incident management system with all the details, error metrics, and dashboards. And also, it checks out the code and creates a PR for you to fix the root cause of the issue. All of this happens in a matter of a few minutes, which would have easily taken a few hours if you were to do it manually. This is not fiction. This is how teams at LinkedIn are using coding agents as effective coworkers with deep understanding of LinkedIn's internal systems and code to help the teams be really productive. And this is possible because of a system that we built called contextual agent playbooks and tools at LinkedIn. And today I'm going to talk about why we built the system, how we built it, and what are our learnings from the success. To understand why we built the system, we have to go back to the early days of coding agents. Just like any other company, even at LinkedIn, we wanted to use the coding agents to help our engineers be really productive with AI. So we started giving these coding agents to all of the engineers. And the problem was the coding agents don't really work in a large enterprise like LinkedIn. The biggest problem is the coding agent or the LLMs are trained on open source repos. So they don't have the context of how we maintain code bases at LinkedIn or our internal frameworks, our internal systems. What used to happen was the engineers used to do wide code or try agentic coding, but because the agents lacked context, they used to hallucinate and get stuck in between, or even more dangerous, they used to make up things which are not correct. So the engineers had to prompt these agents manually to do the right thing, which used to take more time than the manual coding itself. So a lot of engineers went back to manual coding. Coding agents were not effective. To understand the problem and get more perspective, if you look at the LinkedIn stack, we have over 1,000 repos, which make up thousands of microservices and apps. And all of these apps and services are built on a lot of internal frameworks and libraries. And we also have a lot of custom built infra. For example, we have our own databases. We have our own experimentation and tracking platform. We have our own configuration management system, which is purely internal to LinkedIn, and coding agents don't have any idea about them. And engineers go through a week-long boot camp whenever a new engineer joins, just to get familiar with these systems. So we looked at this problem and asked ourselves the question: how can we make any coding agent like Cursor or Cloud Code or GitHub Copilot understand our LinkedIn's internal systems so well that they can ship code that our engineers can trust? By trust, I mean the code should be correct, and the quality of the code should be as good as it is written by an actual engineer. So that is the bar we set up, and we wanted to figure out how to get there. In early 2025, Anthropic released MCP, and it quickly became the industry standard for building tools for agents. We leveraged that, and pretty early on, we built our own internal MCP, and the first tool that we built was CodeSearch. We have a pretty sophisticated code search system at LinkedIn, where engineers can go and search for code. It will search across thousands of repos using keywords and custom filters and regex. So we made that available to the coding agents via MCP. This was a really powerful unlock, because now you don't have to manually figure out how to do better search. The agent, you ask a question like, how do I set up a particular thing, and the agent can use the code search tools to figure out the right examples of how we do things at LinkedIn and use that to give you an answer and also implement it based on its findings. This was really powerful. So we added more tools. We added docs, Jira, Slack, and even connected to all of our data platforms and even feature flags. So every tool that we added to our internal MCP created more value. It's almost like a compounding effect, because now an engineer can bring in the PRDs, product requirement documents, and design docs, and also their Jira tasks, which have different contexts, and use all this to give to the coding agent to automate or help with their coding. But there was a problem. Even with a slightly complex workflow, the agents used to not do really well. For example, with the context and tools, the agent was able to answer basic questions and find code examples, but it could not do a complete job reliably end-to-end. The main problem was that to do a specific job end-to-end, it needs to have a lot of tribal knowledge. For example, how to fix a particular error, or how to configure, or how to debug a particular error log. All of this knowledge, even though you have access to the tools, is scattered across a lot of different surfaces. For example, docs, wikis, and Slack conversations. And most of the time, docs and wikis might be outdated or written poorly, and there might be duplicate docs. So the problem is the agents, even though they have access to the tools, used to get lost. The second problem was context overload. As agents use more and more tools, their context gets overloaded, which means every tool output takes up space in the context, which will eventually cause the agent to compact its context while it is working, which causes it to lose some of the information. Then it has to do it all over again. And the third problem was, even if the agent was able to figure out all these details, it doesn't have a way to retain this information. It doesn't have durable memory. So every time an engineer asks the agent to do a certain task, they have to start from scratch. So how do we solve this problem? We built a system in early 2025 called playbooks, where we not only provide the tools to the agents via MCP, we also allow the agents to access these instructions and prompts via MCP. We call it playbooks. And a playbook just appears just like any other regular tool. They have names and descriptions on what they do. And the agent can decide to invoke that playbook just like any other regular tool. And when the playbook is invoked, the instructions and the context within that playbook are returned as the tool output to the coding agent. So the agents have both tools as well as instructions on how to use tools to set up or perform a task. For example, if the engineer asks, how do I set up an Airflow DAG at LinkedIn, the agent will first decide, okay, I have a playbook for creating that specific task. And it will use that playbook to get the information. And then it follows that instruction and calls the relevant tools to get the job done. This was really powerful. Mainly because now anyone at LinkedIn can go ahead and create a playbook and check it into a repository and make it available for everyone else at LinkedIn. So as people started creating more playbooks, we wanted to establish some foundational principles for creating a playbook. The first one is a playbook should be self-contained, which means it should do a very specific task. For example, if it is for setting up an Airflow DAG, the instruction and the construct should be about one specific task. This helps the agents pick the right playbook for the right task. And the second most important one is to break a big playbook into multiple smaller playbooks and reference those smaller playbooks from a bigger playbook. This is a really powerful principle because it has two main advantages. The first one is reusability. If you have a small self-contained playbook, it can be referenced from multiple playbooks. And the other big advantage is progressive discovery of context, which means the agent only reads a smaller playbook when it needs to, instead of reading all of the playbooks at once. It can progressively go and read the playbooks as needed. So this is the same concept as skills as well. Playbooks are very similar to skills, but we developed this entire system around playbooks even before skills was a thing. And playbooks are a little bit more nuanced because it helps us seamlessly capture all of the organizational context and service via MCP without much of a setup. And another cool thing about playbooks is the self-improving loop. You have engineers creating these playbooks and checking into the repository, and one of the main problems with any knowledge base is it gets outdated. The biggest problem is how do you keep the context fresh. The great thing about agents is they can improvise. We encourage the agents to, whenever they use a particular playbook, at the end of the session, identify the learnings. Any outdated information or any discrepancy or any missing information. And we also encourage the agents to figure out how to improve the playbook and use that context to check it out, to update the playbooks, and create a PR. And once it gets approved, the playbook gets updated. This creates a really seamless self-learning loop. So what does the architecture of an MCP server look like? We have one local MCP server, and it is automatically installed on all of the LinkedIn laptops by default. So if you join LinkedIn and you get a laptop, it is pre-installed. And any updates to the MCP server or the playbooks or the tools automatically get updated every hour on all the laptops. And we have a concept of local playbooks and central playbooks. Central playbooks are the playbooks which are cross-cutting in nature. These playbooks apply to multiple repositories, not just one code repository. And then you have local playbooks where these are the playbooks which are very specific to your code repository, and you can just have them checked in with your repo. And only when the coding agents are working in your repo, those playbooks will be automatically picked up. So this helps us scale the local playbooks which are very specific to a repo without having to worry about changing the central repository. And also this is one MCP server which is serving all of the playbooks and tools. So this helps us do a lot of central things like seamless authentication, telemetry, and that we can use for learning to make the whole ecosystem better. You may be wondering how many tools and playbooks it can support. This is a common problem with MCP. We cannot scale it beyond 30 or 40 tools without degrading the context or degrading the performance of the system. So what we do is instead of surfacing all of these playbooks and tools through MCP, we replace them with three meta tools. The first one is search. The agent first uses this tool to search for the relevant tools and playbooks using keywords and tags. We also control the system instructions. Every coding agent is pre-configured with system instruction on how to use these tools and how to use the search really efficiently. And once it finds the right set of tool or playbook, it can then get more details about that particular tool using get schema and then execute that tool or playbook. So this has allowed us to scale to thousands of tools and playbooks. This is the growth chart. Now we have over 8,000 users daily using the system, using tools and playbooks. We have over 1,300 tools and over 600 playbooks. And it's not just engineering. It is not just engineers, but also product managers, designers, TPMs. Across different functions, they are using the tools and bringing their playbooks to automate their workflows. I'll leave you with these key takeaways based on our learning. The first one is the system was successful because we thought about quality and reliability from day one. Even before creating an MCP server, we thought, okay, our fundamental principle should be how do we ensure not just productivity, but how do we ensure the quality and reliability of the system so that it doesn't degrade as we move fast. And the second one was build the right infrastructure for agents. In a large enterprise like LinkedIn, it's not just enough to give all of the engineers all the latest and greatest tools and models. They are not very effective if you don't build the right infrastructure for the agents to operate within your enterprise. That's my time. Thank you for attending and feel free to connect with me on LinkedIn. coworkers with deep understanding of LinkedIn's internal systems and code to help the teams be really productive. And this is possible because of a system that we built called as contextual agent playbooks and tools at LinkedIn. And today I'm going to talk about why we built the system, how we built it, and what are our learnings from the success. To understand why we built the system, we have to go back to the early days of coding agents, right? So just like any other company, even at LinkedIn, we wanted to use the coding agents to be for our engineers and everyone to be really productive with the AI. So we started giving these coding agents to all of the engineers. And the problem was the coding agents doesn't really, or the wide coding doesn't really work in a large enterprise like LinkedIn. So the biggest problem is the coding agent or the LLMs are trained on open source repos, right? So they don't have the context of how we are mature code bases at LinkedIn or our internal frameworks, our internal systems. So what used to happen was the engineers used to do wide code or try the agentic coding, but because the agents lacked context, they used to hallucinate and get stuck in between, or even more dangerous, they used to make up things which is not correct. So the engineers had to prompt these agents manually to do the right thing, which used to take more time than the manual coding itself. So a lot of people, a lot of engineers went back to manual coding. So coding agents was not effective. To understand the problem, to get more perspective, so if you look at the LinkedIn stack, we have over 1,000 repos, which make up thousands of microservices and apps. And we have a lot of all of these apps and services are built on a lot of internal frameworks and libraries. And we also have a lot of custom built infra. For example, we have our own databases. We have our own experimentation and tracking platform. We have our own configuration management system, which is purely internal to LinkedIn, and coding agents doesn't have any idea about them. And engineers go through a week-long boot camp whenever a new engineer joins, just to get familiar with these systems. So we looked at this problem, and we asked ourselves the question, how can we make any coding agent like Cursor or Cloud Code or GitHub Copilot understand our LinkedIn's internal systems so well that they can ship the code that our engineers can trust? By trust, I mean the code should be correct, and also the quality of the code should be as good as it is written by an actual engineer. So that is the bar we set up, and wanted to figure out how do we get there. So in early 2025, last year, Anthropic released MCP, and it quickly became the industry standard for building tools to the agents. We leveraged that, and pretty early on, we built our own internal MCP, and the first tool that we built was CodeSearch. So we have a pretty sophisticated code search system at LinkedIn, where engineers can go and search for code. It will ingest all of, search for any code across thousands of repos, using keywords and custom filters and rejects, etc. So we made that available to the coding agents via MCP. This was a really powerful unlock, because now you don't have to manually figure out how to do better search. The agent, you ask a question, hey, how do I set up a particular thing, and the agent can use the code search tools to figure out the right examples of how we do things at LinkedIn, and use that to give you an answer, and also implement it based on its findings. This was really powerful. So we added more tools. We added docs, Jira, Slack, even connected to all of our data platforms, and even feature flags. So every tool that we added to our internal MCP, it created more value. It's almost like a compounding effect, because now an engineer can bring in the PRDs, product requirement documents, and design docs, and also their Jira tasks, which has different contexts, and use all this to give to the coding agent to automate or help with their coding. But there was a problem. So you connect all these tools, but it's not enough. So even with a slightly complex workflow, the agents used to not do really well. For example, if you give a context, with the tools, the agent was able to answer basic questions and find code examples, but it cannot do a complete job reliably end-to-end. The main problem was to do a specific job end-to-end. It needs to have a lot of tribal knowledge. So all of, for example, how to fix a particular error, or how to configure, how do you debug a particular error log. So all of this knowledge, even though you have access to the tools, it is scattered across a lot of different surfaces. For example, docs, wikis, and Slack conversations, et cetera. And most of the times, I may have experienced that docs and wikis might be outdated, written, and there might be duplicate docs, right? So the problem is the agents, even though they have access to the tools, they used to get lost. The second problem was context overload. As agents use more and more tools, their context gets overloaded, which means every tool output, it takes up space in the context, which will eventually cause the agent to compact its, while it is working, compact its context, which causes it to lose some of the information. Then it has to do all over again. And the third problem was, even if the agent was able to figure out all these details, it doesn't have a way to retain this information. It doesn't have a durable memory. So every time an engineer asks the agent to do a certain task, they have to start from scratch. So how do we solve this problem? So we give these instructions right away, right? So we built a system, we invented a system in early 2025 called as playbooks, where we not only provide the tools to the agents via MCP, we also allow the agents to access these instructions and prompts via MCP. We call it playbooks. And playbook, it just appears just like any other regular tool. And they have names and description on what it does. And the agent can decide to invoke that playbook just like any other regular tool. And when the playbook is invoked, the instructions and the context within that playbook are returned as the tool output to the coding agent. So that way, the agents have both tools as well as instructions on how to use tools to set up or perform a task. For example, if the engineer goes and asks, like, how do I set up an Airflow DAG at LinkedIn, the agent will first decide, okay, so I have a playbook for creating that specific task. And it will use that first effect, uses that playbook to get the information. And then it calls the necessary, follows that instructions and calls the relevant tools to get the job done. This was really powerful. Mainly because now anyone at LinkedIn can go ahead and create a, set up a playbook and check it into a repository and make it available for everyone else at LinkedIn. So as people started creating more playbooks, so we wanted, so this is one of two foundational principles we want everyone to follow when creating a playbook. The first one is, a playbook should be self-contained, which means it should do a very specific task. Only, for example, if it is for setting up an Airflow DAG, it should be about, the instruction and the construct should be about one specific task. This helps the agents pick the right playbook for the right task. And the second most important one is to break a big playbook into multiple smaller playbooks. So this has, and reference those smaller playbooks from a bigger playbook. So this is a really powerful principle because just like, so it has two main advantages, right? So the first one is reusability. So if you have a small self-contained playbook, it can be used from multiple, reference from multiple playbooks. And if you, the another big advantage is progressive discovery of context, which means the agent, only when it needs to read a smaller playbook, instead of reading the entire, all of the playbooks at once, it can progressively go and read the playbooks as it wants. So this is the same concept as skills as well. So playbooks are very similar to skills, but we developed this entire system around playbooks, even before skills was a thing. And playbooks are a little bit more nuanced because it helps us seamlessly capture all of the organizational context and service via MCP without much of a setup. And another cool thing about this playbooks is the self-improving loop. So you have engineers creating these playbooks and checking into the repository, and one of the main problems with any knowledge base is it gets outdated. How do you, the biggest problem is how do you keep the context fresh, right? So great thing about agents is they can improvise. So we have, we encourage the agents to, whenever they use a particular playbook, at the end of the session, to identify the learnings. So any outdated information or any discrepancy or any missing information. And we also encourage the agents to figure out how to improve the playbook and use that context to check it, to update the playbooks, check out the repository and update the playbooks and create a PR. And that, once it gets approved, it gets, the playbooks gets updated. Right? This creates a really seamless fly. of self-learning loop. So what does the architecture of a MCP server looks like? So this particular system, we have one local MCP server, and it is automatically installed on all of the LinkedIn laptops by default. So if you join LinkedIn and you get a laptop, it is pre-installed. And any updates to the MCP server or the playbooks of the tools, it automatically gets updated every one R on all the laptops. And we have a concept of two local playbooks and central playbooks, which means, so central playbooks are the playbooks which are cross-cutting in nature. So you have, these playbooks apply for multiple repositories, not just one code repository. And then you have local playbooks where these are the playbooks which are very specific to your code repository, and you can just have them checked in with your repo. with your repo, and only when the coding agents are working in your repo, those playbooks will be automatically picked up. So this helps us scale the local playbooks which are very specific to repo without having to worry about changing the central repository. And also this is one MCP server which is serving all of the playbooks and tools. So this helps us do a lot of central things like seamless authentication, telemetry, and that we can use for learning to make the whole ecosystem better. You may be wondering like how many tools and playbooks it can support, right? So this is a common problem with MCP. We cannot scale it beyond 30 or 40 tools without degrading the context or degrading the performance of the system. So what we do is instead of surfacing all of these playbooks and tools through MCP, we replace them with three meta tools. So the first one is search. The agent first uses this tool to search for the relevant tools and playbooks using keywords and tags. So we also control the system instructions. So every coding agent is pre-configured with system instruction on how to use these tools and how to use the search really efficiently. And once it finds the right set of tool or playbook, it can then get the more details about that particular tool using get schema and then execute that tool or playbook. So this has allowed us to scale to thousands of tools in playbook. So this is the growth chart. So now we have over 8,000 users daily using the system daily, using tools and playbooks. So we have over 1,300 tools and over 600 playbooks. And it's not just engineering, right? So it is not just engineers, but also product managers, designers, TPMs. So across different functions, they are using the tools and bringing their playbooks to automate their workflows. So I'll leave you with this takeaway, key takeaways based on our learning. The first one is the system was successful because we thought about quality and reliability from day one. So even before creating an MCP server, we thought, okay, our fundamental principle should be how do we ensure not just productivity, but how do we ensure the quality and also reliability of the system so that it doesn't degrade as we move fast? And the second one was build the right infrastructure for agents. In a large enterprise like LinkedIn, it's not just enough to give all of the engineers all the latest and greatest tools and models. They are not very effective if you don't build the right infrastructure for the agents to operate within your enterprise. Yeah, that's my time. Thank you for attending and feel free to connect with me on LinkedIn. Another thing about the limitations of the system !