SPEAKER_00
Everyone ready? Awesome. Welcome to AI Engineer. Thanks for joining my session. We are going to be talking about agentic search for context engineering today. My name is Leonie. I work at Elastic, the company behind Elasticsearch. And usually I like to talk about retrieval on Twitter. Today I'm super excited to be doing this in person. A little bit of housekeeping. If you want to access the slides and the code we will be looking at, you can scan the QR code. So let's start with why I'm excited about search and retrieval and hopefully why you are excited about it by the end of this workshop as well. Who here has built an agent or some form of it before? Awesome. Then you're probably not intimidated by this image. You've probably seen some alternative to this one before. This is essentially what context engineering looks like. So context engineering, when you talk about it, is the art or engineering techniques about how from all of the possible context sources we have, how do we actually decide what goes into the context window so our LLMs can generate the best responses. Often when we talk about this, we talk about context curation and we mean this little arrow from context sources to context window. But we're not giving this little arrow right there enough credit in my opinion because what's powering this is the search tool or search tools that actually decide what goes from context sources to the context window. And today we're going to be looking at the different search tools we have. So this is my personal hot take. I like to say that context engineering is about 80% agentic search because it's this little box right here. All right. Let's start with a little bit of history. And when I say history, I mean the last three years. RAG. When we started with RAG, the original idea was that we had a fixed retrieval pipeline. So the user message would usually more or less verbatim be used as a search query to be used usually as a vector search query to pull some data or chunks from a database. And together with retrieved context, the user message would go into the context window and then it would be fetched to the LLM. Nice. This has clearly many limitations. So since this is a fixed pipeline, whether or not you actually need any context, you're still retrieving additional information. And in the worst case, that can actually confuse your LLM, right? On the other hand, if you're only retrieving once, let's say you need some multi-hop retrieval, you're asking your LLM something more complex. Then if you're only retrieving once, maybe the retrieved chunks reveal some information about another search query you need, then you would actually might want to have a second round of search, right? So that's why we then moved on to agentic RAG. So we replaced the fixed pipeline with now a search tool. So now the agent can decide by itself whether or not to call the search tool and retrieve some information. So we don't have the problem anymore of do I actually need any information? And when I actually retrieve information, do I need to retrieve more? Do I actually have to retrieve something or rewrite the search query? Yeah. So we still only have one context source in this case, one database. Now when we look at context engineering, the context lies in many different places, right? So we have context sources in local files. So when you think about your coding agent, you probably have your coding project in or code files laying around in your local file system. Maybe you're using some kind of working memory like a scratch pad. So you're planning with your agent what you want to do, then you probably have something like a plan.md file. So when you have agent skills, you're using a local folder somewhere. We still have databases because many enterprises have their data stored in databases. We have the web as another context source. And I know this is a super controversial image here because I did not commit to having the long term memory in the local file system or the databases. I think currently a very big discussion. We can get into this in the Q&A if you like. But we also have long term memory as another context source, right? So how do we actually retrieve context from these? Usually we have a set of context source native search tools. So for the local files, you usually have something like a search files tool for skills and you usually have a skill loading tool. When you think about databases, we have a little bit more custom tools. So something like a semantic search tool. Maybe you also have something more general purpose like a tool that lets you execute entire search queries against the database like SQL, for example. For web, you have web search tools and for memory, you have something like a dedicated memory tool. If that's not overwhelming enough, we now also have something called a shell tool. Langchain calls it shell tool. Anthropic calls it the bash tool. If you've experienced, if you've played around with open clouds called the exec tool. But what all of these tools do is they let your agent run commands in the terminal. And that actually makes them super versatile because now you can let your agent use CLIs to navigate and explore your local files. So you can just run ls and grep to find data in your local file system. Langchain calls it the shell tool. If your database has a custom CLI, you can actually let the agent also use the shell tool and interact with the database. You could also let your agent write an entire script from scratch, like connect to your database, run a search query. If your database is exposed via HTTPS, you can just run a curl command, interact with the database through the shell tool. Speaking of curl commands, you can also do web searches if you like. So this shell tool is super versatile, right? So the question and the topic of today is what search tool do we actually need? Do we only need a shell tool? Do I need all of these? And if you take home only one thing from today is that doing good search is incredibly difficult. And that's why we have many different techniques to do search, right? We have vector search, we have keyword search. Even in a vector search, we have dense embeddings, sparse embeddings, multi-vector embeddings. Then we have many different indexing techniques. So depending on what kind of search requirements and latency requirements you have, you will need to curate your own stack of search tools, right? So today we're going to be looking at a few of these. Unfortunately, we only have one hour, so I cannot show you all of them. Before we get into some code, I want to give you a few fundamentals of building good search tools. Because agentic search at the surface level seems very straightforward. The user makes a request. The agent calls the right tool with the right parameters. I see someone laughing. The retrieval tool gives you the tool response and then your agent responds to you with the correct answer. At Elastic, we help a lot of internal and external teams build agents based to interact with Elasticsearch data. And the reality is that this can break in many different ways. I'm just going to show you three today. So the first is the agent doesn't call any tool. So this means the agent decides I actually can answer this question based on my parametric knowledge. I don't need to use any context retrieval tool. And the other problem is that the agent calls the wrong tool. I was recently talking to a colleague of mine. I was asking him, what was the most challenging aspect of your project? And he was like, you won't believe it, but it was really difficult to get the agent to actually not call the web search tool,
SPEAKER_00
At Elastic, we help a lot of internal and external teams build agents based to interact with Elasticsearch data. And the reality is that this can break in many different ways. I'm just going to show you three today. So the first is the agent doesn't call any tool. This means the agent decides I actually can answer this question based on my parametric knowledge. I don't need to use any context retrieval tool. And the other problem is that the agent calls the wrong tool.
SPEAKER_00
I was recently talking to a colleague of mine. I was asking him, what was the most challenging aspect of your project? And he was like, you won't believe it, but it was really difficult to get the agent to actually not call the web search tool, but call the database search tool, right? And then depending on how complex your parameters are for your search tool, it can also be quite challenging to get your agent to generate the right search parameters, right? There's many more failure cases, but we're going to limit this to these three today.
SPEAKER_00
So I personally hate this slide because I feel everyone in this room probably knows that the tool description is the most important aspect. But anytime I see a tool description, it's the least effort, one sentence, and then you're wondering why your agent isn't calling the right tool.
SPEAKER_00
So arguably, this is a very long tool description. I'm not saying you have to write it like this. I'm just saying if you just start with a core purpose, if it works fine, great. But if you add more parameters or more tools and your agent is starting to struggle with calling the right tool, then maybe add some trigger condition. When should this tool be used? When should this tool not be used? Especially if you have multiple tools, adding something like relationships is super important. Like first call this agent skill before you actually call this tool or get some confirmation before you call this tool.
SPEAKER_00
If you have the perfect tool description and your agent still doesn't call the right tool, then reinforce it in the agent system prompt. That should actually help out in most cases. Then I want to quickly touch on some on the parameter complexity. If you have a search tool that's just very simple in the sense that something like get customer by ID, it should be fairly straightforward for the agent to generate an ID parameter given that it's a valid ID. Same for if you're doing a semantic search, right? Generating some valid string shouldn't cause any issues.
SPEAKER_00
But let's say if you want to have a semantic search tool and instead of just giving a topic, you now also want to give it some filter conditions. Maybe you want to define the top K. Then you start to have more parameters, right? This isn't very complex here, but the longer the list of parameters you have, the more difficult it's going to get for the agent to generate the right ones.
SPEAKER_00
And I think a very complex one for an agent is to, when you have something that's more general purpose, letting the agent execute entire search queries against the database. So here I have ESQL, which is the Elasticsearch query language. Could be SQL as well. Letting the agent write an entire SQL query from scratch can be quite challenging. Most are pretty good, but some aren't. So just keep in mind, the complexity of the parameter also is a failure mode and you might need to help the agent out with a few of these if they are more complex. Good. Let's look at some code. Okay. So, quick show of hands, who here has built some sort of agentic RAG, agentic search before?
SPEAKER_00
Okay. About half. That's good. So we're going to be looking at three things. I'm going to give you a quick recap or intro to the very vanilla agentic search demo. And then I'm going to show you how easy it is to break this usual demo. And then we're replacing the semantic search tool with something more general purpose. So we're letting the agent write an entire search query from scratch.
SPEAKER_00
And for these two examples, we will be using a local Elasticsearch cluster as a context source. And then for the third part, I'm switching gears and I'm going to be showing you how search over local file system works with the best tool. And then I'm also going to show you some limitations of the shell tool and how you can expand it with custom CLIs.
SPEAKER_00
All right. The example that we'll be doing today is I have the conference session data of this conference here. And let me start, show you this one. So just a quick recap. We have Elasticsearch database and we're going to be writing semantic search tool. And in the database, I have the conference session already chunked. You probably already know how to chunk and store data in a database. So we're skipping this part. This is not the important aspect. So what do we need for an agent and LLM?
SPEAKER_00
We're going to be using Langchain for this session just because it wraps a lot of the complexity. And I like that it lets us concentrate on the high level concepts. Also it has some nice built in features like the shell tool is built in and it has some code samples for skill loading tools, which we will be looking at later. Okay. Switching back. I'm using GPT 5.4 nano for this demo. Then we're defining a very simple system prompt. So the usual, you are a search agent tasked with answering questions. You have access to different context retrieval tools and before answering a question, decide whether or not you need to retrieve additional context.
SPEAKER_00
To help the agent a little bit, I have some information about how the data is structured in Elasticsearch. So here we have a text field. It's comprised of the title of each session and the description of each session. And the text field is what actually gets embedded as the vector embeddings for semantic search. And then I also have some metadata fields. So for example, the day, the time, the room, the speaker's name. Since the metadata is not embedded, I can only run filters over them, but not any semantic search just for your information.
SPEAKER_00
Okay. Now the interesting part, let's build a semantic search tool. So how that works in Langchain is I have to first define an embedding model. In this case, I'm using the new Nomic embeddings V5 model. This is used to embed the search queries at query time. And the embedding model I'm putting in together with Elasticsearch data to create a vector store. So since the metadata is not embedded, I can only run filters over them, but not any semantic search just for your information. Okay. Now the interesting part, let's build a semantic search tool. So how that works in Langchain is I have to first define an embedding model.
SPEAKER_00
In this case, I'm using the new genome embeddings V5 model. This is used to embed the search queries at query time. And the embedding model I'm putting in with Elasticsearch store together with Elastic data to create a vector store. And then I can create a search tool. So in this case, I have, all I have to do is I call the similarity search method.
SPEAKER_00
It takes in a search query. And in this case, I'm setting the limits or the top K to three. This is a little bit of foreshadowing because I'm limiting the capabilities of this tool to just returning three search results, right? And what's nice in Langchain as well is that when you use the tool decorator up here, it lets you convert any Python function into an agent tool. So by default, it takes the functions, Python functions name as the tool name.
SPEAKER_00
And the doc string down here is going to get converted into the tool description. You can see I'm breaking my own rule by having a very short tool description here. Why this works is because I only have one search tool here, right? So you will see I'm adding a few things later on, but it's not going to get very descriptive in this demo here. So now we can run a test and test it for a search query of regulatory constraints.
SPEAKER_00
And you can see it finds a talk by my friend, Birgit, on engineering AI systems under sovereignty constraints. And it also finds some more talks, one by Tejas and one by Pedro. So, all right, let's plug it in. So we're plugging in the LLM, the system prompt and the search tool. I'm leaving out memory. Obviously this would be another core component of an agent. In this case, I'm leaving it out to keep it concise.
SPEAKER_00
Now I can run a simple question like, which sessions discuss regulatory constraints in AI systems. And you can see the agent first calls my semantic search tool. It wrote a quite extensive search query in my opinion, but it works. It finds the right talk by Bilge. Then it decided that apparently wasn't enough. So it rewrote the search query, but got very similar search results back. So after that, it decided that it's now able to respond with the right talks. This is where most agentic search demos fail, but this is very brittle.
SPEAKER_00
Does anyone have an idea how we can break this? Yes, that's a good idea. Anything else? Asking it something that's not in the database would be something great. What about asking it something where semantic search actually falls short? Maybe something where we want to look for a keyword, a specific keyword. Also doing something like filtering because in this search tool, we don't have any filters implemented, right? So my choice of search query is which sessions should I visit to learn more about GEPA? I'm not even sure.
SPEAKER_00
I've heard people talk about GEPA. I'm not even sure if I'm pronouncing it correctly. Sorry. That's why I need to definitely attend this session. So what you can see is the agent now calls the search, semantic search tool. And this time it's looking for GEPA. So far so good. But now you can see it's actually returning a talk for DeepMind's Gemma models. I guess from a tokenization perspective, it could be similar to GEPA or GEPA. I don't know.
SPEAKER_00
Then it returns something on harness engineering. I'm not sure if that's necessarily related. And then a third one, clearly none of these are related to GEPA. Spoiler alert, I know there's a talk about GEPA or GEPA, again, I think it's right after this one. So we can see the search tool we just created, it's not very useful, or at least useful only for a very narrow scope of use cases, right? What if we let the agent now write an entire search query from scratch? Let me show you how we can do this. So we're now replacing the database tool that we had with an execute query tool.
SPEAKER_00
So we're letting the agent not only take in a topic, but this time we're giving it an entire search tool, an entire search query. And I'm going to show you, because this is quite difficult for an agent, we're also combining it with a skill loading tool. So doing the same thing, I'm setting up my LLM. You've probably noticed I'm switching to a little bit more powerful model here. So I'm switching from the GPT 5.4 nano to the mini because I am now anticipating that writing search queries is a little bit more difficult. So the nano is probably not powerful enough.
SPEAKER_00
I'm using the exact same system prompt as before.
SPEAKER_00
And now I'm creating a general purpose database query tool. Since I'm using Elasticsearch, I'm going to be using the Elasticsearch query language, which is a piped query language for filtering, transforming and analyzing data. It looks something like this. Maybe it reminds you of SQL. It's a little bit different. It has different capabilities. Not important for this session. But you can see when I connect to my client and then I use this query method from the SQL class and run this query. You can actually see that there is a session by Samuel which talks a lot about GPA. And see here is a match. Here is another match. So let's wrap this into a search tool.
SPEAKER_00
You can see this time I just use the query method again here.
SPEAKER_00
And the agent takes in the SQL query as a parameter. And I exchanged the tool description with something that's called execute an SQL query against the conference schedule index in Elasticsearch. Notice anything different about how I wrote this search tool versus the other one? This time I added a try except block here for error handling. Generally speaking, you should have error handling. You can actually see that there is a session by Samuel which talks a lot about GPA. And see here is a match. Here is another match. So let's wrap this into a search tool.
SPEAKER_00
You can see this time I just use the query method again here. And the agent takes in the SQL query as a parameter. And I exchanged the tool description with something that's called execute an SQL query against the conference schedule index in Elasticsearch. Notice anything different about how I wrote this search tool versus the other one? This time I added a try except block here for error handling. Generally speaking, you should have error handling. But since I'm anticipating that writing a good SQL query or a valid one is going to cause more problems for the tool, I don't want the agent to just fail and then the whole system to crash.
SPEAKER_00
So instead, I return the error response to the agent so it can self-correct and rewrite the query. Generally speaking, it's super important to have this, right? So the agent can self-correct. So when we test this here, this is not important. Then I'm plugging in the LLM, the system prompt and my new search tool into the agent again. And when I now ask it the exact same question as before, which session should I visit to learn more about GPA? So if you can see it calls the execute ESQL query tool and it generates something that looks like valid ESQL. I'm not expecting anyone to be very familiar with ESQL.
SPEAKER_00
So what's wrong with this is that ESQL doesn't use the percentage sign as a wildcard character. And in ESQL, you would use the asterisk. So in this case, it's actually looking for percentage sign, GPA percentage sign in the data as an exact match. So that's why it's actually returning zero search results. And this is when you're working with search tools. Also, it's super important to think about is returning zero search results actually a valid response, or is it a failure mode, right? So that's why I'm trying to do that. How could I overcome this?
SPEAKER_00
I could probably write a more descriptive tool description, give it a little bit more help on how to write better parameters. I could reinforce it in the system prompt, give it more instructions there. Or I could use an agent skill because you need more documentation than just a one-liner, right? So now I'm going to show you how to add an agent skill. So in this case, I'm going to be writing my own very short custom agent skill. Quick question, who has used and played with agent skills before? Okay, good amount. So I'm going to be writing a very short one here. There are official Elasticsearch agent skills available if you want to play around with it.
SPEAKER_00
In this case, I'm just using my own custom ones. So how that works is you have the skill name and then also skill description, which gets injected into the system prompt. So only the, if you write it in Markdown, I think the front matter, right? That gets injected into the system prompt. And then when you need it, more information on the agent skill is loaded into the context window, right? So it's called something like progressive disclosure, where you add more information about the skill as needed. So in this case, I have some minimal instructions, here's the basic structure of an ESQL query. ESQL uses double quotes for string literals.
SPEAKER_00
So it's just some very basic syntax rules. And I also added some more information about the wildcard pattern. So it's not making this mistake again. And then, as I mentioned, Langchain has some boilerplate code you can just copy and reuse for using agent skills. So I'm skipping over this. All you have to know is we have a tool, a skill loading tool, and it gets combined with something called a skill middleware. Skipping over this because this is not relevant for our session. And now, all I have to do is I updated the tool description of my general purpose search tool. So this is the exact same tool that I had before, except this time I'm adding some relationship to it.
SPEAKER_00
I'm saying always use the Elasticsearch ESQL skill to generate the ESQL query before using this tool. Because otherwise, if the agent then still uses this tool first without using the agent skill, that would be a shame, right? I'm doing the exact same thing. So I'm reinforcing this now in the system prompt. I'm saying the same thing to use the Elasticsearch agent skill first before calling the general purpose search tool. And now I'm plugging it into the agent again. So this time, LLM system prompt for the skill loading tool. I have the skill middleware and then my general purpose ESQL query tool.
SPEAKER_00
And now, when I ask the agent which session should I visit to learn more about GPA, you can see it first loads the skill. So it actually loads everything that's in the body part of the skill into my context window. Then it generates this time a very valid ESQL with the asterisks as my wildcard characters. And you can see it actually finds the right session. And now it tells me that at 10:40, so after this session, I should be going to this session to learn more about GPA and learn how to pronounce it correctly. Okay.
SPEAKER_00
What's also cool about this is now the agent can do a lot of things, right? It can also do aggregations.
SPEAKER_00
So if I ask it something like how many sessions are on April 8th, you can see again, it loads the Elasticsearch ESQL tool. And then it writes an ESQL query. It writes an ESQL query that's using a filter. So it's filtering for April 8th. And then it also does an aggregation, some counting.
SPEAKER_00
And tells me today there are 27 sessions. Okay. What's also cool about this is now the agent can do a lot of things, right? It can also do aggregations. So, if I ask it something like how many sessions are on April 8th, you can see again, it loads the Elasticsearch ESQL tool. And then, it writes an ESQL query. Whoops. No. It writes an ESQL query that's using a filter. So, it's filtering for April 8th. And then, it also does an aggregation, some counting. And tells me today there are 27 sessions. This is nice because if I just do a search, let's say I ask it to tell me which sessions are on April 8th, and it only runs a filtered search, right?
SPEAKER_00
So, just imagine it would give me a list of all 27 sessions that are today, and we let the agent count how many sessions there are. That would probably not be so good because we all know agents or LLMs are notoriously bad at counting things.
SPEAKER_00
And also, it would fill up your context window, right? So, by letting the agent do its own calculation, so letting outsourcing the calculation part into the search tool, it's actually quite an efficient way to do this, right? Any questions so far? Okay. Let's switch gears. This is a very prominent topic at the moment. Maybe you've heard the discussion about all an agent needs is a shell tool and a file system. So, I work at Elastic, but I don't discriminate. Let's look at file systems and how to do this.
SPEAKER_00
Because I think it's a very interesting topic in general. So, what I did here, I prepared the data this time in a local file system. So, I have a folder called session data. And in here, I have for each type of session, keynotes and workshops, I have another folder. And in there, there's per session one file.
SPEAKER_00
Looks something like this. So, with the title, some metadata and the description. And now, I'm going to show you how you can use the shell tool with this. So, I'm switching back to the GPT 5.4 nano, because LLMs are just generally good at navigating file systems, writing shell commands. So, GPT 5.4 nano is sufficient in this case. I define another system prompt. So, the first part of the system prompt is exactly the same as the one we had before. And what I'm replacing this time is instead of explaining how the data is structured in Elasticsearch, I'm explaining how the data is structured in my local file system. Okay. So, let's use the shell tool.
SPEAKER_00
I have to give you a disclaimer using the shell tool. Can be risky, since giving your agent access to a terminal can make it delete files or do other things you don't want it to do. So, always recommend it to use it in a sandbox environment. Also, in LangChain, it doesn't have any safeguards by default. So, please be careful when using this.
SPEAKER_00
But other than that, it's very easy to use. So, you can just use import the shell tool and instantiate it here. And here you can see how you would use it. So, it takes in the commands parameter. So, when I say echo hello world, you can see down here, it actually prints hello world into my terminal. And then, all you have to do, again, plug in the LLM, the system prompt, and the shell tool.
SPEAKER_00
And now, you can ask it this exact same thing we had earlier. So, are there any sessions about GEPA? And you can say, as you can see here, it's called the terminal here. But it's the, it's called, the agent calls the shell tool. And it actually writes a few commands. So, first, it's looking at the folder structure. And then, it runs some grep commands. So, it's looking for GEPA in the session data. And I think it's looking for the first 50 entries. So, you can see it saw the folder structure. And then, it also found the one session we were talking about earlier. So, it's looking for the first 50. And I only found one session.
SPEAKER_00
I decided that it should probably look at the entire session data. So, it finds the exact same session again. So, this time, it decides, okay, then I should probably look at the contents of this session. So, now, it reads the entire file content. And here, you can see the session information as a tool response. And then, at the end, the agent tells me which session I should visit. Okay. GREP works based on exact matches, right? And regex.
SPEAKER_00
And I just want to show you this because I think it's funny how surprisingly good agents are with bash because they can cheat at semantic search. I, let me show you this. So, when I ask it which sessions discuss handling regulatory constraints, this was our semantic search query from the beginning. You can see it again looks at the folder structure. And then, the first command, or the first search it does, it's looking for regulat. It's fair. It's looking for regulation, for regulatory. I guess that's a fair start. But then, it goes ahead and now it just chains a bunch of synonyms together. So, it's looking for compliance. It's looking for constraints.
SPEAKER_00
It's looking for GDPR. It's looking for governance.
SPEAKER_00
Yeah, I guess that's fair.
SPEAKER_00
I think it actually finds the sessions. So, it's okay, let me try a bunch of other synonyms. So, now it's looking again for regulat, compliance, GDPR, serenity. I think, yeah, the list goes on. It's just looking at a bunch of different synonyms. And it actually is successful with this. And it finds the session by Birgit and returns the session information. And then is able to respond correctly. I guess it works. Is that the most efficient way to do this? Probably not. I mean, just as an example. So, let's say you want to search for something like movies with animal superheroes or something. Do you really want your agent to search for a list of all the animals possible?
SPEAKER_00
Will you find all those superhero movies with animal superheroes? It's just looking at a bunch of different synonyms. And it actually is successful with this. And it finds the session by Birgit and returns the session information. And then is able to respond correctly. I guess it works. Is that the most efficient way to do this? Probably not. So, let's say you want to search for something like movies with animal superheroes or something. Do you really want to do your agent to search for a list of all the animals possible? Will you find all those superhero movies with animal superheroes? Probably not. So it works.
SPEAKER_00
Is it the best? I let you decide. I think there's one by Lama Index called Semtools. There's a really cool one by LightOn, which is called ColGrab, based on multi-vector embeddings. Also, there's one by our own Gina. That's called GinaGrab. So today I'm going to show you how easy it is to actually use this together with your agent. So all you have to do is go ahead and install the GinaGrab CLI. And then all you have to do is tell your agent that it now has access to this tool or to the CLI. So this is the exact same system prompt as I had earlier. So this is Gina, and how GinaGrab works, how it should use it. Here are some examples of how you would use GinaGrab.
SPEAKER_00
Just a disclaimer, GinaGrab has many different modes. You can use it for classification, you can use it for re-ranking. Today I'm just showing you how to use it for semantic search. And at the end I'm also explaining to the agent when it should use grep and when it should use genogreps just so it knows for exact matches you probably still want to use grep and for more semantic search or fuzzy queries use genogreps. And then plugging this in to my agent again and when I now run the exact same semantic search query we had earlier, so which sessions discuss handling regulatory constraints, you can see the same behavior so it calls the terminal tool.
SPEAKER_00
It first explores the folder structure and then it actually on the first try is able to correctly use genogreps so it's looking for regulatory constraints. And boom, it actually finds the session by bilge on the first try. Finds a few others because it's looking for 10. It says returned top K of 10. And then it's able to answer me correctly. Nice.
SPEAKER_00
All right, any questions so far? Good. Then I'm switching back. So we were looking at a bunch of different tools today. We saw how big the tool landscape is. I showed you a few of the search tools we have. Now some practical recommendations on when should you actually use what.
SPEAKER_00
So maybe let's start with this. If you're looking for just one silver bullet tool, that's probably not the right way to go. Again, if you think about it, doing good search is incredibly difficult. So ideally you want to have or curate the right set of search tools for your agent's search behaviors. And you want to have a combination of specialized tools and general purpose tools. [SPEAKER_00] So specialized tools are something that the agent can use out of the box, something with a very simple parameter, something where you're not where you don't need a very powerful LLM. You know the agent isn't going to make a lot of mistakes.
SPEAKER_00
The agent can just use this tool out of the box. So at Elastic we like to think about this, about having a low floor.
SPEAKER_00
So this is a concept from user experience where the agent can just use a tool, doesn't make many mistakes. It's also efficient, so it doesn't have to run your tool multiple times. You can think about this as the semantic search tool we had earlier. Maybe you need to look up customers by ID a lot of the time, then having a specialized tool for that exact operation would be helpful. But then you also want to give the agent a high ceiling. That means for unexpected queries, for complex questions, you want the agent to still be able to handle these questions, right? And not have these limited specialized tools and be like, I cannot solve this.
SPEAKER_00
So for this case, something like a shell tool or the very general purpose one we had earlier of a query execution tool would be very helpful. But the problem with the query execution tool or the shell tool you saw earlier is since it's so general purpose, the agent sometimes might need more iterations to actually get to the right answer, right? So this is my practical recommendation of having a balanced set of search tools of a low floor and high ceiling. This is all nice when you already know your agent's behavior. But if you don't know your agent's query behavior yet, then I would recommend to start with a general purpose tool. Then log your agent's behavior.
SPEAKER_00
Generally speaking, logging your agent's behavior is recommended.
SPEAKER_00
But if you notice maybe your agent is taking four or five tool calls per question. That's too many tool calls. Then you probably, that's probably an indicator that the tool your agent has is too difficult for it to use.
SPEAKER_00
Then definitely look at what the agent's actually trying to solve. Maybe scope out something more specialized in that case, right? Also if you notice specific query behaviors, this is what I personally did with my test OpenClaw. I was, it has the exec tool and I started logging its behavior and obviously I was playing around with databases. So after three days I was asking it what kind of interesting patterns do you see? And it was recommending me actually to implement some specific search tools to interact with the database because it was out of the box only using the exec tool. All right. Start with general purpose tools.
SPEAKER_00
If you don't know your user's behavior yet. Love what breaks and a purpose built interfaces. Yeah, that was a lot to take in. I'm sure you have lots of questions. So I'm opening it up for Q and A and otherwise on your way out, don't forget to grab yourself some stickers. And then thank you for joining my session. Thank you. Thank you. Thank you.
SPEAKER_00
So would you say the tool stack you need is also mostly dependent on the model you're willing to use? All right. Start with general purpose tools. If you don't know your user's behavior yet. Love what breaks and a purpose built interfaces. Yeah, that was a lot to take in. I'm sure you have lots of questions. So I'm opening it up for Q and A and otherwise on your way out, don't forget to grab yourself some stickers. And then thank you for joining my session. Thank you. Thank you. Thank you. So would you say the tool stack you need is also mostly dependent on the model you're willing to use?
SPEAKER_00
So if you're using a very good model, it might be fine using the shell tool or search tool, but a very small model and light agent might need more specialized tools? Yeah. Actually, we, I think in our internal testing, we noticed that a more powerful tool actually reduces the error rate of, for the parameters by, I don't know the numbers exactly, but it was a very big amount where it reduced the error rate. So having a stronger model definitely helps for the general purpose tools. But I think you cannot expect just because you have a very strong model that there's going to be no errors. If that makes sense. Yeah.
SPEAKER_00
Thanks for a nice talk.
SPEAKER_00
I have one question. Maybe it's a slightly off topic, but so now we are talking about agentic reg, but it comes with the drawback of having higher latency against typical reg. So would you recommend having like a second pathway for simple reg for fast answers and how would you guide the agent to actually choose the right one? Because I think it's hard to say which question should be answered by agentic reg or by simple reg. That's a good question.
SPEAKER_00
I don't think I have a good answer on the top of my head right now. I'm thinking it was maybe something related. I was asked recently if you have a reg system, should you replace it with agentic reg? And I guess this is going in the same direction of when do you actually need agentic reg, right? I'm not sure because I assume it needs some kind of agentic logic of switching between a specific number of these things. So I'm not sure. I'm sorry. In such cases, when you have the wild card in the GAPA example, why can't we just do you guys perform a hybrid tool that maybe search and replaces common wrong wild card symbols coming from SQL with the correct ones, for instance?
SPEAKER_00
I'm not sure if I understand your question correctly. I mean, there are cases in which the agent doesn't know how to write the correct query because maybe he thinks the placeholders coming from SQL apply to SQL, but why don't we perform a hybrid tool that mechanically search and replaces the wrong placeholder, the percentage symbol with the asterisk? Yeah. Actually, the example I showed you of using the agent skill wasn't necessarily the solution for it. You can also add just some very simple instructions on for ESQL, don't use the percentage sign as a wild card character. It actually works. I tried it when I was building the demo.
SPEAKER_00
But then when the agent now runs in the next issue, then you start adding the next piece of documentation. Then you can start writing the entire ESQL documentation from scratch into your system prompt. And yes, for the demo purposes, it would have worked, but it's probably not how you would do it necessarily when you're building something more robust, right? Because if you just add little band aids every time you run into an error, then what happens when you run into the next edge case? Does that make sense? Yeah. Hi. Thank you for a really wonderful presentation. I have a question.
SPEAKER_00
In the demo, we have worked through the agentic search with dbcurious tool and also another one with shell tool. Would you recommend in practical use we can also combine both tools and then we validate the result from each tool? And then we add the confidence of the result from the LM. Would you like recommend doing this in practical use? Yes. Yes. That's a great question.
SPEAKER_00
Also, I'm cheating in this demo, right? Because I'm only showing you one tool per demo. In reality, you would have something more like a bunch of different tools where you then have to decide which or the agent has to decide which tool to use. I think there was a very interesting blog post by Vercel, I believe. And they did an experiment. I think it's called, if you want to look it up, I think it's called testing is if bash is all you need is the title, I think, of the blog post. And they actually benchmark or tested an agent with a bash tool, an agent with just file search tools, I believe, and an agent with database tools.
SPEAKER_00
And in the end, they also had one agent with a bash tool and the database tool.
SPEAKER_00
And they noticed that was super interesting for a specific set of queries where you have analytical queries.
SPEAKER_00
This is a specific use case. [SPEAKER_06] Actually, the database tool was more effective. [SPEAKER_06] But on the other hand, the file search tool is very effective as you saw for just quickly finding things. [SPEAKER_06] But the very interesting aspect was the hybrid agent with the bash tool and the database tool was actually achieving the highest accuracy. [SPEAKER_06] Because at first, I believe it was first using the database tool and then verifying the results with the shell tool. [SPEAKER_06] And that led the agent to actually achieve better accuracy. [SPEAKER_06] So I think that was a very interesting way and behavior to see in agents.
SPEAKER_00
[SPEAKER_06] Thanks for sharing. One second question. So if we use the semantic search tool, I think in practice you probably would use some kind of threshold to cut the results to not get something if there's no answer. But in the agentic regime, would you then say, okay, let's put a conservative threshold such that we don't confuse our agent? Or would you say the agent is smart enough even if we retrieve results that are not really relevant? It will be good enough to notice that? Yeah. That's a great question actually. So in the examples you probably saw some where the agent was returning.
SPEAKER_00
I think in the last GinaGrab example, you see it's actually returning the top K results where only the first one is the actual relevant one. [SPEAKER_05] And I think because the agent does a little bit of reasoning over whether the search results are relevant to the search query, I think it's much better or they're much better today at weeding out what's not relevant. Or would you say the agent is smart enough even if we retrieve results that are not really relevant? It will be good enough to notice that? Yeah. That's a great question actually. So in the examples you probably saw some where the agent was returning.
SPEAKER_00
I think in the last GinaGrab example, you see it's actually returning the top K results where only the first one is the actual relevant one. [SPEAKER_05] And I think because the agent does a little bit of reasoning over whether the search results are relevant to the search query, I think it's much better or they're much better today at weeding out what's not relevant. [SPEAKER_05] But then you run into the risk if you have longer running conversations where these search results sit in your context window long term, which could have the problem of confusing your agent long term.
SPEAKER_00
[SPEAKER_05] So I think it depends on your use case of how your agent can handle irrelevant search results.
SPEAKER_00
[SPEAKER_05] But generally speaking, based on search results, it can filter out what's irrelevant. [SPEAKER_05] Thanks. [SPEAKER_05] Thank you for the talk. [SPEAKER_05] Amazing.
SPEAKER_00
[SPEAKER_05] Are you utilizing sub-agents for the search queries? Because if we let them decide whether it's relevant to use a question, it's done in the Regas framework for example for evaluation. Sub-agents would help a lot. Do we have any experience with them? Unfortunately not. I have not played around with sub-agents yet. I can only tell you that I know, for example, in Cloud Code. They're using sub-agents for doing specific search tasks. I think there was a blog post on how they're actually using a sub-agent to answer specific questions about Cloud Code because it's a niche question a user would ask. So in this case they outsourced the expertise to a sub-agent.
SPEAKER_00
So having a sub-agent for specific niche questions I think would be interesting but I don't have too much experience. Okay, thanks. [SPEAKER_03] Can I ask another question? [SPEAKER_03] Sure. [SPEAKER_03] I forget, sorry. I tried to catch up later. [SPEAKER_03] Thank you. [SPEAKER_03] Yeah, off topic, but you talked about skills and the big benefit of skills is just having the description in the system prompt and whenever needed, we load this full skill.
SPEAKER_00
[SPEAKER_03] Do you have any recommendation when and how to clear the system prompt again? So because we want to keep the context window small and for a long session, we might have up to 10 skills, full skills in the context. I'm not sure. Joe, do you have a better answer? So what we're doing is that we load the skills definitions within the prompt, but then we use the file store to upload things within the context. So when the context is building up, when you're loading 10 or 11 or 12 skills, we basically refer to, we replace the context and point to the next location. So we refer to where it is in the file store. Yeah, so we keep the context very small.
SPEAKER_00
I'm finished out there. But the way that we are doing it behind the scenes is that we're providing progressive disclosure of skills. So we're providing those skill names and descriptions, the location within the file store. And then from the file store, we're loading into the context window when we need that skill. And then we offload it once it's done, once the context window progresses ahead of time. So we have this on demand approach. And that's the same with our compaction of context. And that's what I would advise you to do, use the file store as much as you can and have those tools available.
SPEAKER_00
Being able to grab the file store for when you want to see previous tool results and then use it from that. [SPEAKER_07] Thanks. [SPEAKER_07] This is my colleague, Joe from Elastic as well. [SPEAKER_07] Awesome. Then you probably, that's probably an indicator that the tool your agent has is too difficult for it to use. Then definitely look at what the agent's actually trying to solve. Maybe scope out something more specialized in that case, right?
SPEAKER_00
Also if you notice specific query behaviors, this is what I personally did with my, with my test OpenClaw. I was, it has the exec tool and I started logging its behavior and obviously I was playing around with databases. So after three days I was asking it what kind of interesting patterns do you see? And it was recommending me actually to implement some specific search tools to interact with the database. Because it was out of the box only using the, the exec tool. All right. Start with general purpose tools. If you don't know your user's behavior yet. Love what breaks and a purpose built interfaces. Yeah, that's, that was a lot to take in.
SPEAKER_00
I'm sure you have lots of questions. So I'm opening it up for Q and A and otherwise on your way out, don't forget to grab yourself some stickers. And then thank you for joining my session.
SPEAKER_00
Thank you. Thank you. Thank you. So would you say the tool stack you need is also mostly dependent on the model you're willing to use? So if you're using a very good model, it might be fine using the shell tool or search tool, but a very small model and light agent might need more specialized tools or? Yeah. Actually, we, I think in our internal testing, we noticed that a more powerful tool actually reduces the error rate of, for the parameters by, I don't know the numbers exactly, but it was a very big amount where it reduced the error rate. So having a stronger model definitely helps for the general purpose tools.
SPEAKER_00
But I think you cannot expect just because you have a very strong model that there's going to be no, no errors. If that makes sense. Yeah.
SPEAKER_00
Thanks for a nice talk. I have one question. Maybe it's a slightly off topic, but so now we are talking about agentic reg, but it comes with the drawback of having higher latency against typical reg. So would you recommend of having like a second pathway for simple reg for fast answers and how would you like guide the agent to actually choose the right one? Because I think it's hard to say which question should be answered by agentic reg or by simple reg. That's a good question.
SPEAKER_00
I don't think I have a good answer on like on the top of my head right now. I'm thinking I was maybe something related. I was asked recently if you have a reg system, should you replace it with agentic reg? And I guess this is kind of going in the same direction of when do you actually need agentic reg, right?
SPEAKER_00
I'm not sure because I assume it's, again, it needs some kind of almost agentic logic of switching to a specific number of these things. switching between them. So I'm not sure. I'm sorry.
SPEAKER_00
In such cases, like when you have the wild card in the GAPA example, why can't we just do you guys perform a hybrid tool that maybe search and replaces common wrong wild card symbols coming from SQL with the correct ones, for instance? I'm not sure if I understand your question correctly. I mean, there are cases in which the agent doesn't know how to write the correct query because maybe he thinks the placeholders coming from SQL apply to SQL, but so why don't we perform a hybrid tool that
SPEAKER_00
the term mechanically search and replaces the wrong placeholder, the percentage symbol with the asterisk? Yeah. Actually, so the example I showed you of using the agent skill wasn't necessarily the the necessary solution for it. You can also add just some very simple instructions on for ESQL, don't use the percentage sign as a wild card character. It actually works. I tried it when I was building the demo. But then when the agent now runs in the next issue, then you start adding the next piece of documentation. Then you can kind of start writing the entire ESQL documentation from scratch into your system prompt.
SPEAKER_00
And yes, for the demo purposes, it would have worked, but it's probably not how you would do it necessarily when you're building something more robust, right? Because if you just add like little like band aids every time you run into an error, then what happens when you run into the next edge case? Does that make sense? Yeah.
SPEAKER_00
Hi. Thank you for a really wonderful presentation. I have a question. In the demo, we have worked through the agentic search with dbcurious tool and also another one with shell tool. Would you recommend in the practical use we can also kind of use combine both tool and then we validate the result from each of tool? And then we kind of add the confidence of the result from the LM. Would you like recommend doing this in a practical use? Yes. Yes. That's a great question. Also, again, I'm kind of cheating in this demo, right? Because I'm only showing you one tool per demo.
SPEAKER_00
In reality, you would have something more like a bunch of different tools where you then have to decide which or the agent has to decide which tool to use. I think there was a very interesting blog post by Vercel, I believe. And they did an experiment. I think it's called, if you want to look it up, I think it's called testing is if bash is all you need is the title, I think, of the blog post. And they actually kind of benchmark or tested an agent with a bash tool, an agent with just file search tools, I believe, and an agent with database tools. And in the end, they also had one agent with a bash tool and the database tool.
SPEAKER_00
And they noticed that was super interesting for a specific set of queries where you have analytical queries. This is a specific use case.
SPEAKER_06
Actually, the database tool was more effective. But on the other hand, the file search tool is very effective as you saw for just quickly finding things. But the very interesting aspect was the hybrid agent with the bash tool and the database tool was actually achieving the highest, like highest accuracy. Because at first, I believe it was first using the database tool and then verifying the results with the shell tool. And that led the agent to actually achieve better accuracy. So I think that was a very interesting way and behavior to see in agents. Thanks for sharing.
SPEAKER_00
One second question. So if we use the semantic search tool, I think in practice you probably would use some kind of threshold to cut the results to not get something if there's no answer. But in the agentic regime, would you then say, okay, let's put a conservative threshold such that we don't confuse our agent? Or would you say the agent is smart enough even if we retrieve results that are not really relevant? It will be good enough to notice that? Yeah. That's a great question actually. So in the examples you probably saw some where the agent was returning.
SPEAKER_00
I think in the last GinaGrab example, you see it's actually returning the top K results where only the first one is the actual relevant one.
SPEAKER_05
And I think because the agent does a little bit of reasoning over whether the search results are relevant to the search query, I think it's much better or they're much better today at kind of weeding out what's not relevant. But then you kind of run into the risk if you have longer running conversations that kind of these search results sit in your context window long term could have the problem of confusing your agent long term. So I think it kind of depends on your use case of how your agent can handle irrelevant search results. But generally speaking, based on search results, it can filter out what's irrelevant. Thanks.
SPEAKER_05
Thank you for the talk. Amazing. Are you utilizing sub-agents for the search queries?
SPEAKER_00
Because if we let them decide to, is it relevant to use a question like, it's done in Regas framework for example for evaluation. I mean, sub-agents would help a lot. Do we have any experience with them? Unfortunately not. I have not played around with sub-agents yet. I can only tell you that I know, for example, I believe in Cloud Code. They're using sub-agents for doing specific search tasks. I think there was a blog post on how they're actually using a sub-agent to answer specific questions about Cloud Code because it's kind of like a niche question a user would ask. So in this case they kind of outsourced the expertise to a sub-agent.
SPEAKER_00
So having a sub-agent for specific niche questions I think would be interesting but I don't have too much experience. Okay, thanks.
SPEAKER_03
Can I ask another question? Sure.
SPEAKER_03
I forget, sorry.
SPEAKER_00
I tried to catch up later.
SPEAKER_03
Thank you. Yeah, kindly off topic, but you talked about skills and the big benefit of skills is just have the description in the system prompt and whenever needed, we need to load this full skill. Do you have any recommendation when and how to clear the system prompt again? So because we want to keep the context window small and maybe for a long session, we might have up to 10 skills, full skills in the context. Yeah.
SPEAKER_00
I'm not sure. Joe, do you have a better answer? Like have an idea? So what we're doing while I'm seeing is that we load the skills, like definitions within the prompt, but then we use the file store to like upload like things within the context. So when the context is building up, when you're loading 10 or 11 or 12 skills, we basically refer to, we replace the context that next year. So we refer to where it is in the file store. Yeah, so we keep the context very small.
SPEAKER_00
I'm finished out there.
SPEAKER_00
But yeah, so like the way that we are doing it behind the scenes is that we're providing like that kind of progressive disclosure of skills. So we're providing those, the skill names and descriptions, the location within the file store. And then from the file store, we're loading into the context window when we need that skill. And then we offload it once it's, once the program, you know, the context window progresses ahead of time around that. So we have this kind of more on demand one around that. And that's the same with our like compaction, like of context.
SPEAKER_00
And that's what I would advise you to do, like some of the questions to try and use the file store as much as you can and have those tools as well. Like being able to grab the file store for when you want to see previous tool results and then use it from that.
SPEAKER_07
Thanks. This is my colleague, Joe from Elastic as well.
SPEAKER_07
Awesome.
SPEAKER_07
Awesome.
SPEAKER_00
Awesome. Awesome. Awesome.
SPEAKER_04
Awesome. Awesome. Awesome. Music
SPEAKER_00
Music Music Music Music Music Music
SPEAKER_04
Music Music Music
SPEAKER_00
Music Music