SPEAKER_01
Hello everyone, can you all hear me well? All right, amazing. Thank you for joining the session. In this one we're going to talk about sovereignty. My name is Vige, I work as a senior developer relations engineer at Deepset. But I want to start with a question. So how many of you are familiar with the term sovereignty? Wow, okay, amazing. Maybe I don't need to do a lot of introduction for this one. And just a little bit of context here about Deepset. Deepset is the company behind the open source orchestration framework called Haystack. We also have our enterprise platform and with those products we are solving custom AI challenges for big organizations like Airbus, Bosch, Siemens, but also public sector organizations like European Commission, Federal Ministry of Research and Technology and Space in Germany and other ministries as well. So you can imagine sovereignty is a very big and important topic for us.
SPEAKER_01
So here is a policy definition of what sovereign AI is. Sovereign AI is the ability of an organization to design, deploy and operate AI systems on its own terms. But we are all engineers here, we are not policymakers, we are not lawyers. So if you turn that into a technical definition, it's having explicit control over data flow, model choice, infrastructure, observability and operations. And for us to understand it better, I want to split that into four pillars. So the first one is data sovereignty. So it's about where you store your data, where does it get processed. The second one is model sovereignty, who controls the running models, the origin of the training data. The third one is infrastructure sovereignty, so where does compute happen. And the last one is operational sovereignty. So is your application, is your system traceable, who can update it, who owns the incident response.
SPEAKER_01
So let's go into the depths of this for a minute. Data sovereignty, data is the most important asset that we have in an enterprise. And for us to have data sovereignty, data should be processed and stored within trusted jurisdictions to meet compliance requirements. What does it mean? So GDPR says that European citizen data should stay within Europe. But if you send that data to an embedding model, to an embedding API hosted in Virginia in the US, then you are already losing the control of your data. So that's against data sovereignty. And the other aspect is access permissions. So it's not exactly about how the data is stored and processed. But if there are users in your organization within your organization that have access to data that they are not supposed to see, there is also a breach of data sovereignty. The other one is infrastructure sovereignty. So where does compute happen? It means we have the AI application layer. So we have our data pipeline, ingestion, we have agents, we have the tools for the agents, and they all run somewhere. And where this application layer runs defines the infrastructure sovereignty of your system. And there's this whole spectrum from max control to max convenience. Maybe you run everything in an air gap environment, it gives you the EU AI Act safe. Maybe you run on a private VPC, this gives you GDPR safety. There is also sovereign cloud, this kind of depends on the provider that you're going with. And there's also at the very end there's SaaS. So with SaaS, you face the cloud act risk, meaning that if you are for example using a US headquartered company, although you store, you run all your data, all your application within Europe, the US government has the ability to get access to your running data. So that's against the sovereignty that you have over your system. The other pillar is model sovereignty. So who controls the model and the origin of the training data? So you should have the freedom to choose and switch models. But what it means is that if your system can only work with one specific model, you are tightly coupled with that model provider. If the API is down, then you lose access. If it increases the price, then you also have a cost issue. This is completely against the model sovereignty idea. And the other one is swappability without architectural changes. So maybe you're not tightly coupled with that model provider, maybe you can use other models as well. But if your system, if your code doesn't allow that immediately, then you are, although technically you are not tied to that provider, but you cannot just change the whole code base within one day. And the last one is training data origin. So this is a bit controversial, because we don't have a way to know where the model was trained, which data was used to train this model. But if a model provider is a European company, then they have a better advantage than other companies based in the US.
SPEAKER_01
The last pillar is operational sovereignty. So building the whole AI system is one aspect, but monitoring and maintaining this whole system by monitoring, evaluating, managing is another. Operational sovereignty is about monitoring how these systems behave in production, including model inputs and outputs. And in high-stake environments like HR or finance, it requires human in the loop. And managing versioning updates to models and the application layer in a controlled, auditable way also goes under this operational sovereignty. But the good news is sovereignty is a spectrum. So not everyone needs to be sovereign in all of these pillars. Of course, if you're in finance, in healthcare, in high-stake environments, in high-stake domains, you might need a fully air-gapped solution. But if you're an enterprise or startup working in a different domain, then maybe you don't need everything at all. You don't need to comply with every sovereignty pillar that I showed you. The important thing here is you need to know the level of control. So the level of vendor lock-in you have with your system.
SPEAKER_01
So you are here at the conference this week, but next week on Monday, you're going to go back to work and maybe your manager or CIO comes and says, okay, we need to make it so that we have this working system, but now we need to make it sovereign. And in this slide, I want to show you what you do first usually and what you break in this existing system. So probably intuitively, the first thing that you do is the model. And you replace the frontier API that you have with a self-hosted model. And what happens? Then you need to translate this whole API logic to this new model architecture, maybe you need to update the prompts, you need to evaluate the performance of the system from scratch. And you need to write a lot of code. The other one is you move private data into the required jurisdiction because you notice that we have some information hosted in the US. Now we need to take it somewhere in Europe. And you do that. But you find yourself managing multiple databases and instances, then search becomes a problem. So how do you handle search? Do you do classification like query classification first? Or do you send the request to both of those databases and get information from those? That's just one challenge that comes with this change. And the other one is you replace managed infrastructure with on-prem. And you immediately notice how much vendor lock-in you had in this area. So you start thinking about Kubernetes cluster management, how do I deal with all these things that all these cloud providers were handling for you. And there are also hardware limitations. Because now you
SPEAKER_01
Now we need to take them somewhere in Europe. And you do that. But you find yourself managing multiple databases and instances, then search becomes a problem. So how do you handle search? Do you do classification like query classification first? Or do you send the request to both of those databases and get information from those? That's just one challenge that comes with this change. And the other one is you replace managed infra with on prem. And you immediately notice how much vendor lock-in you had in this area. So you start thinking about Kubernetes cluster management, how do I deal with all these things that all these cloud providers were handling for you. And there are also hardware limitations. Because now you need to think about, okay, I have this application layer running CPU, but now I have my model running GPU. So I need to connect them in a way and this connection management, network management also becomes a problem. And the last one is you incorporate observability and tracing. If you didn't have observability until this point in your system, that's an important issue already. But if you now when you need to incorporate it, you notice that you have this fully black box thing. So you don't know actually what's happening in this AI application layer. So you need to understand it now because you need to log it somewhere so that your system is auditable. And there's also version control aspect. How do you do version control for your application layer for this whole system that you are managing? And I want to shamelessly plug Haystack. How do you do versioning of Haystack. Haystack has three main features that help with this. The first one is resource efficiency. So you can run complex AI applications on the hardware that you have and this hardware connection. The second one is explicit data flow. So every input and output in a Haystack application is typed and declared. So you can read the pipeline definition or the whole application in a regular way and know exactly what data was where and even in less deterministic architectures like agents, the whole data, these tools, tool calls and tool outputs are traceable. The third one is YAML. So the Haystack applications are serializable to YAML making them very easy to version. So you can just after creating your application with Haystack you can turn it into YAML, put it into your version control and then when you need to go back to the history you just need to go back to the commit and you can see the hash. And the last part is it's truly open source so there's no black box, no hidden assumptions and when you need to customize some code or extend one of the components you can easily do so because you actually understand what's happening under the hood. And here is a sovereign architecture that I want to show you today just as a high level overview. Imagine you are building this agent but it needs to be sovereign. So the first thing that you do probably is add some guardrails before and then because this guardrail needs to check if there's a prompt injection coming from the user input, coming through the user input and it needs to also check some specific regulatory checks maybe because this is a very specific agent that needs to be used in a certain way and you want to check if user has this intention. And if it's unsafe it just leaves the application layer immediately and if it's a safe request, safe input then it goes to the agent. And agent here is basically an LLM with a system prompt and lots of different tools and these tools can be API calls, maybe they are connected to your knowledge base so you are doing some sort of search, maybe you are using other agents if you are dealing with multi-agentic systems and there are also MCP servers of course that are connected to your agent and agent creates an input, does the work for you and there is the last guardrail doing compliance checks because you also don't want to leak sensitive information to your user and then there becomes an output. And of course, how you design a system as a sovereign system is a challenge and the duty of Haystack here is to make sure that everything is swappable, traceable and without vendor lock-in so you know you can work with the right level of sovereignty that you need. And here are some of the tools that I picked that you can work with Haystack. So maybe you decide you need different models based on the task, maybe you decide that for public data you can use some proprietary models from those providers so you don't worry about those but for some of the tasks, for guardrails, maybe for your knowledge base, for your LLM you think that you need local models running on self-hosted local models from different providers like Mistral, Google, Nvidia, and Gina. And then of course there's the traceability aspect and as I said in Haystack everything, every input coming to a component and every output but as well as the traces in the agent is easily visible so you can just connect those to your LLM observability tool and you can since there is OpenTelemetry integration you can also implement your own observability and as the last part there's the storage. I wanted to pick some of the providers who can give you cloud and also the open source version of them so you can easily host these ones locally on prem. And here is the guardrail code, so basically how you connect a guardrail or how you define a guardrail is quite simple with Haystack. You just start with a model provider, for example in this one it's a Mistral chat generator. You give the name of the model that you want to use and by connecting it to your LLM message router it helps you do the classification. So it checks the input if it's safe it goes to the safe route and if it's unsafe it goes to the unsafe route. And then you start adding tools so you connect your MCP server that you host locally with MCP toolset and you give the names of the tools that you want to pick from that MCP server because probably you have lots of tools running on that MCP server you don't want to get all of them, you don't want to have access to all of them with just one agent and you pick for example knowledge base search and generating PDF report and of course you can add different tools. You can start defining tools from a Python function or you can connect other components in Haystack like an agent component into a tool or maybe there is some functionality that you define like data ingestion or RAG pipeline and you can also convert those workflows into tools as well. And after defining all these tools you put them in a searchable toolset so this gives you dynamic tool search with BM25 because you probably now have more than hundreds of tools for your agent but you don't want to fill in the whole context with just tool definitions. Then there's this agent component so basically you define a system prompt you say like you're a sovereign agent with access to multiple tools and you define the intention of this agent and then you put a brain to this agent basically a chat generator.
SPEAKER_01
In Haystack, you can convert an agent component into a tool, or you can define functionality like data ingestion or RAG pipeline and convert those workflows into tools as well. After defining all these tools, you put them in a searchable toolset. This gives you dynamic tool search with BM25 because you probably now have more than hundreds of tools for your agent, but you don't want to fill in the whole context with just tool definitions.
SPEAKER_01
Then there's the agent component. You define a system prompt, saying you're a sovereign agent with access to multiple tools, and you define the intention of this agent. Then you put a brain to this agent—a chat generator. If you are running this model locally on-premises, you create your own custom component connecting to your internal company URL and getting the model inference from that. But of course, if you have an OpenAI API-compatible endpoint, you can already use an existing component in Haystack. Then you give the tools and define confirmation strategies. You incorporate human in the loop, saying that if the user wants to submit a request, the agent should always ask for human approval. But if the agent wants to use the list payment request tool, ask for permission first from the human, and then you can use this tool as much as you want in this whole cycle.
SPEAKER_01
Then you bring it all together by defining your pipeline. You first add the component as a tracer that connects to your LLM observability. Then you put the input guardrail, connect your agent, the output guardrail, and you can run this agent saying "pull the outstanding payment request for Q3 and generate a PDF for me." By using the tools that this agent has, it creates the PDF for you and saves it under this directory under this name.
SPEAKER_01
So at the end of the presentation, I want to show you a checklist. If you want to check if your system is sovereign, you can look at these questions: Can you swap models without changing the application logic? Do you have reproducible run logs stored in a compliant way? And can your team respond to an incident without calling a vendor like one of those hyperscalers? Thank you for joining this session. You can get the presentations and ask questions to me by filling in the form, and you can also find me on social media if you want to connect there. I'm happy to talk with you about agents, Haystack, and especially in a sovereign setting. Thank you.
SPEAKER_01
organization to design, deploy and operate AI systems on its own terms. But we are all engineers here, we are not policymakers, we are not lawyers. So if you turn that into a technical definition, it's basically having explicit control over data flow, model choice, infrastructure, observability and operations.
SPEAKER_01
And for us to understand it better, I want to split that into four pillars. So the first one is data sovereignty. So it's about where you store your data, where does it get processed. The second one is model sovereignty, who controls the running models, the origin of the training data. The third one is infrastructure sovereignty, so where does compute happen. And the last one is operational sovereignty. So is your application, is your system traceable, who can update it, who owns the incident response. So let's go into the depths of this for a minute. Data sovereignty, data is the most important
SPEAKER_01
assets that we have in an enterprise. And for us to have data sovereignty, data should be processed and stored within trusted jurisdictions to meet compliance requirements. What does it mean? So basically GDPR says that your the European citizen data should stay within Europe. But if you send that data to an embedding model, to an embedding API hosted in Virginia in the US, then you are already losing the control of your data. So that's against data sovereignty. And the other aspect is access permissions. So it's not exactly about how the data is stored and processed. But if there are users in your organization, within your
SPEAKER_01
organization, that are that has access to data that are not that they are not supposed to see, there is also a breach of data sovereignty. The other one is infrastructure sovereignty. So where does compute happen? It means like we have the AI application layer. So we have our right pipeline, ingestion, we have agents, it is the tools for the agents, and they all run somewhere. And where this application layer runs, defines the infrastructure sovereignty of your system. And this and there's this whole spectrum from max control to max convenience. Maybe you run everything in an air gap environment, it gives you the EU AI act safe.
SPEAKER_01
Maybe you run on a private VPC, this gives you GDPR safety. There is also sovereign cloud, this kind of depends on the provider that you're going with. And there's also at the very end there's SaaS. So with SaaS, you kind of face the cloud act risk, meaning that if you are for example using a US headquartered company, although you store, you run all your data, all your application within Europe, you have the ability to get access to your running data. So that's against the sovereignty that you have over your system. The other pillar is modern sovereignty. So who controls the model and the
SPEAKER_01
origin of the training data? So you should have the freedom to choose and switch models. But what it means that if your system can only work with one specific model, you are tightly coupled with that model provider. If the API is down, then you lose access. If it increases the price, then you also have a cost issue. This is completely against the model sovereignty idea. And the other one is swappability without architectural changes. So maybe you're not tightly coupled with that model provider, maybe you can use other models as well. But if your system, if your code doesn't allow that immediately, then you are,
SPEAKER_01
although technically you are not tied to that provider, but you cannot just change the whole code base within one day. And the last one is training data origin. So this is a bit controversial, because we don't have a way to know like where the model was trained, which data was used to train this model. But if a model provider is a European company, then they have a better advantage than other companies based in the US. The last pillar is operational sovereignty. So like building the whole AI system is one aspect, but monitoring is basically maintaining this whole system by monitoring, evaluating, managing is also
SPEAKER_01
another. Operational sovereignty is about monitoring how these systems behave in production, including model inputs and outputs. And in high-stake environments like HR or finance, it requires human in the loop. And managing versioning updates to models and the application layer in a controlled, auditable way also goes under this operational sovereignty. But the good news is sovereignty is a spectrum. So not everyone needs like needs to be sovereign in all of these pillars. Of course, like if you're in finance, in healthcare, in high-stake environments, in high-stake domains, you might need a fully air-gapped solution. But if you're an
SPEAKER_01
enterprise or startup working in a different domain, then maybe you don't need everything at all. You don't need to comply with every sovereignty pillar that I showed you. The important thing here is you need to know the level of control. So the the level of vendor lock-in you have with your system. So you are here in the conference this week, but next week on Monday, you're gonna go back to work and maybe your manager or CIO comes and says like, okay, now we need to make like we have this working system, but now we need to make it sovereign. And in this slide, I want to show you what you do first usually and what you break in
SPEAKER_01
this existing system. So probably intuitively, the first thing that you do is the model. And you replace the frontier API that you have with a self-assisted model. And what happens? Then you need to translate this whole API logic to this new model architecture, maybe you need to update the prompts, basically, you need to evaluate the performance of the system from scratch. And you need to write a lot of code. The other one is you move private data into the required jurisdiction because you notice that okay, we have some information hosted in in the US. Now we need to take them somewhere in Europe. And you do that. But you find yourself managing multiple
SPEAKER_01
databases and instances, then search becomes a problem. So how do you handle search? Do you do classification like query classification first? Or do you send the request to both of those databases and get information from those? That's just one challenge that comes with this change. And the other one is you replace manage infra with on prem. And you immediately notice how much you had vendor lock in in this area. So you start thinking about Kubernetes cluster management, how do I deal with all these things that all these cloud providers were handling for you. And there are also hardware limitations. Because now you
SPEAKER_01
need to think about, okay, I have this application layer running CPU, but now I have my model running GPU. So I need to connect them in a way and this connection management network management also becomes a problem. And the last one is you incorporate observability and tracing. I mean, if you if you didn't have observability until this point in your system, that's an important issue already. But if you now when you need to incorporate it, you notice that you have this fully black box thing. So you don't know actually what's happening in this AI application layer. So you need to understand now because you need to log it somewhere
SPEAKER_01
so that your system is auditable. And there's also version control aspect. How do you do version control for your application layer for this whole system that you are managing? And I want to hear shamelessly plug Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack. How do you do version of Haystack.
SPEAKER_01
on the hardware that you have and this hardware connection. The second one is explicit data flow. So every input and output in a haystack application is typed and declared. So you can read the pipeline definition or the whole application in a regular way and know exactly what data was where and even in less deterministic architectures like agents and like agents the whole data this tools, tool calls and tool outputs are traceable. The third one is YAML. So the haystack applications are serializable to YAML making them very easy to version. So you can just after creating your application with
SPEAKER_01
haystack you can turn it into YAML, put it into your version control and then when you need to go back to the history you just need to go back to the the commit and you can see the hash. And the last part is it's truly open source so there's no black box, no hidden assumptions and when you need to customize some code or extend one of the components you can easily do so because you actually understand what's happening under the hood. And here is one of, here's a sovereign architecture that I want to show you today just as like a high level overview. Imagine you are building this agent but
SPEAKER_01
it needs to be some sort of sovereign so first thing that you do probably you add some guard deals before and then because this guardrail needs to check if there's like a prompt injection coming from the user import, coming through the user input and it needs to also check some specific regulatory checks maybe because this is like a very specific agent that needs to be used in a certain way and you want to check if user has this intention. And if it's unsafe it just leaves the application layer immediately and if it's a safe request, safe input then it goes to the agent. And agent here is basically an LLM
SPEAKER_01
with a system prompt and lots of different tools and these tools can be API calls, maybe they are connected to your knowledge base so you are doing some sort of search, maybe you are using other agents if you are dealing with a multi-agentic systems and there are also MCP servers of course that are connected to your agent and agent creates an input so does the work for you and there is the last guardrail doing a compliance checks because you also don't want to leak sensitive information to your user and then there becomes an output. And but of course like how you design a system as a sovereign system is a challenge
SPEAKER_01
and the the duty of Haystack here is to make sure that everything is swappable, traceable and without vendor lock in so you know you can work with the right level of sovereignty that you need. And here are some of the tools that I picked from that I picked that you can work with Haystack. So maybe you decide you need like different models based on the task, maybe you decide that for public data you can steal some proprietary models from those providers so you don't worry about those but for some of the tasks for guardrails maybe for your knowledge base for for your LLM you think that you need local models running
SPEAKER_01
on on like self-assist local models from other different providers like Mistral, Google, Nvidia, and Gina. And then of course there's traceability aspect and as I said in Haystack everything like every input coming to a component and every output but as well as the traces in the agent is easily visible so you can just connect those suspends to your LM observability tool and you can since there is open telemetry integration you can also implement your own observability and as the last part there's the storage I wanted to pick some of the providers who can give you cloud and also the open source version of them so you can easily host these ones locally on your prem.
SPEAKER_01
And here is the guardrail code so basically how you connect a guardrail or how you define a guardrail is quite simple with Haystack you just start with a model provider for example in this one it's a media chat generator you give the name of the model that you want to use and by connecting it to your LLM message router it helps you do the classification. So it checks the input if it's safe it goes to the safe route and if it's unsafe it goes to the unsafe route. And then you add you start adding tools so you connect your MCP server that you host locally with with MCP toolset and you give the names of the tools that you want to pick from that MCP server because probably
SPEAKER_01
you have lots of tools running on that on that MCP server you don't want to get all of them you don't want to have access to all of them with just one agent and you pick for example knowledge-based search and generating PDF report one and but of course like you can add the different tools you can add just like you can start defining tools from a Python function or you can connect other components in Haystack like agent component into a tool or maybe there is some functionality that you define like data ingestion or rack pipeline and you can also convert those workflows into tools as well. And by after defining all these tools you put them in a searchable toolset
SPEAKER_01
so this gives you a dynamic tool search with BM25 because you probably now have more than hundreds of tools for your agent but you don't want to fill in the whole context with just tool definitions. Then there's this agent component so basically you define a system prompt you say like you're a sovereign agent with access to multiple tools and you define the intention of this agent and then you put a brain to this agent basically a chat generator. If you are running this model locally on-prem maybe you create your own custom component saying now this is my on-prem chat generator connected to this internal company URL and getting the model inference from that.
SPEAKER_01
But of course if you have an open AI API compatible endpoint you can already use an existing component in Haystack and then you give the tools and then you define confirmation strategies. So basically you incorporate human in the loop saying that so if the user wants to submit a request the agent should always ask for human approval but if the agent wants to use the list payment request tool maybe ask for permission first from the human and then you can use this tool as much as possible as much as you want in this whole cycle. Then you bring it all together. So you start defining your pipeline.
SPEAKER_01
You first add the component as a tracer that connects to your LLM observability. Then you put the input guardrail. You connect your agent the output guardrail and you can just run this agent saying that pull the outstanding payment request for Q3 and generates a PDF for me and by using the tools that this agent has it creates the PDF for you and saves it under this directory under this name. So with that I want to give you just like when we are coming to the end of the presentation I want to show you a checklist. So if you want to check if your system is sovereign you can you can take a look at these questions.
SPEAKER_01
So you can think about can you swap models without changing the application logic? Do you have reproducible run logs stored in a compliant way? And can your team respond to an incident without calling a vendor like one of those hyperscalers?
SPEAKER_01
Thank you for joining this session and you can get the presentations and ask the questions to me by filling in the and the form and you can also find me on social media if you want to connect there. I'm happy to talk about you about agents, haystack and especially in a sovereign setting. Thank you.
SPEAKER_01
Haystack. Haystack.