Structuring the Unstructured - Cedric Clyburn, Red Hat

Transcript

Hey, hey, welcome. My name is Cedric Laburn. I'm an open source engineer here at Red Hat, and I think we can all agree that context is the most important aspect to building an AI application or an agent, right? It's the reason that harnesses have become so popular in order to manage the LLM's context. But the thing is, no matter what model or agent that you're using, there is so much data that we're not able to use properly because it's in unstructured formats. I'm talking everything from PDFs to presentations to contracts and technical docs, even meeting notes, scan documents, diagrams, tables, images, and more. And sorry, I know that's a lot, but you understand what I mean, right? All this data needs to be transformed into something that an LLM can actually understand. And that's why by the end of this session, you'll understand how to extract structure between raw enterprise documents and use it to power better downstream AI systems like RAG and agents. So let's get started. Now, I think Jensen from NVIDIA made this point super clear at his keynote that unstructured data is becoming this new context layer for AI. And the reality, though, for many teams, and I know this personally working at Red Hat, that PDFs and data are spread across dozens of different systems. So we've got a lot to cover today. As you might know, a large majority of the world's data is unstructured. And so no matter what model you're using, if you're working with data like PDFs and unstructured types of formats, this is a bit tricky to work with. Because there are solutions out there, but they might be proprietary or require you to send your private data to someone else's server. And for not just text, how do we take documents and their graphs or tables and images to formats that LLMs can understand like Markdown or JSON? And I'm going to show you how in the session today, because we're going to be using an open source tool, part of the Linux foundation that is called Dockling, and learn about extraction, parsing, chunking, and much more. And I've got some live demos for you. So we're going to have some fun. And just in case you'd like it, we have the session slides here on the right, and a little overview of the specifics I'll be showing you in today's session. But without further ado, let's get started. So why is there a need for advanced document processing? As I briefly mentioned before, you might have a lot of technical documentation or meeting minutes or different types of documents and invoices that you need to use and maybe RAG or different type of applications where the context is provided to an LLM. So whether it's RAG or retrieval augmented generation to answer questions based on this data, or you're using this to fine tune a new specialized model, well, data is this key ingredient behind those applications. And it doesn't matter if you're using Nvidia acceleration or an open source or proprietary model, that data and the way you process it is the key determining factor in whether your answer is going to be correct or incorrect for the user or customer at the end of the day. And that's what's most important. And how important is it? Well, I have this viral tweet from earlier where 20 scientific papers now feature a new nonsensical term that doesn't exist because AI misinterpreted a very old article that was scanned and taken to a PDF, merging two different words from two different columns in this PDF. And because researchers are using these models in order to help them write, now we have different types of scientific papers that all feature this word and are even being cited by other people. And so that's how important it is to make sure that the data that we're processing is processed in a way that's accurate and not hallucinated and able to be used confidently in our applications that we're delivering to users and customers. So it's quite important. Now, if we were to use a tool like dockling that I can run on my own machine, you could see that these two words are quite far away from each other and shouldn't have been combined in the first place. But that's how we're going to learn about extracting this text here in a second. So if we were to try a simple PDF parser for a PDF like this, that includes a table here, it has an image, there's captions, and there's regular sections of text. Well, we might get an answer like this here on the right in Markdown. You know, this might be very fast and cheap to run, even on CPU. But the issue is, is that a lot of this text has been truncated, has been merged, and isn't decipherable even by me as a human. And if I sent this to a model, I don't think I could trust that the model could extract specifics from, say, for example, this table, because the table has been sort of just spit out linearly. And this information isn't fit for most use cases where I need to ask questions or have an agent do validation and extraction on this source data. So this isn't going to cut it, right? There's undesired page headers. We don't understand the table. And where's the content from the image, right? It's not even there. When we're using frontier models, this is not bad, right? But quite expensive. I'm sending this to a model that's maybe $30 per million output tokens. You can see how this can get quite expensive as I scale this up to dozens or hundreds or, in a lot of cases, thousands of PDFs that organizations have to work through to use an AI application. And the differences between maybe a 5.1 of a model that was depreciated and a 5.2 version of a model make it tricky to have structured output that's consistent every single time. And so while it might be good quality, and I can see that most of this looks accurate in the exported markdown, we might be susceptible to hallucinations because models are non-deterministic. And this is really tricky at scale. And so what is the middle ground? Well, that's where Dockling comes in. It's a fast and cheap and most importantly, local CLI and library that I can use to take various types of input sources and convert this to markdown, JSON, and a pydantic data type that I can use in my applications and that I can scale up if I have thousands of different types of formats that need to be used or translated to something like markdown. And so we rendered this as HTML, but you can see here we've got this export of the specific table that we had here previously, because as a PDF, this data type is spread out and it's proprietary. So it's hard to extract this type of data from the source PDF here. And I'll show you how Dockling does it by using a combination of OCR and specific vision models that extract the format and allow me to do things like structured output if I only say, for example, want a specific column to be outputted from a content source. So it's really cool. And I've been using it at Red Hat, my team uses it because we have thousands of PDFs that we need to work with specifically from product documentation. But also we have content like images that we want to extract using vision models. And what Dockling does with a here previously, because as a PDF, this data type is spread out and it's proprietary. So it's hard to extract this type of data from the source PDF here. And I'll show you how Dockling does it by using a combination of OCR and specific vision models that extract the format and allow me to do things like structured output if I only want a specific column to be outputted from a content source. So it's really cool. And I've been using it at Red Hat, my team uses it because we have thousands of PDFs that we need to work with specifically from product documentation. But also we have content like images that we want to extract using vision models. And what Dockling does with a pip install Dockling is allow me to convert single documents or websites or anything else to markdown or whatever file type I want, and be able to work with the page layout so I don't lose the consistency in the structure of the source document. And there's a lot of other integrations and for situations where maybe you need to run this locally, or you don't want to pay for a service or you have an air-gapped environment. So you're able to do this using this open source project that is part of the Linux Foundation. So before I show you the quick demo that I want to highlight the project, I want to talk a little bit about scale and cost. As I just mentioned it there earlier, but there is a public use case I want to show you from Landro at Hugging Face where he compared a source of common crawl PDFs and where he did a little bit of pre-work on them and actually extracted the structure using OCR and using Dockling in order to remove certain parts and clean this up for the FineWeb PDFs export, which is thousands of tokens from PDFs around the web that you could use for training a model, etc. But the two comparisons he did using GPU and using CPU for Dockling allow him to do this at 50 times cost savings compared to VLMs and OCR naively. And so this is what's really cool is that you can really scale this up. And he did this on CPU and not needing GPU, which is really cool. And it's not just document conversion, right? So his example was getting things ready from PDFs. But let's say we have images, right? This image with a vision language model was able to be described and annotated, right from this specific image right here. Now we have all of this really important context that can be used in a RAG application as an example, but also just for the end user to be able to understand what's happening in this image. And say, for example, the SME doesn't work there anymore at the organization. Now we have a way to understand what this is with the help of an LLM, whether proprietary or local. And then finally, it's not just document conversion and annotation, but also structured output. Say for example, I have this invoice and I need to extract specifically the bill number, the total invoice price, the name of the sender, we're able to get that in a format that is structured. So I don't have to worry about it pulling in the headings and titles. No, I just want the total invoice price and the bill number. And I can get that in a format that's Pydantic, but also just super simple if I'm just trying to get a couple things out of a huge document. So let's hop over to the demo. Let me show you what I'm talking about. And by the way, we're going to be using this docling workshop repository for the demos today. So feel free to check it out. In this first example, we're going to be using docling to convert a popular file type like a PDF, because remember that data is the foundation for all AI systems. And in order to leverage that data, we have to properly ingest different file formats with accuracy. And without doing that, we can lose information or information can become unreadable from tables and diagrams and images. So with docling, what we're able to do with a simple pip install down here is start processing some of these different file types. So I'll show you how we do that. First off, we're going to be importing essential components like the document converter, as well as some other dependencies. And I'll show you the simplest way, which is to just start out with a PDF that we have online, which is Dockling's own research paper. And here you can see we have different titles and subtitles, we have components such as images and captions that we have here. And at the bottom of this PDF, we also have something like a table here that we need to extract as well as images that might be helpful for our AI application or agent. So let me come back to the notebook here. And I'll show you how we do this simple conversion by using the Dockling document converter, and exporting this to markdown. So here you can see is a rough example of how fast this can be for an eight page PDF, to be able to export this in a way that an LLM can start using right. But at the same time, this is a Pydantic data type. So you can see we can explore this PDF, the number of pages and tables and see what is on which page of this PDF, and export this into a variety of different formats, like markdown, HTML, dictionary, and much more. But the real value here is not just with basic text and columns, but working with tables. So this PDF here has a variety of different tables that we want to be able to extract. And so by doing a converter for this specific document, we can then extract these tables here, and be able to export this to a data frame. So we can render this out in our Jupyter notebook here. So as I run this cell, you can see that we've exported eight different tables from that source PDF. And we can list these, but also be able to get these in a format ready to use in a RAG application, or just to query with our LLM. So we've taken a look at how to pull text and tables from a PDF. But what about visualizing and extracting images from that source document as well. Now for this specific example, what we'll do is set up a PDF pipeline that will allow us to scale up the images and push this into a document converter. So now when I go to inspect the images and picture content in the cells, we can see this nicely mapped out where we have the picture. So the source image, the caption of that, and all of the embedded text elements that we could use in some type of retrieval augmented generation application to ask questions about what's happening in these different photos in our source PDF. I think what would help here is also to be able to visualize the document layout by using the bounding boxes provided by the layout visualizer here. So here, what we're going to do is visualize all of the different elements and components that can be extracted from, say, for example, that source PDF. So section headers or text or subtitles or different components, such as that photo that we just pulled and extracted from the PDF. So this is one of the models that Dockling provides, which can be used for situations where you might have personally identifiable information from a customer that you want to remove from a source document type before you extract that into your application. We can also use vision About what's happening in these different photos in our source PDF. I think what would help here is also to be able to visualize the document layout by using the bounding boxes provided by the layout visualizer here. So here, what we're going to do is visualize all of the different elements and components that can be extracted from, say, for example, that source PDF. So section headers or text or subtitles or different components, such as that photo that we just pulled and extracted from the PDF. So this is one of the models that DocLean provides, which can be used for situations where you might have personal identifiable information from a customer that you want to remove from a source document type before you extract that into your application. We can also use vision language models in order to enrich the source images and diagrams that might be in these document types using something like OLAMA or a third party LLM. So what we're going to do here is set up a PDF pipeline that's going to use a local running granite model and say, hey, give us a detailed description of what's happening in this image. And with that document converter, we're going to go ahead and display that enriched document by calling the open AI endpoint with OLAMA that we have running locally at the completions endpoint. And so here we have an annotated caption of what's happening with that DocLean pipeline image that we were taking a look at earlier, where originally we were just pulling the caption. But now we can use a vision language model to describe what's happening in these photos. And with all this additional information, this can help us to build a solid RAG pipeline to where we can do questioning and answering over our source data. And for this example, I want to show you what's known as chunkless RAG or agentic RAG using docling. And now by starting off with our document outline, so processing a PDF with docling like we just did, we can allow an LLM to be able to pick the most relevant part of the document that is related to the user's question and pull that full text from the docling document itself to try to answer that question for the user. And this can run in an agentic loop. And what's really important here is that we're doing RAG but without having to use a chunker or embedding model or vector database, etc. So the index ends up being the markdown outline of the document. Now when the user asks a question, the entire retrieval index typically would be thousands of vectors in a database where we would do semantic similarity to make sure these sections are similar to the user's question. But for us, what we're going to do is be able to see a markdown outline of the document with each section summary outline. So if the LLM is looking for something about getting started with docling, well, it can just pull from this reference of text to see that docling can be installed for PyPI. And that's the entire retrieval index. So let's say we have a query, what are the main AI models using docling? Well, we're setting up a RAG agent here to be able to iterate on that specific question about five times. So we see that there are 20 sections available when the users ask the question. And we're going to search for that specific part of text that talks about the AI models and be able to determine, is this relevant to answering the question. And you can see here the final answer in one iteration was pulled from that source material without having to go in a vector database, but instead search that docling document structure for the specific text. So it's a quite interesting way to be able to answer users questions through this chunkless retrieval augmented generation pattern. And while this first example might have been simple, what we can do is also pull the IBM 2025 annual report into the context here, which has 418 sections. So it's much larger and ask a question like, what was Red Hat's revenue growth in 2025? And how did it contribute to overall software segment? And so here we're going to be iterating multiple times to figure out, is this section relevant to the user's question? And if not, we need to pull in more information. And so that is how chunkless RAG can work in a situation where you're using a tool like docling. But what happens when we have hundreds or hundreds of thousands of PDFs that we want to have processed? Well, this is where we can deploy docling as a REST API service using something that's known as docling serve. This allows us to scale things up and to run this as a microservice as a container or through Kubernetes. So when we set things up here, we're going to do a pip install docling serve. And we're going to be able to serve this from a CLI with docling serve on a specific port. Or once we had that server started, be able to send things to that specific endpoint with different types of options and arguments, do we want OCR? Do we want a specific backend? Or do we want images annotated? And so that's how we can scale things up and allow this API endpoint to be able to handle hundreds or thousands of different document types at a time. And let's say for example, that you're trying to build an AI agent, you can also use the docling MCP server. So this allows us to automate things with our AI agent and give it the capabilities that docling has through that model context protocol and allow us to standardize the communication between, say, for example, cloud code or continue in our developer CLI to the MCP server, which can handle the document processing for us without us having to know all of those different arguments and commands. So it makes it quite easy. And the example I show here is by using one of the client models on my own Mac itself and connecting this to my VS code instance. So let's understand the tools that are available. For the MCP server, we have conversion tools, generation tools, say, for example, if I want to process a specific part of a PDF and manipulation tools. And this is all provided to the LLM and the agent that we're going to be using with the MCP server. So here, I'm just checking that my local MLX server to run an LLM is running. And it looks like we've got Qwen 3.6 here. And we're going to verify that the docling MCP server is also running, which would be done using UVX here. And so we'll do that in this cell here to make sure that the MCP server is available. Now, in cloud code or codex or another type of AI application, we're going to install the extension. So for us, this means adding an MCP server in the config.yaml. And now at this point, we can use a model and an MCP server to do things such as, convert this document, give me a summary, or create a document with a section of action items and pull in a list from another PDF and export that as markdown. So we can use all of those docling components through the MCP server in order to agentically process and parse these documents using an AI agent like cursor or cloud code, or one of the many open source options that are out there. Now let's head back to the slides and wrap things up. So let's put it all together with docling. We've seen that we can take a PDF into a format like markdown or JSON with a fully local operation from our own machine without even needing model and an MCP server to do things such as convert this document, give me a summary, or create a document with a section of action items and pull in a list from another PDF and export that as markdown. So we can use all of those docking components through the MCP server in order to agentically process and parse these documents using an AI agent like cursor or cloud code, or one of the many open source options that are out there. Now let's head back to the slides and wrap things up. So let's put it all together with docking. We've seen that we can take a PDF into a format like markdown or JSON with a fully local operation from our own machine without even needing a GPU. It's fast, it's cheap, and most importantly, it's open source. So I encourage you to check it all out. Behind the scenes, there's different pipelines that use a combination of OCR and layout analysis to structure everything together, put it together as a Pydantic dockling document that then you can use to export to different types of formats, create datasets, chunk that using the hybrid chunker, and do so much more. It integrates with a lot of different RAG frameworks and agentic systems and harnesses. So feel free to try it out. And I want to give a big thank you to the AI engineering team for having me on. This is something that I'm really passionate about with processing documents, but also doing this in an open source way because we're here at Red Hat and we love open source. So feel free to check out the slides, connect with me on LinkedIn. I appreciate the opportunity. Enjoy the conference and keep up with the AI engineering ecosystem. Things are looking super bright right now for the open source world and for AI engineers in general. So see you next time. And thanks for watching. you understand what I mean, right? All this data needs to be transformed into something that an LLM can actually understand. And that's why by the end of this session, you'll understand how to extract structure between raw enterprise documents and use it to power better downstream AI systems like RAG and agents. So let's get started. Now, I think Jensen from NVIDIA made this point super clear at his keynote that unstructured data is becoming this new context layer for AI. And the reality, though, for many teams, and I know this personally working at Red Hat, that PDFs and data are spread across dozens of different systems. So we've got a lot to cover today. As you might know, a large majority of the world's data is unstructured. And so no matter what model you're using, if you're working with data like PDFs and unstructured types of formats, this is a bit tricky to work with. Because there are solutions out there, but they might be proprietary or require you to send your private data to someone else's server. And for not just text, how do we take documents and their graphs or tables and images to formats that LLMs can understand like Markdown or JSON? And I'm going to show you how in the session today, because we're going to be using an open source tool, part of the Linux foundation that is called Dockling, and learn about extraction, parsing, chunking, and much more. And I've got some live demos for you. So we're going to have some fun. And just in case you'd like it, we have the session slides here on the right, and a little overview of the specifics I'll be showing you in today's session. But without further ado, let's get started. So why is there a need for advanced document processing? As I briefly mentioned before, you might have a lot of technical documentation or meeting minutes or different types of documents and invoices that you need to use and maybe RAG or different type of applications where the context is provided to an LLM. So whether it's RAG or retrieval augmented generation to answer questions based on this data, or you're using this to fine tune a new specialized model, well, data is this key ingredient behind those applications. And it doesn't matter if you're using, you know, Nvidia acceleration or an open source or proprietary model, that data and the way you process it is the key determining factor in whether your answer is going to be correct or incorrect for the user or customer at the end of the day. And that's what's most important. And how important is it? Well, I have this viral tweet from earlier where 20 scientific papers now feature a new nonsensical term that doesn't exist because AI misinterpreted a very old article that was scanned and taken to a PDF, merging two different words from two different columns in this PDF. And because researchers are using these models in order to help them write, now we have different types of scientific papers that all feature this word and are even being cited by other people. And so that's how important it is to make sure that the data that we're processing is processed in a way that's accurate and not hallucinated and able to be used confidently in our applications that we're delivering to users and customers. So it's quite important. Now, if we were to use a tool like dockling that I can run on my own machine, you could see that these two words are quite far away from each other and shouldn't have been combined in the first place. But that's how we're going to learn about extracting this text here in a second. So if we were to try a simple PDF parser for a PDF like this, that includes a table here, it has an image, there's captions, and there's regular sections of text. Well, we might get an answer like this here on the right in Markdown. You know, this might be very fast and cheap to run, even on CPU. But the issue is, is that a lot of this text has been truncated, has been merged, and isn't decipherable even by me as a human. And if I sent this to a model, I don't think I could trust that the model could extract specifics from, say, for example, this table, because the table has been kind of just spit out linearly. And this information isn't fit for most use cases where I need to ask questions or have an agent do validation and extraction on this source data. So this isn't going to cut it, right? There's undesired page headers. We don't understand the table. And where's the content from the image, right? It's not even there. When we're using frontier models, this is kind of not bad, right? But quite expensive. I'm sending this to a model that's maybe $30 per million output tokens. You can see how this can get quite expensive as I scale this up to dozens or hundreds or, in a lot of cases, thousands of PDFs that organizations have to work through to use an AI application. And the differences between maybe a 5.1 of a model that was depreciated and a 5.2 version of a model make it tricky to have structured output that's consistent every single time. And so while it might be good quality, and I can see that most of this looks accurate in the exported markdown, we might be susceptible to hallucinations because models are non-deterministic. And this is really tricky at scale. And so what is the middle ground? Well, that's where Dockling comes in. It's a fast and cheap and most importantly, local CLI and library that I can use to take various types of input sources and convert this to markdown, JSON, and a pydantic data type that I can use in my applications and that I can scale up if I have thousands of different types of formats that need to be used or translated to something like markdown. And so we rendered this as HTML, but you can see here we've got this export of the specific table that we had here previously, because as a PDF, this data type is spread out and it's proprietary. So it's hard to extract this type of data from the source PDF here. And I'll show you how Dockling does it by using a combination of OCR and specific vision models that extract the format and allow me to do things like structured output if I only say, for example, want a specific column to be outputted from a content source. So it's really cool. And I've been using it at Red Hat, my team uses it because we have thousands of PDFs that we need to work with specifically from product documentation. But also we have content like images that we want to extract using vision models. And what Dockling does with a pip install Dockling is allow me to convert single documents or websites or anything else to markdown or whatever file type I want, and be able to work with the page layout so I don't lose the consistency in the structure of the source document. And there's a lot of other integrations and for situations where maybe you need to run this locally, or you don't want to pay for a service or you have an air-gapped environment. So you're able to do this using this open source project that is part of the Linux foundation. So before I show you the quick demo that I want to highlight the project, I want to talk a little bit about scale and cost. As I just mentioned it there earlier, but there is a public use case I want to show you from Landro at Hugging Face where he compared a source of common crawl PDFs and where he did a little bit of pre-work on them and actually extracted the structure using OCR and using Dockling in order to remove certain parts and clean this up for the find PDFs export, which is thousands of tokens from PDFs around the web that you could use for training a model, etc. But the two comparisons he did using GPU and using CPU for Dockling allow him to do this at 50 times of a cost savings compared to VLMs and OCR naively. And so this is what's really cool is that you can really scale this up. And he did this on CPU and not needing GPU, which is really cool. And it's not just document conversion, right? So his example was getting things ready from PDFs. But let's say we have images, right? This image with a vision language model was able to be described and annotated, right from this specific image right here. Now we have all of this really important context that can be used in a RAG application as an example, but also just for the end user to be able to understand what's happening in this image. And say, for example, the SME doesn't work there anymore at the organization. Now we have a way to understand what this is with the help of an LLM, whether proprietary or local. And then finally, it's not just document conversion and annotation, but also structured output. Say for example, I have this invoice and I need to extract specifically the bill number, the total invoice price, the name of the sender, we're able to get that in a format that is structured. So I don't have to worry about, hey, it's going to pull in like the headings and titles. No, I just want the, you know, total invoice price and the bill number. And I can get that in a format that's pydantic, but also just super simple if I'm just trying to get a couple things out of a huge document. So let's hop over to the demo. Let me show you what I'm talking about. And by the way, we're going to be using this docling workshop repository for the demos today. So feel free to check it out. In this first example, we're going to be using docling to convert a popular file type like a PDF, because remember that data is the foundation for all AI systems. And in order to leverage that data, we have to properly ingest different file formats with accuracy. And without doing that, we can lose information or information can become unreadable from tables and diagrams and images. So with docling, what we're able to do with a simple pip install down here is start processing some of these different file types. So I'll show you how we do that. First off, we're going to be importing essential components like the document converter, as well as some other dependencies. And I'll show you the simplest way, which is to just start out with a PDF that we have online, which is doclings own research paper. And here you can see we have different titles and subtitles, we have components such as images and captions that we have here. And at the bottom of this PDF, we also have something like a table here that we need to extract as well as images that might be helpful for our AI application or agent. So let me come back to the notebook here. And I'll show you how we do this simple conversion by using the doclings document converter, and exporting this to markdown. So here you can see is a rough example of how fast this can be for a eight page PDF, to be able to export this in a way that an LLM can start using right. But at the same time, this is a Pydantic data type. So you can see we can explore this PDF, the number of pages and tables and see what is on which page of this PDF, and export this into a variety of different formats, like markdown, HTML, dictionary, and much more. But the real value here is not just with basic text and columns, but working with tables. So this PDF here has a variety of different tables that we want to be able to extract. And so by doing a converter for this specific document, we can then extract these tables here, and be able to export this to a data frame. So we can render this out in our Jupyter notebook here. So as I run this cell, you can see that we've exported eight different tables from that source PDF. And we can list these, but also be able to get these in a format ready to use in a RAG application, or just to query with our LLM. So we've taken a look at how to pull text and tables from a PDF. But what about visualizing and extracting images from that source document as well. Now for this specific example, what we'll do is set up a PDF pipeline that will allow us to scale up the images and push this into a document converter. So now when I go to inspect the images and picture content in the cells, we can see this nicely mapped out where we have the picture. So the source image, the caption of that, and all of the embedded text elements that we could use in some type of retrieval augmented generation application to ask questions about, hey, what's happening in these different photos in our source PDF. I think what would help here is also to be able to visualize the document layout by using the bounding boxes provided by the layout visualizer here. So here, what we're going to do is visualize all of the different elements and components that can be extracted from, say, for example, that source PDF. So section headers or text or subtitles or different components, such as that photo that we just pulled and extracted from the PDF. So this is one of the models that DocLean provides, which can be used for situations where you might have personal identifiable information from a customer that you want to remove from a source document type before you extract that into your application. We can also use vision language models in order to enrich the source images and diagrams that might be in these document types using something like OLAMA or a third party LLN. So what we're going to do here is set up a PDF pipeline that's going to use a local running granite model and say, hey, give us a detailed description of what's happening in this image. And with that document converter, we're going to go ahead and display that enriched document by calling the open AI endpoint with OLAMA that we have running locally at the completions endpoint. And so here we have an annotated caption of what's happening with that DocLean pipeline image that we were taking a look at earlier, where originally we were just pulling the caption. But now we can use a vision language model to describe what's happening in these photos. And with all this additional information, this can help us to build a really solid rag pipeline to where we can do questioning and answering over our source data. And for this example, I want to show you what's known as chunkless rag or agentic rag using docling. And now by starting off with our document outline, so processing a PDF with docling like we just did, we can allow an LLN to be able to pick the most relevant part of the document that is related to the user's question and pull that full text from the docling document itself to try to answer that question for the user. And this can run in an agentic loop. And what's really important here is that we're doing rag but without having to use a chunker or embedding model or vector database, etc, etc. So the index ends up being the markdown outline of the document. Now when the user asks a question, the entire retrieval index typically would be thousands of vectors in a database where we would do semantic similarity to make sure hey, these sections are similar to the user's question. But for us, what we're going to do is be able to see a markdown outline of the document with each section summary outline. So if the LLN is looking for something about getting started with docling, well, it can just pull from this reference of text to see that docling can be installed for PyPy. And that's the entire retrieval index. So let's say we have a query, what are the main AI models using docling? Well, we're setting up a rag agent here to be able to iterate on that specific question about five times. So we see that there are 20 sections available when the users ask the question. And we're going to search for that specific part of text that talks about the AI models and be able to determine hey, is this relevant to answering the question. And you can see here the final answer in one iteration was pulled from that source material without having to go in a vector database, but instead search that docling document structure for the specific text. So it's a quite interesting way to be able to answer users questions through this chunkless retrieval augmented generation pattern. And while this first example might have been simple, what we can do is also pull the IBM 2025 annual report into the context here, which has 418 sections. So it's much larger and ask a question like, hey, what was Red Hat's revenue growth in 2025? And how did it contribute to an overall software segment? And so here we're going to be iterating multiple times to figure out, hey, is this section relevant to the user's question? And if not, we need to pull in more information. And so that is how chunkless RAG can work in a situation where you're using a tool like docling. But what happens when we have hundreds or hundreds of 1000s of PDFs that we want to have processed? Well, this is where we can deploy docling as a REST API service using something that's known as docling serve. This allows us to scale things up and to run this as a microservice as a container or through Kubernetes. So when we set things up here, we're going to do a pip install docling serve. And we're going to be able to serve this from a CLI with docling serve on a specific port. Or once we had that server started, be able to send things to that specific endpoint with different types of options and arguments like, hey, do we want OCR? Do we want a specific backend? Or do we want images annotated? And so that's how we can scale things up and allow this API endpoint to be able to handle hundreds or thousands of different document types at a time. And let's say for example, that you're trying to build an AI agent, you can also use the docling MCP server. So this allows us to automate things with our AI agent and give it the capabilities that docling has through that model context protocol and allow us to standardize the communication between, say, for example, cloud code or continue in our developer CLI to the MCP server, which can handle the document processing for us without us having to know all of those different arguments and commands. So it makes it quite easy. And the example I show here is by using one of the clint models on my own Mac itself and connecting this to my VS code instance. So let's understand the tools that are available. For the MCP server, we have conversion tools, generation tools, say, for example, if I want to process a specific part of a PDF and manipulation tools. And this is all provided to the LLM and the agent that we're going to be using with the MCP server. So here, I'm just checking that my local MLX server to run an LLM is running. And it looks like we've got Quinn 3.6 here. And we're going to verify that the docling MCP server is also running, which would be done using UVX here. And so we'll do that in this cell here to make sure that the MCP server is available. Now, in cloud code or codex or another type of AI application, we're going to install the extension. So for us, this means adding an MCP server in the config.yaml. And now at this point, we can use a model and an MCP server to do things such as, hey, convert this document, give me a summary, or create a document with a section of action items and pull in a list from another PDF and export that is markdown. So we can use all of those docking components through the MCP server in order to agentically process and parse these documents using an AI agent like cursor or cloud code, or one of the many open source options that are out there. Now let's head back to the slides and wrap things up. So let's put it all together with docking. We've seen that we can take a PDF into a format like markdown or JSON with a fully local operation from our own machine without even needing a GPU. It's fast, it's cheap, and most importantly, it's open source. So I encourage you to check it all out. Behind the scenes, there's different pipelines that use a combination of OCR and layout analysis to structure everything together, put it together as a Pydantic dockling document that then you can use to export to different types of formats, create datasets, chunk that using the hybrid chunker, and do so much more. It integrates with a lot of different RAG frameworks and agentic systems and harnesses. So feel free to try it out. And I want to give a big thank you to the AI engineering team for having me on. This is something that I'm really passionate about with processing documents, but also doing this in an open source way because, you know, we're here at Red Hat and we love open source. So feel free to check out the slides, connect with me on LinkedIn. I appreciate the opportunity. Enjoy the conference and keep up with the AI engineering ecosystem. Things are looking super bright right now for the open source world and for AI engineers in general. So see you next time. And thanks for watching.