Your Agreements Are a Database You Can't Query — Hiral Shah, Docusign & Sean Sodha, NVIDIA

Transcript

Welcome everyone to our session. I'm Haral, I'm a senior director of products at DocuSign, and I'm joined by Sean. Hello everyone, I'm a product manager at NVIDIA. So today Sean and I are going to talk about a massive problem that every enterprise faces, which is agreement data, large-scale agreement data. Agreements are a big part of any relationship, any B2B organization goes through, day in, day out, and a lot of that data is captured inside agreements, and it's very critical, whether it's pricing tables, and it's all in a lot of different unstructured format, and that's what we're going to show is how DocuSign partnered with NVIDIA are fixing that on making that data available, readable, usable for a lot of our organizations. So here is a quick map of our talk today. We'll start with just the stakes. Why does this matter? Why is the scale so large? And then we'll dive deep into the technical architecture of how we are approaching it, how we have tackled this thing, especially document processing at scale. And then finally we'll cover what we have learned from our evaluation of all of the different models we've tried for different purposes and share our learnings with you. So why, just to understand why this is a big problem. When you think about DocuSign, raise your hands, how many of you have used DocuSign? Anyone who's employed probably the HR docs, right? So it's a massive scale. Everyone uses DocuSign. For us, it's a massive engineering problem as well, because just look at the scale. We have 1.9 million customers who are paying us and a billion users. What does that imply? We process a million agreements a day that needs to now be structured, made readable, made queryable, usable. And in the past we've worked with Deloitte on a study and it says that there's $2 trillion captured in this agreement negotiated value that no one capitalizes, no one goes back and gets that data back. And why? It's because they have to do a lot of human reading, human reviews, there's disconnected systems, a lot of manual workflows that are there. So that's why DocuSign built IAM, an intelligent agreement management platform that takes the entire approach in an AI first way, the entire agreement life cycle. Whether you are creating agreements, Gen AI helps a lot with that, whether you're negotiating to understanding and redlining, all the way to after signing, storing, and making a lot of insights from this data. So when you think about the challenges involved, in an agreement, there is the unstructured data. It could be a PDF, it could be a PNG. And what are people wanting to do is simple questions. They can't get that. That is the data that is trapped inside one agreement, but also the whole corpus of millions of agreements, 10 years, 20 years of business has put into that. So agreements aren't flat. They're also hierarchical in nature. One agreement governs the other. The other does something else. So you always are needing a lot of things to answer a question. Simple thing. I'm sure you all are using a lot of Claude and all, especially at your company's organization. Simple thing. What did we contract for the total tokens with Claude? No one knows. That's captured inside this agreement in different forms and fashion. So you need to be extracting this data to find things, but also insights and push it downstream where you're tracking doing more things. And when you analyze an enterprise contract, a big set of things are captured, what I call vital terms, like pricing tiers, the SKUs, the information, SLAs, rate cards, they're all in table format. Now, traditional document extraction tools or a generic LLMs completely fail here. We've tried, we've definitely done this, because they're reading text line by line, which breaks a lot of that concept within the table. Merged cells. Some things are not bounded. So this makes a massive operational overhead for the downstream legal teams, procurement teams, sales teams to get that queries and get that answers done. And they spend hours and hours digging through this to even just locate a basic thing. And that's where we partnered with NVIDIA and leveraged a purpose-built model, which is really making our architecture for table extraction take us to make and solve these complex use cases. So with this, we are making things scalable. We can understand it with the layout, but also deliver really accurate results. And to share more about how we're leveraging the NemoTron, I'm going to hand it to Sean. Sean Huston: All right, hello everyone. So real quick on the NemoTron Retriever initiative. So for those who know about NemoTron, you can raise your hand real quick. Awesome. So NemoTron is all about building world-class open source models and publishing the datasets, the techniques, the quantization approaches, distillation approaches, pruning approaches, every technique possible, blueprints to go with that, you name it. Throughout the NemoTron portfolio, we have specifically NemoTron Retriever, which is building embedding models, re-ranking models, and document extraction models. So real quick here, our first initiative is, if you're a large scale enterprise that deals with petabytes scale data, our first initiative is how do we make sure that you find the right document given a certain query, your agent sends a set of queries to the corpus afterwards. Once you find those top five documents, whatever it may be, then we say, okay, you found the right document, now how do you then find the right information within the document? And this is where the work with the DocuSign team has gone really great, where we've worked with them to build the NemoTron parse model to focus specifically on table extraction, which is a really complicated technique. If you think about it, the number of permutations of tables are quite vast when you think about nested tables, merged cells, merged columns, merged rows, whatever it may be, and that can get really complex and really hairy of a problem. So real quick, as I mentioned before, our team is responsible for leading a lot of leaderboards in the retrieval space. So Vidori, V1, V2, V3, MTab, MMTab. So our team knows how to build world-class retrieval models, given a lot of leadership, given a lot of leaderboard winnings that we've had in the last year or so. And then of course, as I mentioned before, we open source everything. So we share the open source model weights, the techniques, and then we release with those blueprints and skills that agents can use afterwards. So touch a little bit on the actual model that we are working with with DocuSign was the NemoTron parse model. So when you think VLM, you generally think a multi-billion parameter model. It's very heavy. It's high latency. This is a very small, tiny C-radio VLM. It's about 850, 900 million parameter model designed to be that all-in-one package model where you deploy it. And instead of having small, let's say, YOLOX models that do table extraction or page element extraction, wherever it may be, this is a single-shot model that you can feed a document in, and out comes the semantic formatting layouts, the text, the reading order, the preserved structure of the table, et cetera. This can be served via the NVIDIA NIM or via VLLM as well. And so it's a tiny, small model that you can use. It's not a generator. It's more of an extractor at the end of the day. So real quick as well, we always want to make sure that we're building towards benchmarks that matter most to the enterprise space. So we want to make sure that both on the Pareto curve of accuracy versus performance, we'll make sure that we're going to be releasing world-class models to the ecosystem. So you'll see here generally is just a very standard benchmark of table extraction. I believe this one was RD TableBench. And we compare some popular open source models here, and then we compare how our NemoTron parse model does compared to that industry. And we continue to strive to improve this as time goes on. So I think we believe we have a demo as well. Yeah. Is that as well? Just press one. Yeah. There we go. Let me show you how easy it is to turn any agreement into structured, usable data with Agreement Manager, which is a central repository of every agreement an organization has ever signed. Let's look at this. So when we look at the Agreement Manager view here, we have an ability to see the entire list of agreements, but also go and upload a new agreement. So I'm uploading a new order form into Agreement Manager. As you can see, I can select from a computer, import from other places. The moment I select the agreement, it starts uploading. of table extraction. I believe this one was RD TableBench. And we compare some popular open source models here, and then we compare how our NemoTron parse model does compared to that industry. And we continue to strive to improve this as time goes on. So I think we believe we have a demo as well. Yeah. Is that as well? Just press one. Yeah. There we go. Let me show you how easy it is to turn any agreement into structured, usable data with Agreement Manager, which is a central repository of every agreement an organization has ever signed. Let's look at this. So when we look at the Agreement Manager view here, we have an ability to see the entire list of agreements, but also go and upload a new agreement. So I'm uploading a new order form into Agreement Manager. As you can see, I can select from a computer, import from other places. The moment I select the agreement, it starts uploading and starts processing with AI. And just like that, you can see that the jobs engine has processed it. Let's take a closer look at this agreement. So when you go into the action, you can go and browse the file. Within seconds, Agreement Manager has extracted a rich set of metadata. Everything from key terms to commercial details are automatically structured, highlighted. And immediately, you can jump to that section where the details are found. Built-in goes deeper. This is where the NVIDIA's model comes in, that it's extracted all the structured pricing data around this agreement. It goes in, breaks down these complex tables into order details. And as you can see, we can break it down. We can download all of this data. This is powered by the advanced parsing, leveraging NVIDIA's NemoTron model, turning even dense tables into something that is instantly usable. And of course, you can take this data with you. You can see when we've downloaded into CSV, how we've structured all of it for your finance team, procurement team, even further analysis. All of it is also available through API. And that's how Agreement Manager has transformed agreements into actionable insights in seconds, leveraging NVIDIA. So I think what you saw there from a demo perspective, we've tried to shorten it. We have a whole repository. What you see a list, we get customers which has thousand agreements to all the way millions of agreements within. But the big piece is how do we understand and get that data that makes it very valuable to an end business user, right? A legal person, a procurement person, a salesperson who's doing a lot of the deals, or even a leader, right? A business unit. The CTO goes and asks, what did we do? This is how we have making each of the things a lot more structured, so we have our own proprietary agreement data model, which we are structurizing each agreement but also at a whole organization level, and leveraging a lot of the NVIDIA things. We've been able to do a really good job, especially with all of those tables, pricing, SLAs, and then make that available. And then we also have a robust search that is on top of it. So when you think about what have we learned, right? When you think from a NemoTron plus DocuSign, one of the biggest things for us, we definitely have done a lot of different models for different purposes. So a purpose-built model for the job you're trying to do is a big, big part of how we've been thinking about, and that's where we've been able to accelerate, bring things to market much faster. The second big piece around the model efficiency, so as Sean was talking about, the number of parameters, context and all that matters in a different environment for different things. For us, the lower context also meant lower latency, lower cost, to deliver the scale that we are talking about. Last, around the faster extraction. So we ran this against a lot of the other open source models. When you think about how many tables can it extract per second, NemoTron was 20x faster, which helps us when we're talking about the millions and billions of scale that we are serving for all of our customers. So a lot of it is having that smaller purpose-built things is the way for an enterprise as an organization to go and leverage, and then serve that from an end user perspective. And then what's next? So let Sean talk through those. Yeah, so working with the DocuSign team has been awesome so far, and we're going to continue to deepen that partnership as well over the next few months. So with them, we started with, how do I extract as much possible information from a page, and now we'll scale to, how do I now find that page to begin with? So we'll start a little bit with the NemoTron, NemoRetriever effort, and then of course we'll talk a little bit about the NVIDIA agent toolkit with them over the next few months, and then actually start scaling into more production-scale agents then. So, perfect. I think that's what we had. We have time for a couple questions. Anyone in the room? Okay, I have someone there. So just to recap for everybody, if you didn't hear, the question is, right, OCR is always a thorn in the whole process, so are we thinking about letting that go and starting from agentic from the get-go? I can talk from my perspective. So I think for us, right, there are different use cases at different points in time. Many times, if you are reactive, you have a question and you're coming, some of that can work dynamically at a smaller scale. The question is the latency. When I am querying at that scale of thousands, I do need to have pre-processed and identified, so that's one. I think the second big part of the use case for us, a lot of times businesses want to use this data to do a lot of downstream work. So an example is a procurement team. This is my pricing table. I want to put it into Coupa to make sure when I'm paying that works. At that time, the agent is helping, but I can't do that on a one document by document basis. That said, there are ways that we are compressing. That's why NemoTron worked for us. It's how do you do it from a layout understanding just for that purpose? But I would like to let you add. Yeah, I think it depends on the use case a little bit. I think for this specific instance, right, you have petabytes of documents that you want to be queryable at some point, right? So you are heavy on the compute at the upfront side with all the OCR. So you don't have to worry about it later on, right? And there are some instances where people may upload a contract to begin with for Q&A, and that's a very high, that's a very low latency use case, right? So you have a high throughput versus a latency use case. And in that scenario, your different batch sizes, your concurrences, your different techniques on how you process the document will be different. And where you spend that compute in that cycle will be changing between the different use cases. One more there. Yeah. We do a lot of what I call hybrid approach and purpose built for the needs and the use cases. So from a table piece, it does break down the whole layout along with extracting. We still do OCR from a lot of other fields and metadata and the clauses, all of the text type of thing. So I think we had an architecture where we have our pipeline going through two different routes for that. As a follow-up, we have a blog out there. How are we really solving this at scale across? And if you look at that, there are a lot of different piecemeal modules and things together. Yeah, one more. Not last. At scale, what's the modification? What do you mean to do with that model? So we, this model is currently on FP16, but there are paths towards going to FP8 and FP4 in the next few months as well. What about... We do use that. And then we are also using some of the older ones. And that's the journey as a partnership is to tweak as you get more of the customers. Yeah. So for this, there are many techniques on how to improve the performance side, right? So quantization, right? We're trying to move everyone to Blackwell, right? That's why NVFB is the big thing now, as well as multi-token generation for this. It's a VLM architecture, right? So your encoder decoder techniques can definitely be further optimized. Right now this model just generates one token at a time. You can do multi-token generation, of course. So there's plenty of performance things. Right now we're focusing on the accuracy side. Are we adding value to the system? And then from there, we'll then push out that Pareto curve on the performance side. So what's up? Are you guys using the catalogs on the video GPUs, or are you primarily running through the port? I believe they just deploy the VLM directly. So there are many techniques on how to improve the performance side, right? Quantization, right? We're trying to move everyone to Blackwell, right? That's why NVF before is the big thing now, as well as multi-token generation for this. It's a VLM architecture, right? Your encoder decoder techniques can definitely be further optimized. Right now this model just generates one token at a time. You can do multi-token generation, of course, too. So there's plenty of performance things. Right now we're focusing on the accuracy side. Are we adding value to the system? And then from there we'll push out that Pareto curve on the performance side. Are you guys using the catalogs on the video GPUs, or are you primarily running through the port? I believe they just deploy the VLM directly. I've done a lot on recall side. When you do that demo, you ask the question, is that graph, or is this a new port? This is just an extraction. This is now a retrieval. It's coming from that agreement data that we've extracted and stored. Maybe I can chat with you offline about how our architecture works fully as well. We're almost coming up on time there. I think that's all we have. I'm happy to hang around in the back with more questions. Good luck with a lot of your challenges with AI. Thank you. Thank you. I would like to catch up in November. I think that's what we had. We have time for a couple questions. Anyone in the room? Okay, I have someone there. So just to recap for everybody, if you didn't hear, it was, the question is, right, like OCR is always a thorn in the whole process, so are we thinking about letting the go of that and starting from agentic from the get-go? I can talk from my perspective. So I think for us, right, like there are different use cases at different points in time. Many times, if you are reactive, you have a question and you're coming, some of that can work dynamically at a smaller scale. The question is the latency. When I am querying at that scale of thousands, I do need to have pre-processed have identified, so that's one. I think the second big part of the use case for us, a lot of times businesses want to use this data to do a lot of downstream work. So an example is a procurement team. This is my pricing table. I want to put it into Coupa to make sure when I'm paying that works. At that time, like, you know, the agent is kind of helping, but I can't do that on a one document by document. That said, there is ways that we are compressing. That's kind of why NemoTron worked for us. It's like, how do you do it from a layout understanding just for that purpose? But I would like to let you add. Yeah, I think it depends on the use case a little bit. I think for this specific instance, right, you have petabytes of documents that you want to be queryable at some point, right? So you are heavy on the compute at the upfront side with all the OCR. So you don't have to worry about it later on, right? And there's some instances where people may upload a contract to begin with for Q&A, and that's a very high, that's a very low latency use case, right? So you have a high throughput versus a latency use case. And in that scenario, your different batch sizes, your concurrences, your different techniques on how you process the document will be different. And where you spend that compute in that cycle will be changing between the different use cases. One more there. Yeah. We do a lot of like more of what I call hybrid approach and a purpose built for like the needs and the use cases. So from a table piece, it does kind of, you know, do the whole layout along with extracting. We still do OCR from a lot of other fields and metadata and the clauses like all of the text kind of thing. So I think we had an architecture where we have our pipeline going through two different routes for that. As a follow, we have a blog out there. How are we really solving this at scale across? And if you look at that, there's a lot of different piecemeal modules and stuff together. Yeah, one more. Not last. At scale, what's the modification? What do you mean to do with that model? So we, this model is currently on FP16, but there are paths towards going on to FP8 and FP4 in the next few months as well too. What about . We do use that. And then we are also kind of using some of the older ones. And that's the journey as a partnership is to kind of go tweak as you get more of the customers. Yeah. So for this, there are many techniques on how to improve the performance side, right? So quantization, right? So we're trying to move everyone to Blackwell, right? That's why NVF before is the big thing now, as well as multi-token generation for this. It's a VLM architecture, right? So your encoder decoder techniques can definitely be further optimized. So right now this model just generates one token at a time. You can do multi-token generation, of course, too. So there's plenty of performance things. Right now we're focusing on the accuracy side. Like, are we adding value to the system? And then from there, we'll then push out that Pareto curve on the performance side. So what's up? Are you guys using, like, the catalogs on the video GPUs, or are you, like, primarily running through the port? I believe they just deploy the VLM directly. No, I've done a lot on recall side. When you, like, when you do that demo, you ask the question, is that graph, or is this a new port? Oh, this is just an extraction. This is now a retrieval. Yeah. Yeah. Yeah. It's coming from that agreement data that we've kind of extracted and stored. Yeah, maybe I can chat with you offline and how our architecture kind of works fully as well. No, we're almost coming up on time there. But I think that's kind of all we have. I'm happy to hang around in the back with more questions. And good luck with a lot of your challenges with AI. Thank you. Thank you. I would like to catch up on November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November November Thank you.