AI Engineer

Building GTM AI Agents: Lessons from Deploying to 6,000 Users — Sait Izmit, Snowflake

2181 summary words 10 min summary Watch video

Start with the signal

10 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: A GTM AI agent succeeds at enterprise scale not by maximizing initial capability, but by earning trust through narrow high-quality use cases, disciplined activation, rapid architectural iteration, and log-driven feedback loops.
  • Why it matters: Snowflake's internal assistant has served more than 1 million questions at roughly 40,000 questions per week for 6,000 GTM users, offering practical operating patterns for deploying and evolving an agent system rather than merely demonstrating one.
  • Best use: Use this as an operating playbook for launching an internal sales or GTM copilot: define quality gates, stage rollout, instrument adoption separately from retention, budget for change management, and build a telemetry-to-improvement loop.

Executive Summary

Sait Izmit describes Snowflake's internal GTM assistant as a production-scale effort to make sales teams more effective despite fragmented data, excessive tool switching, and account loads too large for humans to monitor. The assistant consolidates access to first- and third-party data, Salesforce, support signals, call transcripts, internal knowledge, and workflow tools so sellers can ask questions, automate work, and ultimately operate with richer customer context.

His central deployment lesson is "quality over coverage." Snowflake began with a deliberately constrained agent, testing it against 150 sales-process questions before launch and accepting that many desired data sources would arrive later. The team preferred answering 50 questions at 95% accuracy over answering 100 at 70%, because a poor first few interactions destroy user trust disproportionately. About 60% of the data in the current system was added after launch.

The rollout was run as a product-and-change-management program: a pilot with AI-native users to prove quality, a 10% beta of roughly 600 people to identify workflow-critical gaps and validate retention, then general availability. Snowflake used a greater-than-70% weekly-active-user retention threshold before proceeding. Izmit argues that low adoption among people who have not tried the product is not a product-quality failure; it is an activation failure requiring executive sponsorship, demos, manager-level adoption visibility, and sustained enablement.

The architecture was intentionally allowed to evolve. The initial 6,000-user launch used nine pages of agent instructions, Cortex Analyst and semantic views for structured data, Cortex Search for unstructured data, and a Google Doc for instruction versioning. It later accumulated CI/CD, evals, skills, MCP connections, progressive disclosure, memory, scheduling, and Slack delivery. At present, the agent includes 15 semantic views, 85 tables, 3,000 columns, five to six MCP connections, and nearly 20 skills; Izmit says 30-40% of ongoing work is re-architecture. Logs are the force multiplier: LLM-classified interaction data reveals product gaps and enablement needs in real time, allowing Snowflake to generate and feed new sales material back into the system quickly.

Key Takeaways

  • Claim: Optimize the initial agent for trustworthy answers in a narrow domain rather than broad but unreliable coverage. | Evidence: Before trying the existing agent, Izmit wrote 150 questions derived from the sales process; the first tests showed roughly 50% accuracy. The team chose a target posture of answering 50 questions at 95% accuracy rather than 100 at 70%, and says 60% of current data was added only after launch. | Implication: For Ken's agents, establish a curated gold-question set and a release-quality threshold before expanding connectors, tools, or domains. Treat unsupported requests as scope signals, not reasons to claim broad capability prematurely. | Caveat: Narrow initial scope must be paired with a visible expansion path; the speaker warns that users rapidly raise expectations once the first useful workflow becomes habitual.
  • Claim: A phased rollout should validate three distinct things: answer quality in pilot, MVP coverage and retention in beta, and organizational activation at GA. | Evidence: Snowflake pilots with AI-native users willing to give feedback, then runs a 10% beta of about 600 people to identify recurring missing data and workflow needs. It exited beta after more than 70% retention among weekly active users. | Implication: Set explicit stage gates for pilot accuracy, beta retention, recurring unmet-demand concentration, and GA activation. Do not equate a successful launch announcement with product-market fit inside the company. | Caveat: Question volume alone is insufficient; a high number of prompts can coexist with low repeat use or poor answer quality.
  • Claim: Enterprise AI adoption often fails because users never try the product, not because the technology is necessarily poor. | Evidence: Izmit distinguishes non-trial from churn: if only 20% of the organization has tried the assistant, he considers that an activation problem; if users try it and do not return, that becomes the product team's problem. He says adoption can take months and reports personally spending 60-70% of his time in sales meetings, demos, adoption dashboards, manager pressure, and leadership-sponsorship work. | Implication: Assign a named adoption owner and operating cadence alongside engineering. Track exposure, first-use activation, repeat use, and depth separately, and make team-level adoption visible to accountable leaders. | Caveat: Change management cannot rescue an agent that fails its first interactions; it follows rather than replaces quality discipline.
  • Claim: The value ladder for GTM agents progresses from answering questions to workflow automation, team-built capabilities, and hyper-personalized customer operations. | Evidence: Snowflake started with "talk to your data," then added MCP-based integrations for workflows such as monitoring inboxes and Slack customer questions, drafting Gmail replies for human review, and automating outreach. Izmit expects teams to next build their own skills, dashboards, applications, automations, and alerts, followed by personalized context for individual sellers and customers. | Implication: Design the roadmap around increasing operational agency, not a static chat interface: retrieval first, then supervised action execution, then reusable team-level workflows and persistent customer context. | Caveat: Automation examples retain a human review step for outbound email drafts, indicating that autonomy should be graduated rather than assumed.
  • Claim: Treat current agent architecture as disposable enough to exploit rapidly emerging capabilities instead of delaying launch for a perfect design. | Evidence: Snowflake launched to 6,000 users with nine pages of instructions, a few Cortex Analyst tools and semantic views, Cortex Search, and instruction versions managed in Google Docs. Its original PRD and architecture now differ from the current system by about 80%; 30-40% of sprint work is ongoing re-architecture while 60-70% is features and quality. | Implication: Avoid six-to-nine-month architecture programs for an uncertain agent workflow. Build a minimal governed foundation, plan engineering capacity for continual migration, and modularize instructions, tools, skills, and routing so each can change independently. | Caveat: Flexibility does not mean absence of engineering discipline: as scale and complexity grew, Snowflake added CI/CD, unit tests, routing tests, eval infrastructure, a skill library, and progressive disclosure.
  • Claim: Interaction logs are not merely observability data; they are the primary product, enablement, and cross-team intelligence feedback loop. | Evidence: At 1.2 million questions and about 40,000 per week, Snowflake uses LLMs to classify logs into topic and subtopic structures, identify repeated questions and user frustration, and surface feature gaps in real time. For a new product launch, it can use observed seller questions plus Confluence, Jira, Slack, and PRDs to generate battle cards or enablement documents in minutes and feed them back into the agent. | Implication: Build a governed telemetry taxonomy from day one: classify intent, outcome, repeat attempts, unresolved requests, and dissatisfaction signals; connect those findings to a content-production and agent-update workflow rather than relying on periodic user interviews. | Caveat: The transcript does not specify the privacy, redaction, retention, or evaluation controls used when processing potentially sensitive sales conversations and internal data.
  • Claim: Centralizing enterprise data and access controls makes no-code or low-code business agents safer and more deployable. | Evidence: Snowflake brought first-party, third-party, Salesforce, and call-transcript data into Snowflake so agents could inherit role-based access controls. The internal product, built as customer zero for Snowflake Cowork (formerly Snowflake Intelligence), uses out-of-the-box Cortex Analyst, Cortex Search, Cortex Science, chat UI, data-source curation, and security guardrails. | Implication: For an internal control plane, prioritize identity-aware data access and curated source boundaries before democratizing agent creation; the delivery UI matters less than enforcing the same permissions users already have. | Caveat: This reflects Snowflake's platform-specific approach and does not prove that a single data platform is necessary for every agent architecture.

Detailed Brief

Trust decay and the "collapsing wow factor"

  • Claims: User trust in a non-deterministic chat system is hard to earn and can be lost overnight after a few poor answers.; Initial delight is temporary: once users can talk to data or automate a workflow, that capability becomes a baseline expectation rather than a differentiator.; Because switching to another AI product is easy, an internal team must assume users will seek alternatives if its pace of improvement slows.
  • Evidence: Izmit's team uses the phrase "quality is p minus one" to emphasize that the first interaction quality is decisive.; Users initially praised the ability to avoid dashboard sprawl and analyst queues, but within a few months began asking why the agent could not perform functions they had seen in other AI products.; He recommends being "paranoid" whenever users are satisfied: plan what new value to deliver one or two months later.
  • Caveats: The talk makes a strong strategic case for continuous iteration but provides no quantified evidence connecting each capability stage to win rates, deal-cycle reduction, or incremental revenue.
  • Implications: Maintain a rolling capability roadmap and explicitly measure whether users are advancing from Q&A to repeated workflow execution rather than merely generating more chats.; Position agent delivery as an ongoing product commitment with a visible improvement cadence, not a one-time automation project.

Current system footprint and implications for agent composition

  • Claims: A mature GTM agent becomes a composite system of structured-data semantics, unstructured retrieval, external connections, skills, orchestration rules, and multiple interfaces.; Prompt/instruction files alone eventually cease to be a workable container for all business processes and tool-orchestration logic.
  • Evidence: The current Snowflake agent has 15 semantic views, 85 tables, 3,000 data columns, five to six MCP connections, and close to 20 skills.; When workflow logic no longer fit in agent instructions, Snowflake adopted a skill library; when added MCPs created new orchestration burden, it used progressive disclosure.; The roadmap includes user memory, task scheduling, and delivery beyond chat into Slack.
  • Caveats: The transcript does not explain the selection/routing policy among skills and MCP tools, nor does it provide benchmark results for the claimed eval infrastructure.
  • Implications: Separate declarative data semantics, retrieval, tool contracts, workflow skills, user-context memory, and delivery channels rather than accumulating all behavior in one system prompt.; As tool count grows, make routing evaluation and progressive context loading first-class reliability concerns.

Notable Concepts & Terms

  • Quality is p minus one: The team's shorthand for the idea that one poor early interaction can disproportionately destroy user trust in a non-deterministic agent.
  • Collapsing wow factor: The recurring pattern in which a newly impressive AI capability quickly becomes a user expectation, requiring continual delivery of higher-value capabilities.
  • Semantic views: Snowflake's structured-data layer used to make business data understandable and queryable by the agent through Cortex Analyst.
  • MCP connections: Model Context Protocol integrations that expand the agent from data Q&A into connected workflows and external tools.
  • Progressive disclosure: Loading relevant instructions or operational context incrementally, used after the agent's growing skill and MCP orchestration requirements exceeded practical prompt size.
  • Cortex Analyst / Cortex Search / Cortex Science: Snowflake components that provide structured-data analysis, unstructured-data search, and related AI functionality in the Cowork platform.
  • Snowflake Cowork: The renamed Snowflake Intelligence platform: a no-code agent environment with built-in UI, data tooling, role-aware access, curation, and guardrails.
  • LLM-classified logs: Using language models to categorize interaction data at scale, turning agent conversations into product-gap, enablement, and organizational-intelligence signals.

Operator Notes / Why Ken Should Care

  • Create a pre-launch evaluation corpus drawn from real GTM workflows; designate a smaller set of supported questions and set an explicit accuracy or acceptance threshold for them.
  • Instrument a rollout dashboard that separates eligible users, exposed users, first-time users, retained weekly users, workflow depth, unresolved intents, repeat prompts, and dissatisfaction language.
  • Require a beta exit review that combines retention, concentration of missing-data requests, safety/permission failures, and evidence that the agent fits a recurring workflow.
  • Allocate a dedicated post-GA adoption budget and owner; secure leadership sponsorship before launch rather than treating enablement as an engineering afterthought.
  • Architect agent behavior as versioned, testable components—data semantics, retrieval, skills, tools, routing, memory, and delivery channels—and reserve 30-40% engineering capacity for platform evolution.
  • Establish governance for interaction-log processing, especially redaction, permissions, retention, and safe use of sensitive sales/customer content, before using logs as a broad intelligence layer.
  • Build a closed loop from recurring unanswered questions to source-document creation, approval, ingestion, and regression testing so content updates improve the agent without introducing new failure modes.

Source/Metadata

  • Title: Building GTM AI Agents: Lessons from Deploying to 6,000 Users — Sait Izmit, Snowflake
  • Transcript words: 6155
  • Duration seconds: 1239
  • Timestamp note: No timestamps or chapters were present in the supplied transcript; the latter portion contains duplicated transcript content.
Search this video Resume transcript at 0%
Full transcript 3851 words · 27 min read
0:12

Hi everyone, so I think I'm one of the last speakers standing between you and the long weekend, so I hope I can get your energy levels up. I'm responsible for our internal AI tools for our sales team, and the reason I'm here today is indeed we launched our internal go-to-market assistant in September last year. It answered more than 1 million questions so far. We have roughly answered 40,000 questions a week, and we are the customer zero for all of Snowflake products, so this is built on Snowflake co-work, and I meet a lot of customers every week, okay? I meet a lot of enterprises, Fortune 500 companies, and they're all trying to build similar things, and they all struggle, right? Then I end up having this discussion with them all the time. They ask, how did you guys do it? And then we share our best practices. So I will try to share some of those things with you. I'm told that I need to have some code in my presentation. I don't, but I will try to show you at least some architectural diagrams, just to make it more interesting for the engineering audience. But let's jump into it.

0:16

Before we start, I think I already, I was watching the other presentations, I think everyone tries to give their interpretation of why are we even building things for go-to-market, okay? So this is how I explain it to family and friends. So let's take Snowflake, okay? We are a company of close to 10,000 people. So if you look at our organization, almost half of our workforce is sales, right? And what are they responsible for? They're responsible for revenue generation. What do they struggle with? And I interview, I talked to a lot of customers. It's very common. Everyone's data is siloed. We work with a lot of first-party data, a lot of third-party data. It's all locked down in these SaaS tools and things like that. And literally we have, for example, reps who are using 15 different tools, not because they love the UI of those tools, but because every tool has a different data point, and then they end up stitching all of that together in spreadsheets and running it there, right? And the data is endless. We have reps who have 1,000 accounts assigned to them, 1,000 customers. They have to stay on top of their recent news, what's happening with their consumption, did they get support tickets recently, what were their latest earning results, everything. It's not a single human on this planet can stay on top of that much data. And then they need to do that 30 times, 40 times a day, right?

0:22

So what does AI offer for them? It offers that data democratization, no more thousand dashboards, right? No more access to analysts. It offers automation possibilities for them, right? It frees up their inbox. It offers tool consolidation, no longer 15 different tools that I need to work for. And that brings productivity savings, it frees up your time, right? You can use that time on other things. It helps you become a better seller, you are more effective with your customers. And that translates to business results. You can cover more of your book, you can have better win rates, you can have shorter deal cycles, and ultimately what everyone cares about, you can get incremental revenue. Okay, so that's the reason why I'm working for making the go-to-market organizations more effective.

0:28

But there's a catch. These are non-deterministic systems, right? And I run into this problem every time with users. I see many, many, many AI projects fail, fail, and then it fails on this principle. User trust is earned extremely hard, and it's lost overnight. Right? So at the end, what you are doing is you are putting a free-form chat box there. Right? And people will come in and they will ask any question they can think of. If they like what they see in the first five questions, they come back. If they don't like what they see, it's 10 times more effort for you to win them back, if you can ever win them back. Right?

0:35

So we have a saying in our team. We say quality is p minus one. Right? And basically, we take that very, very seriously. So one of the things that we really cared about is when I first joined the team, the team had all these data sources connected from our top dashboards. We had a knowledge assistant built into it and so on. We had three lines of agent instructions. And then before I even tried the agent, I opened the spreadsheet. I took the sales process. I wrote down 150 questions. And then our engineering team was like, what are you doing? We don't have that data in the agent. I was like, it doesn't matter. These are the questions your sellers are going to ask. Right? And then we ran our tests, 50% accuracy, everyone's depressed and so on. So we said, okay, let's make sure that we don't go for coverage, but we go for quality. Right? We don't want to try to answer 100 questions and get them 70% right. We want to answer 50 questions, but get them 95% right. Right? Because with that, you get a good first impression, you build the trust with them. And then rather than being in that boat of, oh, this thing doesn't work, people are like, oh, this thing is awesome. Can I get more of that? Right?

0:41

So we started small, and 60% of the data we actually added after the launch, after the six, seven months post-launch. Today, if you look into our agent, it's not a small agent. We have 15 semantic views, 85 tables, 3,000 columns of data. We have five to six different MCP connections on it, close to 20 skills connected to that, and so on and so on. Right? So it's a huge system that we are managing here.

0:48

And then you cannot just launch these things to everyone. Right? So we said that, look, we need to do this in a controlled way because we want to make sure that we earn that first five questions. We don't want to burn our bridges in that first five questions. Right? So that's why with every product we do, we do a phased launch. The first one is a pilot. The goal of the pilot is to prove the accuracy, prove the quality. Right? You get your top AI-native folks in the organization who are eager to work with you, give you feedback, improve the product. Make sure that you got the rough edges through that. Right? And then after a couple of weeks, you come to a point where it looks like, okay, those rough edges are smoother now. Okay? Then you go into your beta launch. We do a 10% beta, right, with 600 people. There you are looking at, do I truly have a minimum viable product? Is the MVP really there? Right? And what will happen is that you will start getting tons of requests. Can you connect this data? Can you connect that data? And everything. And then you are looking at where actually the concentrations are happening. Because that means that if you don't get those things in, you don't really have an MVP. Right? Then it's not going to work for their daily workflows.

0:53

And then at that stage, you're also trying to prove, are they coming back? Right? So the things that we really track there are basically, okay, how many questions they're asking and everything, but what is the retention rate? Right? So we exited, for example, that at more than 70% retention rate, that the weekly active users were coming back. Okay, now we're in a good place. Right? We have confidence in the accuracy. We have confidence in the coverage of the product we have, and people are coming back. Okay, now let's go to GA, and then you launch to the GA.

0:57

But then you have your next problem. So I know that this is a technical conference, but this is also where a lot of these products fail. It's basically how do you drive change management. So you launch your product, you are two weeks into the launch, and then you are here. And all your management is disappointed or frustrated. Why aren't people using this? Why are the numbers very low? Right? And I show them this graph. I say that only 20% of your organization actually tried the product. I cannot do anything. It's not the product's fault if people are not even taking five minutes to try the product. Right? If they try it and if they don't come back, okay, that's my problem. Right? But if they don't try it, then we have another problem.

1:02

Okay, now let's go to GA, and then you launch to the GA. But then you have your next problem. So I know that this is a technical conference, but this is also where a lot of these products fail. It's how do you drive change management. So you launch your product, you are two weeks into the launch, and then you are here. And all your management is disappointed or frustrated. Why aren't people using this? Why are numbers very low? Right? And I showed them this graph.

1:18

I say that only 20% of your organization actually tried the product. I cannot do anything. It's not the product's fault if people are not even taking five minutes to try the product. Right? If they try it and if they don't come back, okay, that's my problem. Right? But if they don't try it, then we have another problem.

1:21

So the first, and I've seen this with many, many sales organizations in my past life as well, and so on. Usually this is a couple of month process. And then you significantly invest in change management, in activation. I will spend 60, 70% of my time in sales meetings, giving demos, building dashboards to see which teams adapted, shaming the managers whose team is actually doing good, getting sponsorship from sales leaders to make sure that they push their people to try these things, and so on. And then ultimately that gets your blue line up, and then your questions start coming up, and then your focus can shift into, okay, how do I drive more depth? Right?

1:33

And I want to really, really emphasize this because if you hadn't done this, we would probably be doing half of where we are today. So it's a very, very important part. And then as engineers, if you spend all your effort, you want to have a good product, make sure that the activation and the change management is lined up post-launch of the product as well. Now we run into another issue.

1:49

Okay, let's say that you are four to six months down the run, right? What happens is you successfully launch the product. You are first rock stars in the company, right? People literally show you in the corridor, hey, your product is awesome. We can talk to our data now. We don't need to wait in the queue to get access to analysts to answer our questions every two weeks, and so on. Right? And after a couple of months, they start coming back to you with frustrations. Hey, I cannot do this in the product anymore. Right? I would like to, I saw this other AI product that does this, and so on. This is what I call the collapsing of the wow factor. Okay?

1:58

So initially you are cool, but then that becomes a habit, right? You change their habit, and it becomes standard for them. Now you need to raise the bar again. So the journey that we usually see with the sales teams is you start with talk to your data. How do we get you out of those hundreds of dashboard situations, dependency to the analyst? And then first we democratize the data for you so that you can talk to your data. Then the next wave comes with all the MCP connections. Right? All the integrations that you are building. Now it becomes automate my workflows.

2:07

We literally have now sellers who are going to use our agent to monitor their inbox. They monitor their Slack channels, keep track of all the customer questions coming about product questions, use the agent to draft responses, save that in Gmail, review them afterwards, send those things out. Right? Or they automate their outreach workflows, and so on. Okay, that's great. Now I became an orchestrator. Right? I'm automating my workflows.

2:17

Now the next thing you see start happening is teams get this tool democratization, this empowerment coming to them. Right? Because historically, a lot of these go-to-market teams have always been in the backlog of someone. Backlog of an IT team or trying to get a SaaS budget to learn and get a vendor on board to actually enable something. And now all of a sudden, they're able to build team skills. They're able to build custom dashboards that are fully optimized for what their team needs, are able to deploy applications, automations, alerts, and things like that. Right?

2:26

And then there comes a phase of hyper-personalization. Right? Everyone is able to now get everything personalized for them, not only for themselves, but also for their customers with living context of customers, contacts, and things like that. I think the main message I want to give here is if you just do the first stage, and if you just wait there, you will get disrupted in a month or two. Right? Because now you already raised their expectations, that already became a baseline, and then they will find another product that does better than you. And right now, the switch is very easy. They're going to just switch overnight. Okay?

2:38

So you need to keep iterating. You need to keep that wow factor. And then you cannot just rely on the fact that what I built so far is going to stay cool forever. And the next thing is how they deal with the changing technology. So I talk to a lot of customers, and then sometimes you run into these customers, big enterprises, very big brands, and then they are still trying to purchase that perfect architecture. They are trying to test different frameworks. They are trying to see how the technology is maturing and everything, and so on. But the thing that they don't do is they don't build, and then they don't launch, and they don't learn.

2:48

All these blue boxes that you see here, those are all the things we added after the launch. Right? When we literally first launched the agent, it was a nine-page-long agent instructions. It was a couple of Cortex analyst tools, semantic views. It was a Cortex search service for our unstructured data, and we were managing the agent instruction versions out of a Google Doc. That's how we launched it to 6,000 people. Right?

2:54

Now we realize, okay, it's not going to work out. Let's figure out CI, CD. It's not going to work out. Let's figure out our eval infrastructure with all the unit tests, routing tests, and everything. Right? Then we start coming to a point where, for example, we were creating all these business processes and workflows. We couldn't fit them into the agent instructions anymore. And then the skills came, and we were like, oh, perfect. Let's build a skill library. Then the MCPs came. Perfect. But now we had to put a bunch of other instructions to orchestrate that. We hit the limits on the agent instructions. What do we do? Okay, let's do the progressive disclosures. Right?

2:58

And then user memory comes. Task scheduling comes. We want to go beyond the chat screen and chat interface and start doing the Slack interface and things like that. If I look at the PRD and the architectural diagram we wrote in the beginning of the project, if I compare it to this architecture we have now, 80% of it doesn't match. Okay? So if you look at our sprints, maybe 60%, 70% of the work we are doing is adding new features, improving quality, and all kinds of things. But 30%, 40% of the work is that we are constantly re-architecting with the new technology.

3:08

So this is a time where you need to get your hands dirty, you need to run with the new technology, and then you shouldn't be too much tied to your architecture. You should be okay to pivot very easily so that you can double down on these new capabilities and things like that. And then the longer you wait, the more you lose to your competition, because if your competition is doing these kind of things three, four months ahead of you, right, that means that they're also getting more customers.

3:20

Last thing is I would really, really recommend investing in your logs, okay, because they create the feedback loop. So first of all, technically, it's very fun, okay, so you use LLAMs to classify your logs and things like that. As I said, we have 1.2 million questions. We get 40,000 questions every week. It's technically very fun how you do that at scale without breaking the bank, and so on. Our data scientists love working on those things, and then they really experiment with new things. But as a result of that, what we get is a very extremely detailed breakdown of topics and things that we are having. I'm just showing you the top categorization level there.

3:36

Last thing is I would really, really recommend investing in your logs, okay, because they create the feedback loop. So, first of all, technically, it's very fun, okay, so you use LLAMs to classify your logs and things like that. As I said, we have 1.2 million questions. We get 40,000 questions every week. It's technically very fun how you do that at scale without breaking the bank and so on. Our data scientists love working on those things, and then they really experiment with new things. But as a result of that, what we get is a very extremely detailed breakdown of topics and things that we are having. I'm just showing you the top categorization level there.

4:33

But then we are able to track what kind of questions they are asking. We are able to break down each of those categories. The subcategories, we are able to get detailed example questions, this and that, and so on. All good, but how do we use that? Then we start creating the feedback loops, right? I know I still interview, of course, users, but now I see in real time what my feature gaps are. I'm clearly seeing what people are asking and what we are not able to answer, or where we have a quality issue where they're swearing at the agent or repeating their question so that we see where to improve. For sales enablement, it's a goldmine.

5:12

Let's say that we launch a new product. Usually, they would need to interview maybe 100 sellers a week to be able to understand how the topics are changing, where there's gaps in terms of knowledge documents, battle cards. I see that in real time in a minute or two by just asking another question. And then we can connect to Confluence. We can connect the JIRA. We can connect the Slack channels. We can ingest the PRDs. And in a couple of minutes, we can actually generate battle cards, sales enablement document, and then feed it back into the agent, right? You cannot do that kind of feedback loop with humans, right? So then we can automate these kinds of things.

6:09

Within the sales organization, there will be different teams that are trying to connect each other. They're trying to maybe target similar accounts from different angles. They don't know about each other. We do. We are now able to ping them. And then we are able to do matchmaking, right? I'm just giving you a couple of examples. But this is also one of those areas where you start building your AI platform. You start building your architecture. The first features are difficult to get out. The next ones are easy. And then once you start tapping into your logs, this hockey stick exponential thing actually starts happening, and it's magical.

7:02

So if you were to take a couple of things from this talk, quality over coverage. I'm very, very religious about this. If you go for the coverage, you are going to shoot yourself in the foot, okay? Change management. A lot of engineers don't think about this, right? A lot of these AI initiatives don't fail because there's an issue with the technology. There's an issue with that, assuming you did the first one, right? So they fail actually in activation. So make sure that you have a plan for that, especially in larger organizations where we are dealing with 6,000 go-to-market users, right? Again, don't forget this concept of collapsing wall factor.

7:50

You cannot stay where you are. You cannot just say that, hey, I did an innovation. I'm going to surf that for a year. Every time people are happy, you should be paranoid. You should be like, okay, what am I going to show them in a month or two now? How do I keep that excitement going on? Right? Build fast with today's tech. Don't try to invest in these super architectures and have these six, nine months of long projects and things like that. How do you turn around these things in weeks, days, and so on? And just be comfortable with the fact that you are constantly going to be re-architecting. That's fine, right? Just don't over-invest in the current architecture.

8:54

Just make sure that you keep your flexibility out there. And then the feedback loops. I think that's what gives you the incremental part of that hockey stick exponential part of the thing. We constantly publish blog posts where we try to have our learnings shared with our team, customers, and so on. We have blog posts on how we do agent instructions. How we do our structured data with semantic views. How we build our rack-based knowledge assistance. The non-technical side of the story, like how do you drive change management and so on. So feel free to check those. And, yeah, I think that's the end of my talk. Okay. We have time for one question. Okay. There you go.

9:33

Thanks for the talk. I don't know if you already said this, but I saw in the titles of the articles, Snowflake Intelligence. Is that an underlying context or layer that the tool or system you built was on top of? Or was that the tool itself or something else? Yeah. Snowflake Intelligence, we renamed that the Snowflake Cowork a couple of weeks ago in our summit. That's our no-code agent platform that we have available for our business users. The advantage of that is that a lot of these tools around cortex analysts or cortex search or cortex science, a lot of those things come out of the box.

9:59

We made a strategic choice for our internal thing where we said that, look, it is important that we bring all our data together. And we do that in Snowflake. We bring all the first-party, the third-party data, all the Salesforce data, everything, the code transcripts and so on, all together. And then these agents can inherit a lot of the role-based access controls and so on. And then literally you can deploy these agents without writing a single line of code, right? And then you don't need to worry about the UI. The chat UI comes out of the box and so on. And then we have been the customer zero of that internally to build this ourselves.

10:22

And then our customers are able to go and build similar things basically on Snowflake Cowork platform as well. And it comes with the guardrails and things where you don't really need to worry about them going very crazy on what data sources to do things and so on. So we are able to do a lot of curation. We are able to do a lot of security guardrails on there as well. Thank you. Thank you. Backlog of an IT team or like trying to get a SaaS budget to learn and get a vendor on board to actually like enable something. And now all of a sudden, they're able to build team skills.

10:50

They're able to build like, you know, custom dashboards that are basically like fully, you know, optimized for what their team needs. Are able to like deploy applications, automations, alerts, and things like that. Right? And then there comes a phase of hyper-personalization. Right? Everyone is able to now like get everything personalized for them, not only for themselves, but also for their customers with living context of customers, contacts, and things like that. I think the main message I want to give here is if you just do the first stage, and if you just wait there, you will get disrupted in a month or two. Right?

11:25

Because now you already raised their expectations, that already became a baseline, and then they will find another product that does better than you. And right now, the switch is very easy. They're going to just switch overnight. Okay? So you need to keep iterating. You need to keep that wow factor. And then you cannot just rely on the fact that, you know, what I built so far is going to stay cool forever. And the next thing is how they deal with basically the changing technology.

11:49

So I talk to a lot of customers, and then, you know, sometimes you run into these customers, big enterprises, very big brands, and then they are still trying to purchase that perfect architecture. They are trying to like test different frameworks. They are trying to see how the, you know, the technology is maturing and everything and so on. But the thing that they don't do is they don't build, and then they don't launch, and they don't learn. All these blue boxes that you see here, those are all the things we edit after the launch. Right? When we literally first launched the agent, it was a nine-page long agent instructions.

12:22

It was a couple of Cortex analyst tools, semantic views. It was a Cortex search service for our unstructured data, and we were managing the agent instruction versions out of a Google Doc. That's how we launched it to 6,000 people. Right? Now we realize, okay, it's not going to work out. Let's figure out CI, CD. It's not going to work out. Let's figure out our basically eval infrastructure with all the, like, the unit tests, routing tests, and everything. Right? Then we start basically, like, coming to a point where, for example, we were creating all these, like, business processes and workflows. We couldn't fit them into the agent instructions anymore.

12:53

And then the skills came, and we were like, oh, perfect. Let's build a skill library. You know, then the MCPs came. Perfect. But now, like, we had to put a bunch of other instructions to basically orchestrate that. We hit the limits on the agent instructions. What do we do? Okay, let's do the progressive disclosures. Right? And then user memory comes. Task scheduling comes. We want to go beyond the chat screen and then, you know, chat interface and start doing the Slack interface and things like that. If I look at the PRD and the architectural diagram we wrote in the beginning of the project, if I compare it to this architecture we have now, 80% of it doesn't match. Okay?

13:27

So, like, if you look at our sprints, like, maybe 60%, 70% of the work we are doing is adding new features, improving quality, and all kinds of things. But 30%, 40% of the work is that we are constantly re-architecting with the new technology. So, this is a time where, like, you need to get your hands dirty, you need to run with the new technology, and then you shouldn't be, like, you know, too much tied to your architecture. You should be okay to, like, pivot very easily so that you can basically double down on these, like, new capabilities and things like that.

13:55

And then the longer you wait, the more, you know, you lose towards your competition, because if your competition is doing these kind of things, like, three, four months ahead of you, right, that means that they're also getting more customers.

14:08

Last thing is I would really, really recommend investing in your logs, okay, because they create, basically, the feedback loop. So, first of all, technically, it's very fun, okay, so you basically use LLAMs to, like, classify your logs and things like that. As I said, like, we have 1.2 million questions. We get 40,000 questions every week. It's technically very fun, you know, how you do that at scale without breaking the bank and so on. You know, our data scientists love working on those things, and then they really experiment with new things.

14:36

But as a result of that, what we get is we get a very extremely detailed breakdown of topics and, you know, things that we are having. I'm just showing you the top categorization level there. But then, basically, we are able to track, like, you know, what kind of questions they are asking. We are able to break down each of those categories. The subcategories, you know, we are able to get, like, detailed example questions, this and that, and so on. All good, but how do we use that? Then we start creating, basically, the feedback loops, right? I know, I mean, I still interview, of course, users, but now I see in real time what my feature gaps are.

15:09

I'm clearly seeing what people are asking and we are not able to answer or where we have, like, a quality issue where they're swearing at the agent or, like, repeating their question so that we see where to improve. For sales enablement, it's a goldmine. Let's say that we launch a new product. Usually, you know, they would need to interview maybe 100 sellers a week to be able to understand, like, how, basically, the, you know, the topics are changing, where there's gaps in terms of, like, knowledge documents, battle cards. I see that in real time in a minute or two by just asking another question. And then we can then, you know, connect to Confluence.

15:41

We can connect the JIRA. We can connect the Slack channels. We can ingest the PRDs. And in a couple of minutes, we can actually, like, you know, generate battle cards, sales enablement document, and then feed it back into the agent, right? I mean, you cannot do that kind of, like, a feedback loop with humans, right? So then we can basically automate these kind of things. Within the sales organization, there will be different teams that are trying to connect each other. They're trying to maybe target similar accounts from different angles. They don't know about each other. We do. We are now able to ping them. And then we are able to basically do matchmaking, right?

16:12

I'm just giving you a couple of examples. But this is also one of those areas where, like, you start building your AI platform. You start building your architecture. The first features are difficult to get out. The next ones are easy. And then once you start tapping into your logs, this, like, hockey stick exponential thing actually starts happening, and it's magical.

16:34

So if you were to take a couple of things from this talk, like quality over coverage, I'm very, very, like, religious about this. If you go for the coverage, you are going to shoot yourself in the food, okay? Change management. A lot of engineers doesn't think about this, right? A lot of these AI initiatives, they don't fail because there's an issue with the technology. There's an issue with that, assuming you did the first one, right? So they fail actually in activation. So make sure that you have a plan for that, especially in larger organizations where we are dealing with, like, 6,000 go-to-market users, right?

17:12

Again, don't forget this concept of collapsing wall factor. You cannot stay where you are. You cannot just say that, hey, I did an innovation. I'm going to surf that for a year. You know, every time people are happy, you should be paranoid. You should be like, okay, what am I going to show them in a month or two now? How do I basically keep that excitement going on? Right? Build fast with today's tech. Like, don't try to invest in these, like, you know, super, like, you know, architectures and have these, like, six, nine months of long projects and things like that. How do you turn around these things in weeks, days, and so on?

17:46

And just be comfortable with the fact that you are constantly going to be re-architecting. That's fine. Right? Just don't over-invest in the current architecture. Just make sure that you keep your, like, flexibility out there. And then the feedback loops. I think that's what kind of, like, gives you really that, like, you know, the incremental part of, like, that hockey stick exponential part of the thing. We constantly publish, like, blog posts. I mean, where we kind of, like, try to have our, you know, learnings shared with our team. Customers and so on. Like, we have blog posts on, like, how we do agent instructions. How we do our structured data with semantic views.

18:19

You know, how we basically build our rack-based, like, knowledge assistance. The non-technical side of the story. Like, how do you drive change management and so on. So feel free to check those. And, yeah, I think that's the end of my talk. Okay. We have time for one question. Okay. There you go.

18:47

Thanks for the talk. I don't know if you already said this, but I saw in the titles of the articles, Snowflake Intelligence. Is that an underlying context or layer that the tool or system you built was on top of? Or was that the tool itself or something else? Yeah. Snowflake Intelligence, we renamed that the Snowflake Cowork a couple of weeks ago in our summit. That's basically our non-code agent platform that we basically have available for our business users. I mean, the advantage of that is that a lot of these tools around, like, you know, cortex analysts or cortex search or cortex science. A lot of those things are basically comes out of the box.

19:26

We made a strategic choice for our internal thing where we said that, look, it is important that we bring all our data together. And we do that in Snowflake. We bring all the first-party, the third-party data, all the Salesforce data, everything, the code transcripts and so on all together. And then these agents can basically inherit a lot of the role-based access controls and so on. And then literally you can deploy these agents without writing a single line of code, right? And then, you know, you don't need to worry about the UI. The chat UI comes out of the box and so on. And then we have been the customer zero of that, like, internally to build this ourselves.

19:58

And then, you know, our customers are able to go and then build similar things basically on Snowflake Cowork platform as well. And it comes with the guardrails and things where you don't really need to worry about them going very, you know, crazy on, you know, what data sources to do things and so on. So we are able to do a lot of curation. We are able to do a lot of security guardrails on there as well.

20:21

Thank you.

20:34

Thank you.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note