Crack This Interview to Reach $500K as an AI PM

Transcript

Product design is dead. And that's not really clickbait, but it's actually from some of my recent AI PM interview experiences. [SPEAKER_01] AI product managers are making millions of dollars. At OpenAI, the average stock grant per employee is $1.5 million. At companies like Google and Meta, employees are getting huge stock grants. [SPEAKER_00] I was not really asked any of those conventional make-up-rich-for-blind-people kind of questions. It has moved to AI system design, where they are not just testing product sense, but they're also trying to test your system design knowledge. If you want to get one of the top AI product management roles at companies like OpenAI or Google or Meta, and you want it to be at the high end of the pay, the $1 million plus roles, the roles that have $1.5 million in yearly stock grants, but not just anyone can land these high-paid AI product management roles. [SPEAKER_00] So with Akash, we'll walk you right through how a great response looks, and we hope you benefit the most of it. So without any further ado, let's get right into it. When it comes to the AI system design interview, they're looking for your ability to go deep on a technical topic. So we're going to show you how to ace this with the mock interview question: Build the system design for a churn reduction agent. So we need to build a churn reduction agent. Am I right? I would like to think before we jump into solutioning, I have some clarifying questions. Is it fine if I ask some of them? [SPEAKER_01] Yeah, go for it. I would actually like to understand. I know what churn generally means, but in this specific scenario, does it have any specific meaning or any relation to any kind of product or anything which you would like to give some detail about? [SPEAKER_01] Yeah, churn, you can define it. I think that at the high level, there's an engagement churn that we're going to see, which is they use the product, they tried it, and then their engagement drops off. And then the lagging thing we'll see is they stopped paying us. All right. So we don't have a more precise input for this interview. [SPEAKER_00] Sounds good. [SPEAKER_00] Also, with regards to the scope of this, are we talking about users of any specific domain, like mobile app versus desktop app, or is there anything there which needs to be understood here? I guess we want to consider all platform behavior. The important thing to think about is what is going to help us secure more revenue? And if we can get signals about mobile versus desktop usage and it's relevant, then I think you should focus on it. [SPEAKER_00] Got it. Cool. [SPEAKER_00] Are there any constraints in regards to time or the effort or budget we have to launch this feature or product? [SPEAKER_01] I would say think about the best case scenario. What would be the ideal thing to design? And don't worry too much about constraints for this interview. Got it. Cool. Okay. So no constraints. I'll try to keep it to six months, if not more. But finally, I know the goal is churn reduction. Do we have any other goals associated with the product apart from churn reduction? That's a good question. I guess just to build a good system. Think about it. It's like its own code base, maybe its own repo. So how are we going to build it specifically? The technical areas is what I'm most interested in. [SPEAKER_00] All right. Okay. Cool. So the reason I ask is because there are a lot of things which a company does because of its competitors. So if our competitors are trying to do X, Y, Z, is that one of the reasons we are trying to do this? Or there is no independent, there is no dependent correlation is what I'm trying to deduce from it. But it looks like in your case, we just need to build an independent system and we don't really have any other goals apart from churn in a nutshell. Yeah. I mean, just the normal reasons, right? Okay. If we can identify churn, we can take actions in the product to improve engagement. We can secure more revenue. Generally tends to be more profitable for us than focusing on acquisition. So we have an overall push there. [SPEAKER_00] Sounds good. Cool. [SPEAKER_00] And do you have any suggestions as to what kind of an industry product I should go towards or is that up to me to decide which product to pick for this? [SPEAKER_01] Just consider software. Okay. Cool. Okay. Sounds good. So I'll just take a minute. I'll try to define the product and the vision I have. And then I'll just get back to you in about 30 seconds, if that's fine. [SPEAKER_01] Sure. Take your time. So I think with regards to the vision, as you're speaking more about Agente KI, customer care is something which we are trying to, as an industry, not just telecom industry, but even other industries, we're trying to use Agente KI to make our purpose more efficient. So I think what I'll do is I'll try to take an example from telecom nations industry because I think it will be easier for people to understand. And also in terms of the product, I'll try to take a mobile app, which is, Customer care is something which we are trying to, as an industry, not just telecom industry, but even other industries, we're trying to use Agente KI to make our purpose more efficient. So I think what I'll do is I'll try to take an example from telecom nations industry because I think it will be easier for people to understand. And also in terms of the product, I'll try to take a mobile app, which will have the necessary Agente KI pipeline built in, which effectively should help us produce churn. But this is my vision for the product. Is there any question before I jump into the next section or what are your thoughts on this? Sure. Yeah. Makes sense to focus it in a specific real life, more textured situation. Cool. Okay. So what I'll do now is I'll start talking about some of the users. I'll try to prioritize one of them and I'll go towards the pain points. And finally, I'll move towards solutioning where we'll discuss about the different AI solutions. And finally, we'll jump into system design in terms of the diagram as well as some of the matrix we can track. Does that sound good? [SPEAKER_01] Yep. Okay. I'll just take 40 seconds. I'll get back with some of the users we can target. So I think with regards to users of any of, of not just this kind of product, but any kind of product, there are different ways how you can segment users. What I would like to potentially do is that I will actually, I'll try to speak to our users in terms of when did they join us. So there are a lot of new users. There are obviously power users. There are users who also focus from B2B perspective. A telecom company has B2B users as well. There are, so I think I'll try to stick to different types of users. Broadly, I think because we want to prioritize to only one user, I will try sticking to power users. And the reason is because these are the users we want to take care of the most. I'm not saying not all users are important, but in any product, the users who are mostly engaging with you, who are most deriving value for you and out of your product, you want to protect them and you want to ensure that their interests are well met. So we do not want to lose them. So from a churn point of view, I think I want to prioritize this segment of power users and proceed from there. Do you have any question, Akash, before I proceed to pain points? [SPEAKER_01] Yeah, I think power users make sense with our broader strategy anyway. So great. Before jumping into pain points, what I would actually, what I like to do is also something called user journey. Pain points are often derived through the different phases of user journey. So I'm just going to spend a few seconds to define what a user journey of a power user might look like. So we can understand that this user already has been part of this network for a while and has been using our app and customer care for a while now. And one thing I definitely know is that there are definitely going to be a lot of, there might be a lot of customer issues. So this user calls or contact customer care through own tries to get a ticket for this user for the problem. Rather, once that happens, the customer normally waits, reaches out again, if required, to the customer care through email maybe. Once issue is resolved, normal system hour is restored and the user effectively starts using the app as well now. Now on the app, now on the telecom app, you can do a lot of things, but just to not take much time, there are a lot of benefits provided on the app these days for our users. So he can browse benefits and also he can actually try to get Wi-Fi of the same company or avail other services. So these are some of the user journey items which I feel is very common among our telecom, our top telecom providers. And there are a lot of pain points among these. Firstly, I think user call, contact, I mean, everyone hates calling customer care. You do not want to spend your time calling customer care if especially you are a power user. So I think one pain point is definitely huge amounts of time and effort taken for each of these customer interactions, customer care interactions rather, which also ultimately causes anxiety, which also causes a lot of frustration, delay. And I think it overall, it doesn't really leave a good mark on the user's experience. You do not want to spend your time calling customer care if especially you are a power user. So I think one pain point is definitely huge amounts of time and effort taken for each of these customer interactions, customer care interactions rather, which also ultimately causes anxiety, which also causes a lot of frustration, delay. And I think it overall doesn't really leave a good mark on the user's experience. With regards to pain points, huge amount of time and effort for customer care. Apart from this, there is also a lot of customer pain points around the app experience. So when you open the app, you might not be able to track, sorry, you might not be able to track your request across customer call and app. So that causes a lot of confusion as well. Was my problem ever filed? What did I do wrong? Do I need to call the customer care again? And this is something which we definitely do not want to do. Finally, I think one pain point is also that regarding benefits. There are a lot of users with different interests and we do not want, if you like dining, we do not want you to unnecessarily show a benefit for Wi-Fi. So I think there are a lot of irrelevance. Irrelevance becomes a huge problem in terms of benefits and services. So we are not able to tap in the right value for the right user and that causes also friction, loss of potential, upsell, which otherwise could have happened. So I think these are some of the pain points. I know there could be a lot, but these are some of the pain points I'm looking at. Before I prioritize, do you have any questions on any of these? Maybe? Yeah, I think these are good ones to focus on for this particular system design. [SPEAKER_00] Okay. With regards to prioritization, I like to consider some of these things. Firstly, alignment to my vision, original vision, because there are a lot of problems. I do not want to focus on problems which do not align with my companies or with my product's vision. Apart from that, it is also frequency and impact. So what is the frequency of the pain point? How much impactful or how much problems is it causing for my consumer? I think with regards to that, I definitely would rate this as a very highly recurring problem with huge impact in terms of user experience. So I'll just try to mark it as high for all three domains. Apart from this, I do understand that inconsistent experience is a big problem. But I feel that even if we do solve your problem at the end of the day, it might not matter as much in the short term. So we might go with mid here and not high for some of these because it doesn't directly align with our immediate vision which is to create a highly engaging product and reduce churn. So I might just go medium here. If irrelevance in benefits causes friction, loss of potential and upsell. I feel this is again very important that we surface relevant benefits and services but that is more to do towards making our user experience highly engaging which I do want to do. But when it comes to churn reduction, I feel that it might again not be our top lever in terms of reducing churn at this moment. So I'll again try to shift towards medium for this. So I'll try to prioritize only one pain point at this time which is this one and I want to try to devise an agentic solution to resolve this kind of pain point in our system. What do you think? Any thoughts there before I move into solutioning? How do we... One of the things we want to solve for in the system design is just giving our product teams the right signals that hey this user is about to churn so we need to do something, take an intervention. So how do we get that early warning signal? [SPEAKER_00] Exactly. So I think that that's a very highly proactive system. [SPEAKER_00] Before I move into solutioning? How do we... One of the things we want to solve for in the system design is just giving our product teams the right signals that hey this user is about to churn so we need to do something take an intervention. So how do we get that early warning signal? [SPEAKER_00] Exactly. So I think that's a very highly proactive system which we want to build so that before the user calls for you to churn before three to four calls or before three to four what do you say or even types of interactions we are able to predict. So that is something which I'll be extending more in my solution section. So if it's fine I can... Now I think that's a good segue to a solutions section. I'll just take 30 seconds and I'll try to jot down all the different solutions possible if that's fine. Akash. [SPEAKER_01] Sure. [SPEAKER_00] So I think with regards to solutions there are a lot of different solutions possible to this one problem. I think one of them is a part which is pretty common where the system effectively tries to understand from your past called transcription and accordingly helps you guide your solutions very quickly but I feel that that might not be as engaging as possible because a human always prefers a more human-like touch. There is also something around voice port and I feel voice port might really be a very good scenario in this where you're not just able to talk to the AI agent but you're able to show your screen to it or able to actually help it exactly end-to-end guide you and not just redirect you to a human agent. So I think an end-to-end AI voice port assistance would be something amazing. In terms of a more basic solution which might not require AI as such we can make our experience more gamified in the app to just show the features and benefits in a way that the user always drives to avoid churning just because they keep getting some of these benefits and features but I feel that because at the core we want to improve the user experience I would like to go and try to build more of a voice port agent which tries to not just speak to you but actually tries to predict from your past data like you mentioned Akash when the user is going to maybe be in that red zone of churning and accordingly tries to activate that system of maybe sending more offers maybe trying to send more benefits to them and also trying to talk to the user end-to-end and solve the problems as soon as possible so this is what I would like to prioritize if that's fine Akash to you or I'd love to know your thoughts It's going to be a big project I don't know six months will depend really on the resources right but as I said we can think about a best scenario yeah Yeah I think no that's true I think six months is a big time but I feel that we can definitely map out an MVP version of it in six months and from there maybe craft a very good agent in maybe nine to ten months [SPEAKER_01] scenario [SPEAKER_01] yeah yeah I think no that's true. I think six months is a big time, but I feel that we can definitely map out an MDP version of it in six months, and from there maybe craft a very good agent in maybe nine to ten months. But yeah. [SPEAKER_01] Sure, what do you think? Let's see what it would look like. [SPEAKER_00] Okay cool. So I think before jumping into our system design diagram, I would like to just talk about different layers of an AI system today and what this point might actually have. So actually go to a new page. Okay, so now that we are targeting this maybe okay. So I think each AI agent or bot has potentially three things. I think one is obviously model, data, memory. These three things are the pillars of any AI system. And in terms of model, to be very honest, you already have some of the best AI models available at your disposal. And we could accordingly do a competitive analysis and basic testing on an engineering level and see which bits are use case. For example, for a voice would you prefer maybe an opus versus and a GPT-5 is a question which you can always try to understand, do some benchmarking, and come to that conclusion. But overall, I think we don't need to spend much time on the model right now. And you can, all the best models are at disposal. So I'm just going to park this as for your use case. In this case it is voice pod. So I might actually go to maybe Gemini because I feel that has a great voice experience. But again, it can be anything based on how your engineering team or ML team perceives this. Now I think coming to data, this is probably the core of it, right? Without data, your very powerful model is completely useless. In terms of data, you definitely need call transcription, right? That's the most basic thing you can provide. But you also need different signals. For example, app usage. For example, your network status through the past, let's say 12 months. Your what has been your not search history, but what has been your connectivity with regards to maybe the different competitors. Are you, for example, if you're traveling a lot, are we able to provide you those benefits of traveling which our competitors are providing? Or what kind of a user are you? So I think having a lot of different data points always helps you understand what kind of user are we dealing with. [SPEAKER_00] Traveling a lot, are we able to provide you those benefits of traveling which our competitors are providing, or what kind of a user are you? So I think having a lot of different data points always helps you understand what kind of user are we dealing with, and accordingly I think putting them, putting all these in a churn bucket would be helpful. For example, if you're a high risk churner, then maybe put it in the red bucket, or maybe if you are not, maybe in a green bucket. So we can, I think the model can effectively try to use these data points and then predict and put you in that bucket so that accordingly we can try to understand you. Also, I think we would also need some last minute retention offer. So let's say if you are in a red bucket and an AI agent might, let's say, is unable to resolve, we need some sort of a last minute retention offer just for you so that we can actually send it, or the AI agent rather can send it to you. I think these are also highly crucial. Finally, jumping into memory, I think in terms of memory there are different types of memory available. There is episodic memory. There is also session memory. So I think we need to understand that we definitely can, we do not want to store anything and everything, so we need to be really efficient here as well. So I would say that an episodic memory is highly important. What do I mean by episodic memory? It's basically in the cases where the user has ever tried to contact customer care through chat or through call, what has the conversations looked like? Probably will be the most important set of transcriptions or data we can analyze from. Remaining data points are important, but if we cannot contextualize their previous conversations, it might be very difficult for us to provide a great solution. So I'll just take to episodic memory for now, but overall I think these are the three pillars of any AI bot. What do you think, Akash? Before proceeding to a system design question, is there anything else you would like me to also cover here? Well, just make sure you think about latency of the bot and how quickly it's working. That'll be one of the key areas we'll need to pressure test it for. [SPEAKER_00] Yeah, that's an amazing thing. Okay, yeah, I think latency and overall performance, whether you are fast and also whether accurate, is something which we want because a lot of times there are wrong prompts which are injected and a lot of times the bots are annoying and not really helpful. So I think doing, I think I'll put in more metrics as to what metrics can we test later, but yeah, I think doing, understanding, I think response time from bot as well as maybe some sort of a basic customer satisfaction score or maybe NPS, maybe feedback might be important, but yeah, I think overall these are some areas which we could think of. So if it's fine... [SPEAKER_00] To what metrics can we test later? But I think understanding response time from bot as well as maybe a basic customer satisfaction score or maybe NPS, maybe feedback might be important. But I think overall these are some areas which we could think of. So if it's fine, can I proceed to a system design diagram to try to portray some of these in more pictures and words? Let's do it. [SPEAKER_01] Cool. Okay. Okay, so at the top we have our very basic interaction layer, which you can say is our mobile app. In this case, it could be web app, whatever it is for your use case. We do need to connect this to our orchestration layer. Now, in any agentic AI system, you do need an orchestration layer which coordinates across different agents and then sends the response to the mobile app. So in this case, I think I have an orchestration layer. Now we do okay. Now in terms of agents, there could be multiple agents. In this scenario, I think you definitely need a data analyst agent, whose sole understanding is to go deep into your data, get all the key details, get all the key points, and then serve it to you, right? Apart from this, we also do need a bot which tries to handle this, right? So we need a very solid customer voice agent in this case who responds to you based on the data. And finally, because we are supporting churn, we would also potentially need an executor agent whose sole task is to make those executions of retention offers. So in today's world, a customer care actually has a lot of different departments within customer care, and we want to try to replicate that entire system where we have a data analyst, we have an agent which talks to you, we also have an agent who ultimately executes some of those actions, whether it is retention offers or whether it is even escalating to a human. And I think underneath, one basic thing which we do need to understand is that they are connected to a data layer. Now it could be RAG, which is probably the most common way you use your data extraction through the vector database. But this is where we could have our data, and effectively our data analyst agent, customer voice agent, and executor agent just uses this. But we need a data layer. And we obviously have our model APIs here, which could be called on as and when we are using it for predicting the different data services or data patterns or even for trying to predict what the human will say. What are we trying to eventually create an offer for them? Let's say if they say that hey, I want to, I like Netflix, so maybe we create an offer especially for them just because they do like Netflix or streaming. So I think these are the basic elements required. I know that there are a lot of other technical terms and a lot of other technical pieces to it, but on a very high level, I feel if we can definitely rely on something like this. What do you think, Akash? I'm happy to also consider any other suggestions. [SPEAKER_01] Yeah, I want to dive deeper into the layer. Like, how are you going to do the churn signals? Are you going to use an LLM or ML models? What models? This kind of thing. Sure. I think in terms... At a level I feel if we can definitely rely on something like this, what do you think, Akash? I'm happy to also consider any other suggestions or audio. I want to dive deeper like how are you going to do the churn signals? Are you going to use an LM or ML models? What models? This kind of thing? [SPEAKER_00] Sure. I think in terms of churn signals, I think our data analyst agent will be strictly working towards it. So for example, if we actually move back, I think we are trying to use different buckets so each user already has been bucketed into different categories of churn. And in terms of, coming back to our diagram, I think data analyst holds all those patterns with regards to each and every user. So once you start interacting with a user, the customer voice agent and the executor agent actually know what exactly the percentage of the user is to churn, what bucket they fall into, and accordingly the voice agent will start taking actions. Like if the user is not really enjoying the experience and if the churn bucket is really high, it is red, will eventually quickly go to serve them with some retention offer. So I think having that kind of framework or system help your agent to ultimately move through those pieces. Okay, so okay, I think now even though we have created some sort of a diagram and have a good solid system for MVP, I think metrics play a huge role and there are different kinds of metrics we can work towards. I think in this particular scenario, as Akash did mention, we do need to focus obviously on latency and performance because those are the end performance indicators. But I think let's start with more regarding, I think model as well. So from just the model perspective, I think some of our metrics like ROUGE, which is recall, which is basically recall over intersection, will really be helpful to understand if the model is able to reuse the data and is able to actually give you very accurate outcomes. Ultimately also gives us understanding of what's the hallucination of our model here. So I think ROUGE is also a good metric, but we can stick to recall right now. I think apart from the model, you also want to understand with regards to our latency, so what's the response rate on average per user? Also, apart from that, we do need to understand from an end user perspective what is the overall NPS. Is the user really liking the performance? Is the problem getting resolved? So what is the percentage of issues resolved by AI bot alone without escalations? That's a huge metric. If our AI bot is just escalating it to human, then it is not really effective. So that's something we need to understand. Also, I think our business metrics are hugely important. I think in terms of retention is one, definitely revenue is a huge factor. I think, again driven by retention, but overall I think a good understanding of what our model is doing, how is it performing, how is the end user actually seeing this, and what ultimately are the levers pushing for business impact is what I think could constitute a good set of metrics for this particular use case. All right, where does this system fall over? You mean like from the mobile app perspective or what? [SPEAKER_01] Would be some of the failure scenarios you would design around? Okay, got it. Yeah, I think that's a good question. From edge cases, I could, let's start from first. I think initially it will be like, let's say our model is down or our model is not working immediately. From the mobile app perspective, what would be some of the failure scenarios you would design around? [SPEAKER_01] Okay, got it. Yeah, I think that's a good question. From edge cases, let me start from the beginning. I think initially, if our model is down or our model is not working, we need to redirect to a human. That's a fail-safe you definitely need to build. You do not want that just because your model is down, the user actually just goes away. You need to redirect to a human on that bot itself and not make them increase the churn percentage. I think apart from that, it's also important to understand the latency. If the model is taking more than 30 seconds to reply on average—I understand that for some decisions it might need more time. For example, retention of course might take time—but on average, if it is taking more than 30 seconds, then I think it's important for us to redirect to a human or try to understand how we can help the user reach that ultimate goal. Apart from that, I also feel that if there is a case where the model is maybe repeating the same response or the user is asking the same questions and maybe getting frustrated overall, it's important for us to understand when we pass the baton to a human. But then overall in terms of failure cases, we understand that the AI agentic system will be the best-case scenario, but we also want to prepare for some of our fail-safes and failure cases. These are the three I can think of right now, but I'm sure there could be much more failure cases we could prepare for. [SPEAKER_01] All right, and what about, let's say, how are you going to build your system to handle a 10x spike in traffic? Oh, so for example, let's say in six months we launch our MVP, we are successful, and now we want to scale it from testing mode to full-time production mode. I think for going to 10x, there are multiple things. Scaling further, I'll break it down. I think you definitely need—from a model perspective—the bandwidth to be able to call that model 10 times or 100 times more. So you definitely would need a very strong server to basically handle all these models. A lot of companies host them on-prem as well just to ensure that. Considering that it's a huge company, we might then switch from just using a cloud to maybe going on-prem so that we can ensure that we are able to control and manage every decision, every API call, or every interaction from the model much more quickly. Apart from the model, because we are going 10x, latency suddenly becomes important. So I feel in my earlier response around on-prem, latency might also improve if we handle it on-premise. So I'm just going to note that, but we'll definitely need much more thorough testing in the new on-prem environment to ensure that our latency is very well-managed. I think data becomes a huge bottleneck because now we suddenly have our data from just maybe a thousand users to maybe 10,000 or 100k plus users. So we need to ensure that we use a vector database. 100 percent for MVP, if we are not using it already. I think ensuring something very quick as a vector database would be highly required because it's much faster and more efficient than a normal database. thousand users to maybe 10,000 or 100k plus users so we need to ensure that our vector we need to ensure that we use a vector database 100 percent for MVB if we are not using it I think ensuring something very quick as a vector database would be would be highly highly required because it's much more faster and efficient than a normal database I think from a memory layer perspective I also there are a lot of intelligent sorry from a memory layer perspective there are a lot of intelligent memory solutions which actually decide whether at all we need to save or process a certain kind of response I like so this is not a sponsored post but I don't know that mem zero is one there are a lot of other solutions as well so I think having an intelligent layer over memory also helps you deal with so many requests much more quickly and ultimately improves latency so maybe using anything or which basically understands whether do we even need to save it or whether we need to process it in this particular field or not and just trying to deal with it is it's really important I feel I think these are some of the different factors Akash do you feel we have covered the ground here or am I missing something oh that's all the time we have for unfortunately but Amman thank you so much for walking through this interview thank you so much Akash so now let's jump out of the interview and let's review right what did Amman do well what could he have done better Amman why don't you go first and I'll add on to it yeah I think asking clarifying questions always helps right so I think that was fine I think with regards to interview process going deeper on users and problem or pain points is very important I think I could have spent more time on users breaking down the user flow much more better because a telecom company has so many different kinds of users so it's a very good opportunity for everyone to spend time there I think I slightly missed out there apart from that I think I also with regards to our solutioning I think I did decently well but I feel that some of the areas which you also pointed out around latency system requirements trade-offs those are some of the things which I could have in just added by myself and just ensured that I build an end-to-end systems and but yeah I think overall it was fine but I would love to know your thoughts as well yeah so I think the big challenge with this type of interview is time management and so yeah overall I wouldn't overly call out the amount of time you spend on users as too small because you want to get to that system design diagram which you got to I think that ultimately where I think you could have improved in this interview is technical fluency and depth and delivery so in terms of technical fluency and depth sometimes I would ask a question like are you going to use an LLM or an ML model what I'm really hoping is for you to first confirm that you understand what I'm talking about and then talk about the pros and cons of each option so an answer might sound a little more tight if it sounded like this that's a great point you know we don't always want to use an LLM when an ML model will do because we know something like an XGBoost algorithm will also be cheaper and a little bit less black box so I think about the pros and cons an LLM model may be able to adapt to new changes in new data sources a little better and might be able to use an agentic search but it'll be much more expensive and it'll be more of a black box so actually I think for the data analyst part of the agent we should use an XGBoost model that might be an example of a strong response and then the other area to improve is delivery you have this crutch of which you basically you don't have any pauses in your speech whenever you have a pause you say and so those are the two big areas to improve in this interview for me but I think that people watching are going to get a lot out of this because now you've seen how to present a structure how to have clarifying questions how to deal with lack of definition from an interviewer and make assumptions how to share [SPEAKER_01] delivery you have this crutch of which you don't have any pauses in your speech whenever you have a pause you say "ah" and so those are the two big areas to improve in this interview for me but I think that people watching are going to get a lot out of this because now you've seen how to present a structure how to have clarifying questions how to deal with lack of definition from an interviewer and make assumptions how to share your screen and draw a system design diagram even if you are a PM so those are the areas for people to take away anything else people should take away or focus [SPEAKER_00] on no I think a great feedback I think one thing you should definitely focus a lot on is asking questions to the interviewer and ensure that it is engaging and not just that you're talking because a lot of times the interviewers also do not expect a one side monologue or conversation so ensure that you're keeping your interviewer very engaged and walking down what you're doing so even if you share your screen there is a tendency for people to just work through silently which I do not suggest so ensure that as you're typing you're talking to an interviewer you're trying to speak your mind out loud so that they can assess what your thought process is and do not be very shy just go there and crush it yeah and in terms of interviewers I would say I was more on the easy side some interviewers will ask more pressing questions and it'll mess up your time management so be really careful about how you handle your interviewer and your time management in this case I modeled a more easy interviewer sometimes you'll get more pushy ones with that we have other mock interviews on this YouTube channel AI product design AI success metrics AI execution go check out those videos and subscribe if you'd like more like this and comment down below what interview mock you would like to see next until the next episode I'll see you later I hope you enjoyed that episode if you could take a moment to double check that you have followed on Apple and Spotify podcasts subscribed on YouTube left a rating review on Apple or Spotify and commented on YouTube all these things will help the algorithm distribute the show to more and more people as we distribute the show to more people we can grow the show improve the quality of the content and the production to get you better insights to stay ahead in your career finally do check out my bundle at bundle.akashg.com to get access to nine AI products for an entire year for free this includes dovetail mobbin linear reforge build descript and many other amazing tools that will help you as an AI product manager or builder succeed I'll see you in the next episode Think about it. It's like its own code base, maybe its own repo. So how are we going to build it specifically? The technical areas is what I'm most interested in. All right. Okay. Cool. So the reason I ask is because there are a lot of things which a company does because of its competitors. So if our competitors are trying to do X, Y, Z, is that one of the reasons we are trying to do this? Or there is no independent, like there is no like dependent correlation is what I'm trying to deduce from it. But it looks like in your case, we just need to build an independent system and we don't really have any other goals apart from churn in a nutshell. Yeah. I mean, just the normal reasons, right? Okay. If we can identify churn, we can take actions in the product to improve engagement. We can secure more revenue. Generally tends to be more profitable for us than focusing on acquisition. So we have an overall push there. Sounds good. Cool. And do you have any suggestions as to what kind of an industry product should I go towards or that's up to me to decide which product to pick for this? Just consider like software. Okay. Cool. Okay. Sounds good. So I'll just take a minute. I'll try to define the product and the vision I have. And then I'll just get back to you in about, let's say, 30 seconds, if that's fine. Sure. Take your time. Okay. So I think with regards to the vision, as you're speaking more about Agente KI, customer care is something which we are trying to, you know, as an industry, not just telecom industry, but even other industries, we're trying to use Agente KI to make our purpose more efficient. So I think what I'll do is I'll try to take an example from telecom nations industry because I think it will be easier for people to understand. And also in terms of the product, I'll try to take a mobile app, which is, which will have the necessary Agente KI pipeline built in, which effectively should help us produce churn. But this is my vision for the product. Is there any question before I jump into the next section or what are your thoughts on this? Sure. Yeah. Makes sense to kind of focus it in a specific real life, more textured situation. Cool. Okay. So what I'll do now is I'll start talking about some of the users. I'll try to prioritize one of them and I'll go towards the pain points. And finally, I'll move towards solutioning where we'll discuss about the different AI solutions. And finally, we'll jump into system design in terms of the diagram as well as some of the matrix we can track. Does that sound good? Cool. Yep. Okay. Okay. I'll just take 40 seconds. I'll get back with some of the users we can target. So I think with regards to users of any of, of not just this kind of product, but any kind of product, there are different ways how you can segment users. What I would like to potentially do is that I will actually, I'll try to speak to our users in terms of when did they join us. So there are a lot of new users. There are obviously power users. There are users who also focus from B2B perspective. A telecom company has B2B users as well. There are, so I think I'll try to stick to different types of users. Broadly, I think because we want to prioritize to only one user, I will try sticking to power users. And the reason is because these are the users we want to take care of the most. I'm not saying not all users are important, but in any product, the users who are mostly engaging with you, who are most deriving value for you and out of your product, you want to protect them and you want to ensure that their interests are well met. So we do not want to lose them. So from a churn point of view, I think I want to prioritize this segment of power users and proceed from there. Do you have any question, Akash, before I proceed to pain points? Yeah, I think power users make sense with our broader strategy anyway. So great. Before jumping into pain points, what I would actually, what I like to do is also something called user journey. Pain points are often derived through the different phases of user journey. So I'm just going to spend a few seconds to define what a user journey of a power user might look like. So we can understand that this user already has been part of this network for a while and has been using our app and customer care for a while now. And one thing I definitely know is that there are definitely going to be a lot of, there might be a lot of customer issues. So this user calls or contact customer care through own tries to get a ticket for this user for the problem. rather, once that happens, the customer normally waits, reaches out again, if required, to the customer care through email maybe. Once issue is resolved, normal, normal system, normal system hour is restored and the user effectively starts using the app as well now. Now on the app, now on the telecom app, you can do a lot of things, but just to not take much time, there are a lot of benefits provided on the app these days for our users. So he can browse benefits and also he can actually try to get Wi-Fi of the same company or avail other services. So these are some of the user journey items which I feel is very common among our telecom, our top telecom providers. And there are a lot of pain points among these. Firstly, I think user call, contact, I mean, everyone hates calling customer care. You do not want to spend your time calling customer care if especially you are a power user. So I think one pain point is definitely huge amounts of time and effort taken for each of these customer interactions, customer care interactions rather, which also ultimately causes anxiety, which also causes a lot of frustration, delay. And I think it overall, it doesn't really like leave a good mark on the user's experience. With regards to pain points, huge amount of time and effort for customer care. Apart from this, there is also the lot of customer pain points around the app experience. So when you open the app, you might not be able to track, sorry, you might not be able to track your request across customer call and app. So that kind of causes a lot of confusion as well. That was my problem ever filed? What did I do wrong? Do I need to call the customer care again? And this is something which we definitely do not want to do. Finally, I think one pain point is also that regarding benefits. We, there are a lot of users with different interests and we do not want, if you like dining, we do not want you to unnecessarily show maybe a benefit for a Wi-Fi. So I think there are a lot of irrelevance. Irrelevance becomes a huge problem in terms of benefits and services. So we are not able to tap in the right value for the right user and that causes also friction, loss of potential, upsell, which otherwise could have happened. So I think these are some of the pain points. I know there could be a lot, but these are some of the pain points I'm looking at. Before I prioritize Akaf, do you have any questions on any of these? Maybe? Yeah, I think these are good ones to focus on for this particular system design. Okay. With regards to prioritization, I like to consider some of these things. Firstly, alignment to my vision, original vision, because there are a lot of problems. I do not want to focus on problems which do not align with my companies or with my products vision. Apart from that, it is also frequency. an impact. So what is the frequency of the pain point? How much impactful or how much problems it is causing for my consumer? I think with regards to that, I definitely would rate this as a very highly recurring problem with huge impact in terms of your experience. So I'll just try to maybe mark it as high for all the three domains. Apart from this, I do understand that inconsistent experience experience is a big problem. But I feel that even if we do solve your problem at the end of the day, it might not matter as much in the short term. So we might go with a mid here and not a high for some of these because it doesn't directly align with our immediate vision which is to create a highly engaging product and reduce churn. So I might just go medium here. If relevance in benefits causes friction, loss of potential and upsell. I feel this is again very important that we surface relevant benefits and services but that is more to do towards making our user experience highly, highly engaging which I do want to do. But when it comes to churn reduction, I feel that it might again not be our top lever in terms of reducing churn at this moment. So I'll again try to shift towards medium for this. So I'll try to prioritize only one pain point at this time which is this one and I want to try to devise an agentic solution just to resolve this kind of a pain point in our system. What do you think? Any thoughts there before I move into solutioning? How do we... One of the things we want to solve for in the system design is just giving our product teams the right signals that hey this user is about to churn so we need to do something take an intervention. So how do we get that early warning side? Exactly. So I think that that's a very highly proactive system which we want to build so that before the user calls for you to churn before three to four calls or before three to four what do you say or even types of interactions we are able to predict. So that is something which I'll be extending more in my solution section. So if it's fine I can... Now I think that's a good segue to a solutions section. I'll just take 30 seconds and I'll try to jot down all the different solutions possible if that's fine. Akash. Sure. So I think with regards to solutions there are a lot of different solutions possible to this one problem. I think one of them is a part which is pretty common where the system effectively tries to understand from your past called transcription and accordingly helps you guide your solutions very quickly but I feel that that might not be as engaging as possible because a human always prefers a more human-like touch. There is also something around voice port and I feel voice port might really be a very good scenario in this where you're not just able to talk to the AI agent but you're able to show your screen to it or able to actually help it exactly end-to-end guide you and not just redirect you to a human agent. So I think an end-to-end AI voice port assistance would be something amazing. In terms of a more basic solution which might not require AI as such we can make our experience more gamified in the app to just show the features and benefits in a way that the user always drives to avoid churning just because they keep getting some of these benefits and features but I feel that because at the core we want to improve the user experience I would like to go and try to build more of a voice port agent which tries to not just speak to you but actually tries to predict from your past data like you mentioned Akash when the user is going to maybe be in that red zone of churning and accordingly tries to activate that system of maybe sending more offers maybe trying to send more benefits to them and also trying to talk to the user end-to-end and solve the problems as soon as possible so this is what I would like to prioritize if that's fine Akash to you or I'd love to know your thoughts it's going to be a big project I don't know six months will depend really on the resources right but as I said we can think about a best scenario yeah yeah I think no that's true I think six months is a big time but I feel that we can definitely map out an MDP version of it in six months and from there maybe craft a very good agent in maybe nine to ten months but yeah sure what do you think let's see what it would look like okay cool so I think before jumping into our system design diagram I would like to just talk about different layers of an AI system today and what this point might actually have so actually go to a new page okay so now that we are targeting this maybe okay so I think each each AI agent or bot has potentially three things I think one is obviously model data memory these three things are the pillars of any AI system and in terms of model to be very honest you already have some of the best AI models available at your disposal and we could accordingly do a competitive analysis and basic testing on an engineering level and see which bits are use case for example for a voice would you prefer maybe an opus versus and like a GPT-5 is a question which you can always try to understand do some benchmarking and come to that conclusion but overall I think we don't need to spend much time on the model right now and you can all the best models are at disposal so I'm just going to park this as for your use case in this case it is voice pod so I might actually go to maybe Gemini because I feel that has a great voice experience but again it can be anything based on how your engineering team or ML team perceives this now I think coming to data this is probably the core of it right without data your very powerful model is completely useless in terms of data you definitely need call transcription right that's the most basic thing you can provide but you also need different signals for example app usage for example your network status through the past let's say 12 months your what has been your not search history but what has been your connectivity with regards to maybe the different competitors are you for example if you're traveling a lot are we able to provide you those benefits of traveling which our competitors are providing or what kind of a user are you so I think having a lot of different data points always helps you understand what kind of user are we dealing with and accordingly I think putting them putting all these in a churn bucket would be helpful for example if you're a high risk churner then maybe put it in the red bucket or maybe if you are not maybe in a green bucket so we can I think the model can effectively try to use these data points and then predict and put you in that bucket so that accordingly we can try to understand you also I think we would also need some last minute retention offer so let's say if you are in a red bucket and an AI agent might let's say is unable to resolve we need some sort of a last minute retention offer just for you so that we can actually send it or the AI agent rather can send it to you I think these are also highly highly crucial finally jumping into memory I think in terms of memory there are different types of memory available there is episodic memory there is there is also session memory so I think we need to we need to understand that we definitely can we do not want to store anything and everything so we need to be really efficient here as well so I would say that an episodic memory is highly important what do I mean by episodic memory is basically in the cases where the user has ever tried to contact customer care through chat or through call what has the conversations looked like probably will be the most important set of transcriptions or data we can analyze from remaining data points are important but if we cannot contextualize their previous conversations it might be very difficult for us to provide a great solution so I'll just take to episodic memory for now but overall I think these are the three pillars of any AI bot what do you think Akash before proceeding to a system design question is there anything else you would like me to also cover here well just make sure you think about latency of the bot and how quickly it's working that'll be one of the key areas we'll need to pressure test it for yeah that's an amazing thing okay yeah I think latency and overall performance whether you are fast and also whether accurate is something which we want because a lot of times there are wrong prompts which are injected and a lot of times the bots are annoying and not really helpful so I think doing I think I'll put in more metrics as to what metrics can we test later but yeah I think doing understanding I think response time from bot as well as maybe some sort of a basic customer satisfaction core or maybe NPS maybe feedback might be important but yeah I think overall these are some areas which we could think of so if it's fine can I proceed to a system design diagram to try to portray some of these in more pictures and words let's do it cool okay okay so in the top we have our very basic interaction layer which you can say is our mobile app in this case could be web app whatever it is for your use case it we do need to connect this to our orchestration layer now in any agentic AI system you do need an orchestration layer which coordinates across different agents and then sends the response to the mobile app so in this case I think I have an orchestration layer now we do okay now in terms of agents there could be multiple agents in this scenario I think you definitely need a data analyst for a data analyst agent for you who whose sole understanding is to go deep into your data get all the key details get all the key points and then serve it to you right apart from this we also do need a bot which which actually tries to hop off this right so we need like a very solid customer voice agent in this case who responds to you based on the data and finally because we are supporting churn we would also then need some we would also potentially need an action not action okay maybe an execute executor agent who whose sole task is to make those executions of maybe let's say the retention offers so in today's world a customer care actually has a lot of different like there are a lot of different departments within customer care and we want to try to replicate that entire system where we have a data analyst we have an agent which talks to you we also have an agent who who ultimately execute some of those actions whether it is retention offers or whether it is even maybe escalating to a human and I think underneath one one basic thing which we do need to understand is that they are connected to now it could be rag which is probably the most common way you use your data extraction through the vector database but yeah this is where we could have our data and we effectively our data analyst agent customer voice agent executor agent just uses this but we need a data layer and we obviously have our yeah okay we do have our model APIs here which which could be called on as and when we are using it for predicting the different data services or data patterns or even for trying to predict what the human will say what what are we trying to eventually like create an offer for them let's say if they say that hey I want to I like Netflix so maybe we create and we create us offer especially for them just because they do like Netflix or streaming so I think these are the basic elements required I know that there are a lot of other technical terms and a lot of other technical pieces to it but on a very high level I feel if we can definitely rely on something like this what do you think Akash I'm happy to also consider any other suggestions or audio yeah I want to dive the layer deeper like how are you going to what how are you going to do the churn signals are you going to use an LM or ML models what models this kind of thing sure I think in terms of churn signals I think our data analyst agent will be strictly working towards it so for example if we actually move back I think we are trying to use different buckets so each user already has been bucketed into different categories of churn and in terms and coming back to our diagram I think data analyst holds all those patterns with regards to each and every user so once you start interacting with a user you effectively the customer voice agent and the executor agent actually know what exactly the percentage of the user is to churn what what kind of bucket they fall into and accordingly the voice agent will start taking actions like if the user is not really enjoying the experience if and if the churn bucket is really high it is red will eventually quickly go to serve them with some retention offer so I think having that kind of framework or system help your agent to ultimately move through those pieces okay yeah so okay I think now even though we have created some sort of a diagram and have a good solid system for MVP I think matrix play a huge role and there are different kind of matrix we can work towards I think in this particular scenario as Akash did mention like we do need to focus obviously on latency and performance because those are the end performance indicators but I think let's start with more regarding I think model as well so from just the model perspective I think like some of our matrix like R-U-G-E which is recall which is basically recall over our intersection will really be helpful to understand if the model is able to reuse the data and is able to actually give you very accurate outcomes ultimately also gives us understanding of what's the hallucination of our model here so I think R-U-G blue is also a good matrix but we can stick to recall right now I think apart from the model you also want to understand with regards to our latency so what what's the response rate on an average per user also apart from that we do need to understand from an end user perspective what is the overall NPS is the user really liking the performance is the user is the problem getting resolved so what is the percentage of issues are resolved by AI bot alone without escalations is a huge metric if our AI bot is just escalating it to human then it is I think not really effective so that's something we need to understand also I think our business metrics are hugely important I think in terms of retention is one definitely revenue is a huge factor I think again driven by retention but overall I think a good understanding of what our model is doing how is it performing how is the end user actually seeing this and what ultimately are the levers pushing for business impact is what I think could constitute a good set of metrics for this particular use case all right where does this system fall over you mean like from the mobile app perspective or what would be some of the failure scenarios you would design around okay got it got it okay yeah I think that's a good question from a weird cases I could let's start from first I think initially it initially it will be like let's say our our model is down or our model is not working immediately we need to redirect to human so that's a fail safe you definitely need to build you do not want you know that just because your model is down that the user actually just goes away you need to redirect to human on that bot itself and not make them make the increase the churn percentage I think apart from that it's also important to understand the latency so if if model is maybe taking let's say more than 30 seconds to reply on an average I understand that for some of the decisions for it to make it might need more time for example retention of course might take time but on an average if it is taking more than 30 seconds then I think it's important again for us to redirect to a human or try to understand how we can help the user reach that ultimate goal apart from that I think I also feel that apart from model and latency if the there is also case where the where a model is maybe repeating same response or user is asking same questions and maybe getting frustrated overall it's again important for us to understand when do we pass the bait on to human but then overall in terms of failure cases we we understand that the AI agentic system will be the best case scenario but we also want to prepare as a car suggested for some of our fail safe and failure cases these are the three I can think of right now but I'm sure there could be much more failure cases we could prepare for all right and what about let's say how are you going to build your system to handle this 10x spike in traffic oh god so for example let's say in six months we launch our MVP we are successful and now we want to scale it from let's say and a testing mode looks to a full time production mode I think for going to 10x there are again multiple things I think I'll uh so scaling further again I'll break it down I think you definitely from a model uh from a model perspective you would uh now need to have that bandwidth to be able to call that model 10 times or 100 times more so you definitely would need I would say like uh a very strong like you you would basically need a much more strong and dense server to basically handle all these models a lot of companies host them on-prem as well you just to ensure that so considering that it's a huge company uh we might then switch from just using a cloud to maybe going on-prem so that we can ensure that we are able to control and uh manage uh every every decision or every call api call or every uh interaction from model from our end much more quicker apart from the model I feel because we are going 10x latency suddenly becomes important so I feel in my earlier response around on-prem I'm uh I feel that a latency might also improve if we uh handle it uh on premise uh so uh I'm just going to take to that but we'll definitely need a much more thorough testing in new on-prem environment to ensure that our latency is very well I think uh data becomes a huge uh bottleneck because now we suddenly have our uh our entire data from just maybe thousand users to uh maybe uh 10,000 or 100k plus users so we need to ensure that uh our vector uh like we need to ensure that we use a vector database 100 percent uh for MVB if we are not using it I think ensuring something very quick as a vector database uh would be uh would be highly highly uh required because it's much more faster and efficient than a normal database I think from a memory layer perspective I also there are a lot of intelligent sorry from a memory layer perspective there are a lot of intelligent memory uh solutions which actually uh decide uh whether at all we need to save uh uh or process a certain kind of response I like uh so this is not a sponsored post but I don't know that uh mem zero is one there are a lot of other solutions as well so I mean I think uh having an intelligent layer over memory also helps you deal with so many requests much more quickly and ultimately improves latency so maybe using anything or which basically understands whether do we even need to save it or whether we need to process it in this particular field or not and just trying to deal with it is it's really important I feel I think these are some of the different factors Akash uh do you feel we have covered the ground here or am I missing something oh that's all the time we have for unfortunately but Amman thank you so much for walking through this interview thank you so much Akash so now let's jump out of the interview and let's review right what did Amman do well what could he have done better Amman why don't you go first and I'll add on to it yeah I think asking clarifying questions always helps right so I think that was fine I think with regards to interview process uh going deeper on users and problem or pain points is very important I think I could have spent more time on users breaking down the user flow much more better because a telecom company has so many different kinds of users so it's a very good opportunity for everyone to spend time there I think I slightly missed out there apart from that I think I also with regards to our solutioning uh I think I did decently well but I feel that some of the areas which you also pointed out around latency system requirements trade-offs those are some of the things which I could have in uh just added by myself and just ensured that I build an end-to-end systems and but yeah I think overall it was fine but uh I would love to know your thoughts as well yeah so I think the big challenge with this type of interview is time management and so yeah overall I wouldn't overly call out the amount of time you spend on users as too small because you want to get to that system design diagram which you got to I think that ultimately where I think you could have improved in this interview is technical fluency and depth and delivery so in terms of technical fluency and depth sometimes I would ask a question like are you going to use an LLM or an ML model what I'm really hoping is for you to first confirm that you understand what I'm talking about and then talk about the pros and cons of each option so an answer might sound a little more tight if it sounded like this that's a great point you know we don't always want to use an LLM when an ML model will do because we know something like an XGBoost algorithm will also be cheaper and a little bit less black box so I think about the pros and cons an LLM model may be able to adapt to new changes in new data sources a little better and might be able to use an agentic search but it'll be much more expensive and it'll be more of a black box so actually I think for the data analyst part of the agent we should use an XGBoost model that might be an example of a strong response and then the other area to improve is delivery you have this crutch of uh which you basically you don't have any pauses in your speech whenever you have a pause you say ah and so those are the two big areas to improve in this interview for me but I think that people watching are going to get a lot out of this because now you've seen how to present a structure how to have clarifying questions how to deal with lack of definition from an interviewer and make assumptions how to share your screen and draw a system design diagram even if you are a PM so those are the areas for people to take away anything else people should take away or focus on no I think of course a great feedback I think one thing you should definitely focus a lot on is asking questions to the interviewer and may ensure that it is engaging and not just that you're talking because a lot of times the interviewers also do not expect a one side monologue or conversation so ensure that you're keeping your interviewer very engaged and walking down what you're doing so even if you share your screen there is a tendency for people to just work through silently which I do not suggest so ensure that as you're typing you're talking to an interviewer you're trying to speak your mind loud so that they can assess what's your thought process and do not really be very shy just go and go there and crush it yeah and in terms of interviewers I would say I was more on the easy side some interviewers will ask more pressing questions and it'll mess up your time management so be really careful about how you handle your interviewer and your time management in this case I modeled a more easy interviewer sometimes you'll get more pushy ones with that we have other mock interviews on this YouTube channel AI product design AI success metrics AI execution go check out those videos and subscribe if you'd like more like this and comment down below what interview mock you would like to see next until the next episode I'll see you later I hope you enjoyed that episode if you could take a moment to double check that you have followed on Apple and Spotify podcasts subscribed on YouTube left a rating review on Apple or Spotify and commented on YouTube all these things will help the algorithm distribute the show to more and more people as we distribute the show to more people we can grow the show improve the quality of the content and the production to get you better insights to stay ahead in your career finally do check out my bundle at bundle.akashg.com to get access to nine AI products for an entire year for free this includes dovetail mobbin linear reforge build descript and many other amazing tools that will help you as an AI product manager or builder succeed I'll see you in the next episode you nope nope nope nope