SPEAKER_01
All right, so I'm in the slot before the snacks, but I'm here to speak about the host engineering org and a case study we've been doing at PFF. It started in January and we finished it in March. PFF is a sports data company. We help NFL and NCAA teams figure out what they should be doing and we also have a consumer arm which does fantasy football, sport betting, and for those of you who know American football, there's a draft happening end of the month and so allows people to play as one of these teams.
SPEAKER_01
We're a fully distributed engineering team, so we've got engineers in India, we've got engineers in Spain and all the states in America. But first let's get into some stats. So we have 100 million page views annually. We have 9 million drafts happen on an annual basis. It's a fairly popular tool. And the issue that we had was we were 200 employees, around 20 engineers, and we were falling behind competitors. We were focusing on sport betting stuff, but a lot of our consumers were actually interested in this. And then with Claude Opus coming out, I started experimenting around November on a personal level. And then it got spun out to two engineers. One of them was our strongest front end engineer, and the other one was one of our strongest full stack engineers. So really the question I was asking was instead of figuring out how we can help engineers output more, how do we help make the agents quicker?
SPEAKER_01
So I think if we all think historically software engineering, you've got the agile manifesto software craftsmanship, you've got a lot of good perks and benefits for engineers from foosball tables to sleeping pods to pretty much every perk that most other industries really don't have. And it's because we're the bottleneck. And if companies are able to optimize for that, it gives them a lot of benefit. But now things are changing.
SPEAKER_01
So I just want to go straight into how the case study ended up. So we had 25 times more deploys. So the two engineers were deploying five times every day. And the other team was a team of around 10 engineers. They're doing it pretty much one deploy every five days. There's a big obvious caveat here, right? Small engineering teams are always going to be quicker than the big ones. So there is an element of that multiplier that is just from it being a small engineering team. However, that smaller target team still had to coordinate all of those daily deployments, still have to go and coordinate all of those employees with the bigger team. So there was still that issue that was happening.
SPEAKER_01
And this one, this is always hard. How do you validate that the output is actually helpful? The number of PRs isn't helpful. The amount of code isn't helpful. So we basically blended the number of tickets with the code complexity. And we found that their output was 10x. Got another slide which might look a little hectic, but just bear with me. So these are the features that we constructed. With those two engineers, it took them under two months. If we're going to do this before, we were estimating it's going to take four months. The big thing you can probably see in the top half is one of the engineers gets unblocked in under a month and can then start building other stuff. Whereas in the old way, they're both blocked for three months. And so you get this thing where you now have a compounding increase where it's not just faster for that stuff, but you're now able to do a lot more stuff than you were able to do before.
SPEAKER_01
And the one thing that I really want to highlight is it doesn't matter if the output is more, it doesn't matter if the number of deployments are higher. What really matters is if our customers are happy. And so we did statistically significant surveys and the average quality score was 8.6 out of 10. What was interesting was before AI, we would probably average 7, 7.5. So we weren't really delivering what the customers had been interested in.
SPEAKER_01
So Scrum did not survive. We were looking not just on an engineering front and all of the delivery gains from that, but also from a process standpoint. And so no need for a project manager. We don't need to play multiple games of telephone. And everything we were doing was optimizing to be as quick as possible. Engineers aren't the bottlenecks. So we don't need to have all the old ceremonies that we had before.
SPEAKER_01
So what did we have? It looks pretty basic. We basically had huddles. So those huddles were basically every other day, they'd be half an hour, maybe an hour. So you'd have the engineers, someone from product, someone from the design team. And you would speak about the things you've been building the last couple days. You get instant feedback. We were trying to deploy to production as fast as possible in an MVP state and getting as much feedback as possible also.
SPEAKER_01
And that development flow is we have a spec. I think most of us are probably fairly familiar with this. But we get the agent to go and interview us. We get feedback on that spec. And then we make a lightweight design document. That's done by the agent. So we have this skill. And the nice thing about that is it analyzes how we've done all the LDDs before. And so anything that we're building is in the same ethos as everything else that's been built. So it's not cloud code specific. And then those LDDs get distributed and we get feedback from all of the engineers. And then at that stage, we automatically create all the tickets and then the PRs after that.
SPEAKER_01
So sprint planning. We don't have sprint planning anymore because we don't need to have an hour going and estimating tickets because those estimations don't really make any difference. Something we're not doing at the moment but we will probably have to do once the subsidization of tokens ends is we'll probably have to go and estimate the token expenditure to see if we want to actually spend this amount of money on it.
SPEAKER_01
The daily stand-ups we don't have to do because all these tickets get auto updated. That is specific to the status of the PR. So if the PR is open, it goes auto in progress. If it goes into review, it updates it. If it gets merged, it gets closed. These things were obviously manageable to do before but it's all been made easier. Sprint refining we don't need to do because that happens in the spec and the LDD flow and then when we automatically create all the tickets, we structure it in a manner so the tickets are made so none of them are blocking each other. And if there are ones that are blocking each other, that information is flagged up and highlighted.
SPEAKER_01
And retrospective, this one might be a little controversial, but we rely on the customer satisfaction, the customer survey. That's the main thing. And then you've got the normal development metrics like the deployment frequency. We ask all of our engineers to flag when there is an issue to flag it immediately and instead of hanging on until the end of the sprint to flag that issue where sometimes
SPEAKER_01
Sprint refining we don't need to do because that happens in the spec and the LDD flow and then when we automatically create all the tickets, we structure it in a manner so the tickets are made so none of them are blocking each other. And if there are ones that are blocking each other, that information is flagged up and highlighted. And retrospective, this one might be a little controversial, but we rely on the customer satisfaction, the customer survey. That's the main thing really. And then you got the normal development metrics like the deployment frequency. We ask all of our engineers to really flag when there is an issue to flag it immediately and instead of hanging on until the end of the sprint to go and flag that issue where sometimes I think all of us have probably felt that our feedback hasn't really been heard because it's been a sprint before.
SPEAKER_01
So how do we start? You got to pick the engineers which have the best system and knowledge. And I think every engineering team probably has one or multiple engineers where if anyone ever gets stuck or hung up on anything, they're going to be able to sort it out. The other one is you should go slowly. So there is an appetite to give everyone all the coding assistance and open it all up. But I think with this, you want a slow-faced approach. And then you should experiment in non-critical systems. So that's what I did in the November and in the two months before the case study. I was making small proof of concept features, going and pushing it to production. Didn't really get much traction. So if there was a bug or a mistake, it didn't really matter. And then we moved it to the thing which gets 100 million page views after that. The big thing here though is not everyone can drive a sports car. And that's all right. That's not a big issue. But I think everyone needs to be a little bit honest around the engineering org and engineering team. And this new era is going to be hard for a few engineers. And I think the type of engineer where they'll really thrive is one who is curious. It's the engineer who, if they haven't figured it out, they're going to spend a bit of time just figuring out how something's been built. They'll be able to smash this easily. The old style of engineer which needs something really prescriptive as a spec, I think they're going to struggle.
SPEAKER_01
So in the engineering side of things, you need verifiable deterministic tasks. I think a bunch of people have spoken about this today. It ranges from everything from all the different types of tasks. But there's also things which are specific for your product or your org. So for us, it's feature flag because we do trunk-based development. Or it could be generating the interactive elements, generating the analytics for the interactive elements.
SPEAKER_01
Next thing is the agentic code review. So with us, we didn't really like relying on agents to do code reviews for systems as Hine and what engineers would usually do code reviews for. We use agents to do code reviews for engineers to get feedback from engineers. So that's variable names, this doesn't fit the style, or those kind of opinionated manners is much easier to offload. And then you can remove that whole emotional aspect out of it. And then you can allow the engineers to focus on the big picture. We still need people involved. But it's really heavy on the spec, really heavy on the LDD, the lightweight design document, where we figure out how we're going to build this. And the thing I haven't touched on at the moment is the product. We're in an era where everyone's able to create anything in an hour. But a lot of these tools have the brand feel and the product feel of something that's been created by cloud code. And so if you really want to get the best out of it, you really need to make sure that the engineering team is spending time making sure it still feels basically every other product from the company.
SPEAKER_01
So how would I recommend tackling this? You should go see your engineering development life cycle as a factory. So you should think you're in a factory. How do you break that up into each small composable element? So in a car factory, there's one thing which is building a door, another one fitting a steering wheel. The exact same thing happens for engineering, right? So you have the branch name, you've got creating feature flags with trunk-based development. You've got if you build APIs with a specific software design pattern, you should abstract that into a composable skill. The one thing I would flag is I'm not a big fan of consuming other people's skills if they have strong software opinions that are in contrast to the engineering org because you're just going to end up an issue. So you just need to make sure that any skills that are getting composed are still patching. And this is just at the moment pretty much everything is fully autonomous from the spec LDD, a ticket from the PR. We have a QA process also. So what happens is whenever we merge a PR, it automatically deploys onto staging. When the deployment on staging has happened, we will spin up a QA agent which has a look at all of the tickets that have happened, has a look at the acceptance criteria, and it will go and then do QA against that. And then if everything's passed, great. And then if everything hasn't, it will flag what those items are. And the bit we haven't gone and done at the moment, but I'm aiming in the next couple months, is to then have an agent have a look at those tickets, find out where the acceptance criteria hasn't happened, and then automatically create the PRs. And so then you get into this flow where agents can basically self-heal. And the thing about that is it lets us do multiple things in parallel because we're now in a position where we basically trust agents as well.
SPEAKER_01
Where do we still use people? They do like to use shortcuts. So we've got to make sure the security side of things is behaving as a product field. I spoke about that a little bit earlier. And then the scale and engineering complexity for the task. DLDD is really meant to help on this. So I think we've all experienced where it'll make a thousand lines of code or over-engineer something. If you're really prescriptive on that engineering document at the start, you can really prevent that from happening.
SPEAKER_01
So what things should you go and aim for? You should start with boring, repetitive tasks. Ideally, things engineers hate because you're going to get the most amount of buy-in from that. Remove as much redundant process as possible. And I know it can be strange when you've been doing something for two decades plus to just chuck it all out. But I would ask you, what is the purpose of this meeting? What's the purpose of this process? Is it just because everyone else has been doing it before? Or is it because it actually helps out? Make sure your team's personal engineering culture and the patterns are encoded in skills. So if there are software design patterns that you really
SPEAKER_01
So what things should you go and aim for? You should start with boring, repetitive tasks. Ideally, things engineers hate because you're going to get the most amount of buy-in from that. Remove as much redundant process as possible. And I know it can be strange when you've been doing something for two decades plus to just chuck it all out. But I would ask you, what is the purpose of this meeting? What's the purpose of this process? Is it just because everyone else has been doing it before? Or is it because it actually helps out? Make sure your team's personal engineering culture and the patterns are encoded in skills. So if there are software design patterns that you really do, so for us, whenever we build an API, we focus on the service repository pattern. Make sure your guard rails are fully functional before you get into an autonomous flow. And you should start out with the best engineers. Things you should not do is try to onboard everyone at the exact same time. I think a big reason why a lot of these things haven't really panned out is because we've given everyone a cloud code codex and given them a hackathon of, sweet, we've done everything. They should now be able to sort it all out. It's just not that easy. Every engineering org is completely different. And every engineering org's style is really different. And you just need to go and have a slow-paced approach to make sure that transition is happening smoothly. And that's why I think those small companies are at a huge advantage over the big enterprise companies because it's really easy for me to scale this out when I've got 20 engineers. It's really hard for me to scale this out if I was in charge of 100 or if I was in charge of 1,000 engineers or if I was in charge of 10,000 engineers. So, but you don't want to be too shy. You don't want to be too conservative. Because the fact is that there's a lot of other companies that are going full speed. And for us, even from our perspective, I felt a few months behind and I was feeling it. And I think if you think about the compounding impact that's being talked about at the start, that's just going to carry on happening. So a few months behind at the moment might be six months behind and a few months might be 12 months behind a little bit afterwards. And then again, just take it slow. It needs a phased approach and rely on your engineers to let you know whether you can scale things faster or if it needs a bit more time. I know I spoke to you about a lot of features that we built, but I didn't show you any of the features. If anyone does want to have a play around, feel free to scan the QR code and then you can have a look at all this stuff. Cool. Thank you, everyone.
SPEAKER_01
deploying five times, five times every five, we're deploying five times every day. And the other team was a team of around 10 engineers. They're doing it pretty much one deploy every five days. There's a big obvious caveat here, right? Small engineering teams are always going to be quicker than the big ones. So there is an element of that multiplier that is just from it being a small engineering team. However, that smaller target team still had to coordinate all of those daily, still have to go and coordinate all of those employees with the bigger team. So there was still that kind of issue that was happening.
SPEAKER_01
And this one, this is always hard. How do you validate? Like the output is actually helpful. The number of PRs isn't helpful. The amount of code isn't helpful. So we basically blended the number of a tickets with the code complexity. And we found that their output was a 10x. Got another slide which might look a little bit hectic, but just bear with me. So these are the features that we went and constructed. With those two engineers, it took them under a two month. If we're going to do this before, we were estimating it's going to take four months. The big thing you can probably have a look in the
SPEAKER_01
in the top half is one of the engineers gets unblocked in under a month and can then start building other stuff. Whereas in the old way, they're both blocked for three months. And so you get this thing where you now have a compounding increase where it's not just faster for that stuff, but you're now able to do a lot more stuff than you were able to do before. And the one thing that I really want to go and basically highlight is it doesn't matter if the output's more, it doesn't matter if the number of deployments are higher. What really matters is basically if our customers, if the customers are happy. And so we did statistically significant surveys and the average
SPEAKER_01
quality score was 8.6 out of 10. What was interesting was before AI, we would probably average 7, 7.5. So we weren't really delivering what the customers had been interested in. So Scrum did not survive. We were basically having a look not just on an engineering front and all of the delivery gains from that, but also from a process standpoint. And so no need for a project manager. We don't need to play multiple games of the telephone. And everything we were doing was optimizing to be as quick as possible. Engineers aren't the bottlenecks. So we don't need to have all the old ceremonies that we had before. So what did we have? It looks pretty basic. We basically had
SPEAKER_01
huddles. So those huddles were basically every other day, they'd be like half an hour, maybe an hour. So you'd have the engineers, someone from product, someone from the design team. And you would speak about the things you've been building the last couple days. You get instant feedback. We were trying to deploy to production as fast as possible in an MVP state and getting as much feedback as possible also. And that development flow is we have a spec. I think most of us are probably fairly familiar around this. But we get the agent to go and interview us. We get feedback on that spec.
SPEAKER_01
And then we make a lightweight design document. That's done by the agent. So we have like a skill. And the nice thing about that is it analyzes how we've done all the LDDs before. And so anything that we're building is in the same kind of ethos as everything else that's been built. So it's not a cloud code specific kind of thing. And then those LDDs, they get distributed and we get feedback from all of the engineers. And then at that stage, we automatically create all the tickets and then the PRs after that. So sprint planning. We don't have sprint planning anymore because we don't need to have
SPEAKER_01
an hour going and basically estimating tickets because those estimations don't really make any difference. Something we're not doing at the moment but we will probably have to do once the once the subsidization of tokens ends is we'll probably have to go and estimate the token expenditure to go see if we want to actually spend this amount of money on it. The daily stand-ups we don't have to do because all these tickets get auto updated. That is specific to the status of the PR. So if the PR is open, it goes auto in progress. If it goes into review, it updates it. If it gets merged, it gets closed.
SPEAKER_01
These things were obviously manageable to do before but it's all been made a little bit easier. Sprint refining we don't need to do because that happens in the spec and the LDD flow and then when we automatically create all the tickets, we structure it in a manner so the tickets are made so none of them are blocking each other. And if there are ones that are blocking each other, that information is flagged up and highlighted. And retrospective, this one might be a little controversial, but we rely on the customer satisfaction, the customer survey. That's the main thing really. And then you got the normal development metrics like the deployment frequency.
SPEAKER_01
We ask all of our engineers to really flag when there is an issue to flag it immediately and to instead of hanging on until the end of the sprint to go and go and flag that issue where sometimes I think all of us have probably felt that our feedback hasn't really been heard because it's been a sprint before. Oops, sorry. So how do we start? You got to pick the engineers which have the best system and knowledge. And I think every engineering team probably has one or multiple engineers where if anyone ever gets kind of stuck or hung up on anything, they're like, you should speak to them,
SPEAKER_01
they'll be able to sort it out. The other one is you should go slowly. So there is an appetite to give everyone all the coding assistance and open it all up. But I think with this, you want a slow-faced approach. And then you should experiment in non-critical systems. So that's what I did in the November and in like the two months before the case study. I was making small proof of concept features, going and pushing it to production. Didn't really get much attraction. So if there was a bug or a mistake, it didn't really matter. And then we moved it to the thing which gets 100 million page views
SPEAKER_01
after that. The big thing here though is not everyone can drive a sports car. And that's all right. That's not a big issue. But I think everyone needs to be a little bit honest around the engineering org and engineering team. And this new era is going to be hard for a few engineers. And I think the type of engineer where they'll really strive is one who is curious. It's the engineer who, if they haven't figured it out, they're going to spend a bit of time just figuring out how something's been built. They'll be able to smash this easily. The old style of engineer which needs something really prescriptive as a spec, I think they're going to struggle.
SPEAKER_01
So in the engineering side of things, you need verifiable deterministic tasks. I think a bunch of people have spoken about this today. It ranges from everything from all the different types of tasks. But there's also things which are specific for your product or your org. So for us, it's feature flag because we do a trunk-based development. Or it could be generating the interactive elements, generating the analytics for the interactive elements. Next thing is the agentic code review. So with us, we didn't really like relying on agents to do code reviews for systems as Hine and what engineers would usually do code reviews for. We use agents
SPEAKER_01
to do code reviews for engineers to get feedback from engineers. So that's like the variable names, this doesn't fit the style, or those kind of opinionated manners is much easier to offload. And then you can remove that whole kind of emotional aspect out of it. And then you just allow the engineers to focus on the big picture. We still need people involved. But it's really heavy on the spec really heavy on the LDD, the lightweight design document, where we figure out how we're going to build this. And the thing I haven't touched on at the moment is the product. We're in an era where
SPEAKER_01
everyone's able to create anything in an hour. But a lot of these tools have the brand feel and the product feel of something that's been created by a cloud code. And so if you really want to get the best out of it, you really need to make sure that the engineering team is spending time making sure it's still feels like basically every other product from the company. So how would I recommend tackling this? You should go see your engineering development life cycle like a kind of a factory. So you should think you're in a factory. How do you break that up into each small composable element? So in a car factory, there's one thing which is building a building a building a
SPEAKER_01
door, another one fitting a steering wheel. The exact same thing happens for engineering, right? So you have the branch name, you've got creating feature flags with trunk-based development. You've got the if you build APIs with a specific software design, a pattern, you should abstract that into a composable skill. The one thing I would flag is I'm not a big fan of consuming other people's skills if they have strong software kind of opinions that are in contrast to the engineering org because you're just going to kind of end up an issue. So you just need to make sure that any skills that are getting
SPEAKER_01
a composer still kind of patching. And yeah, this is just at the moment pretty much everything is fully autonomous from the spec LDD, a ticket from the PR. We have a QA process also. So what happens is whenever we merge a PR, it automatically deploys onto staging. When the deployment on staging has happened, we will spin up a QA agent which has a look at all of the tickets that have happened, has a look at the acceptance criteria, and it will go and then a QA against that. And then if everything's at the past, I mean, great. And then if everything hasn't, it will flag what those items are. And the bit we
SPEAKER_01
haven't gone and done at the moment, but I'm aiming in the next couple months, is to then have an agent have a look at those tickets, find out where the acceptance criteria hasn't happened, and then automatically create the PRs. And so then you get into this flow where agents can basically self-heal. And the cool thing about that is it lets us do multiple things in a parallel because we're now in a position where we basically trust agents as well. Where do we still use people? I mean, I've experienced this, but they do like to use shortcuts. So we've got to make sure the security
SPEAKER_01
side of things is behaving a product fuel. I spoke about that a little bit earlier. And then the scale and engineering complexity for the task. DLDD is really meant to help on this. So I think we've all experienced where it'll make a thousand lines of code or over-engineer something. If you're really prescriptive on that engineering document at the start, you can really prevent that from happening. So what things should you go and aim for? You should start with boring, repetitive tasks. Ideally, things engineers hate because you're going to get the most amount of the buy-in from that. Remove as
SPEAKER_01
much redundant process as possible. And I know it can be strange when you've been doing something for, two decades plus to just chuck it all out. But I would ask you, what is the purpose of this meeting? What's the purpose of this process? Is it just because everyone else has been going into doing it before? Or is it because it actually helps out? Make sure your team's a personal kind of engineering culture and the patterns are encoded in skills. So if there are software design patterns that you really do, so for us, whenever we build an API, we focus on the service repository pattern. Make sure your guard
SPEAKER_01
rails are fully functional before you get into an autonomous flow. And you should start out with the best engineers. Things you should not do is to try and onboard everyone at the exact same time. I think a big reason why a lot of these things haven't really panned out is because we've given everyone a cloud code codex and given them a hackathon of like, sweet, we've done everything. They should now be able to sort it all out. It's just not that easy. Every engineering org is completely different. And every engineering org's style is really different. And you just need to go and have a slow-faced
SPEAKER_01
approach to make sure that that transition is happening smoothly. And that's why I think those small companies are at a huge advantage of the big enterprise companies because it's really easy for me to scale this out when I've got 20 engineers. It's really hard for me to scale this out if I was in charge of 100 or if I was in charge of 1,000 engineers or if I was in charge of 10,000 engineers. So, but you don't want to be too shy. You don't want to be too conservative. Because the fact is that there's a lot of other companies that are going full speed. And for us, even from our perspective,
SPEAKER_01
I felt a few months behind and I was kind of feeling it. And I think if you think about the compounding impact that's talking about at the start, that's just going to carry on happening. So, like, a few months behind at the moment might be six months behind and a few months might be 12 months behind a little bit afterwards. And then again, just take it slow. It needs a phased kind of approach and rely on your engineers to let you know whether you can scale things faster or if it needs a bit more time. I know I spoke to you about a lot of features that we built, but I didn't show you any of the features. If anyone does want to have a play around, feel free to scan
SPEAKER_01
the keyword code and then you can have a look at all this stuff. Cool. Thank you, everyone.