Adopting a Fleet-first Mindset - Andy Beane, Spotify¶

case study · Spotify · 30:28 · Watch on YouTube

Playlists: KubeCon + CloudNativeCon North America 2025

Spotify explains how adopting a fleet-first mindset and its Fleetshift platform enabled fleet-wide changes at scale.

Topics: fleet management, fleetshift, software components, platform engineering, fleet-wide changes

Transcript¶

▶ 0:00 Hello, good morning. Uh, thank you everyone for coming. Welcome to adopting a fleet first mindset at Spotify. So, who am I? Why listen to me today? Well, I'm an engineer at Spotify and I work on a team named Starfleet or Squad as we call it internally where we work on a problem area called fleet management and we build a platform called Fleet Shift among some other pretty cool things. So, what are we going to talk about today? Well, first let's talk about the challenges of managing software at scale, at least at Spotify in the past. Next, to talk about our journey of adopting a fleet first mindset, it's

▶ 0:44 first important to talk about the pillars and building blocks we've used to build fleet shift and perform fleetwide changes at scale. Then, I'd like to dive a little deeper into making fleet first changes and what this looks like. After that, I'd like to share what a fleet first mindset is for engineers building and rolling out fleetwide changes and the impact. Last, let's take a look at some real life examples of changes that are ongoing and recurring, some that are in the past, and some ideas that we're looking at as we consider what the future of adopting a fleet first mindset can be at Spotify. So first when journeying to a fleet first mindset we started to look at our

▶ 1:28 software ecosystem in terms of software components and software components simply put are any type of software entity within your organization. So its backend services its websites its infrastructure and tools its mobile and client features uh its data pipelines and more. Now, with the nearly 20 years that Spotify has been around, the number of software components has overwhelmingly grown larger and faster than the number of engineers in our organization. We have increased complexity, more teams, products, and of course, priorities. This means more maintenance, more migrations, all of which are harder to

▶ 2:10 do with our growing software fleet. So what is some of the engineering impact that all of this has had from all the software that we have written? Well, we have over 3,800 production deployments per day. And we capture over 1.7 trillion events per day, which I don't know about any of you, but numbers like these I I honestly can't even wrap my head around. 1.7 trillion. Uh, and with that, we have over 10 million requests per second of peak daily traffic. at least the way we can best measure it. And so what is the actual impact and what does all of this support and deliver? Well, Spotify has over 713 million users with over 100 million

▶ 2:54 tracks and over 7 million podcasts. And I'm I'm sure you've listened to all of them. So what do we get to? Well, maybe this is always sort of true, but this means that migrations are hard and frankly they can take a long time. So for backend services at at Spotify, where the Java language is our golden standard, alongside this, we have an internal services framework named Apollo. Historically at Spotify, a migration of our fleet of software to this new version of Apollo or to a new version of Apollo would take around 200 days to nearly complete. So what does this mean in practice? Well, it means you can only feasibly release a new version or well

▶ 3:38 at least an adopted one once, maybe twice a year. There is an enormous amount of planning that goes into this and tons of effort. It takes a long time and it impacts many teams. So migrations are hard. Well, as it turns out, all types of changes that can affect your fleet of software can also be hard. Maintaining your fleet of software becomes complicated, slow, and a huge cost that outpaces the scale of your organization. Some examples of these types of things, just to name a few, are updating dependencies, migrating to a new service, tool, or framework, uh patching a security vulnerability or resolving a production incident. It could also be an infrastructure change, something that's improving the scalability and performance of your fleet, or even provide a cost savings to the

▶ 4:22 organization. Internally, we often refer to these sort of things as engineering toil or the arduous painful tasks that you don't often want to work on. At least if I'm honest about it. Uh, and these are the things that get in the way between you and the important problems that your you and your teams are trying to solve. So, let's pretend a little bit or at least let's walk through from a high level what one of these migrations might look like. So, let's say you're on a team that owns a library, a tool, or something else that your fleet depends on. Many components use it, and upgrading it is complex. And now you're tasked with delivering and rolling out a new major version. The new version is great. It patches known security holes. It's faster and more performant and

▶ 5:06 includes many of the most requested features from your users. So, skipping ahead, we've now passed the first hurdle. We've built and designed the new version, and that's probably the funnest part. Now, we're going to document major changes, trying to highlight breaking ones, and maybe we create migration guides, trying to make this as easy as possible for teams to migrate. Maybe we prepare support resources like designated Slack channels or some sort of on call. Now, we shout at the top of our lungs as loud as we can, wherever we can, about the new version. We want to make it clear that this new version is coming and we want everyone to know about it. We know the impact the new version will have and truth is some of these fixes really needed to be shipped

▶ 5:50 yesterday. We try to set timeline expectations with teams but ultimately updating their services is in our consumer's hands and we have little visibility and control into who will do it when. So adoption happens slowly and inconsistently. Some teams start soon, some much later, some don't at all, and they need support reminders and maybe some gentle nudging. We try to handhold as many teams as we can, and we firefight issues with a fragmented organization for months or longer. So, let's now skip straight to the end. What does this migration look like when we adopt a fleet first approach? So you build the new version, then you create a a fleet shift that pushes out that change and upgrades the fleet to that

▶ 6:34 new version. Pull requests are opened automatically for your fleet of services. Teams review and merge or better yet have the PR merged for them automatically. The team receives the updates with little to no impact on them and the upgrade is completed in hours or days with full visibility into the progress of the migration without direct need from the teams that own the consuming services. So now you can go focus on something else. Maybe the more important problems that your team is trying to solve. Or maybe you go celebrate. Well, what does this look like in practice? So in the old approach, your own team pushes a new version or change. Then the owners of all the consuming services that are affected push that

▶ 7:17 change to their repos by one updating their services which may or may not require some sort of code refactoring. So then consumers have to make this change across every single one of their servers ser services again one by one uh one at a time. Now with a fleet first approach the platform team makes a new version. They push out the upgrade with a fleet shift or shift as we call it and then this is rolled out automatically to the fleet by the only team of the change. So this requires no direct involvement from the consuming teams. It's owned by the ones that are have the expertise that can push out this change to the fleet. So, as we've built fleet shift and fleet management, there are some important

▶ 8:01 pillars or building blocks that we've adopted as an organization and these are pivotal parts to our fleet first approach. First of which are our technology standards or what we refer to as golden tech. Maybe this sounds obvious, but to talk about this, we need to talk about Spotify's history a little bit. So in the early days of Spotify and like many other organizations, we started off with small autonomous teams. Teams were fairly independent and they had the resources and staffing to make their own decisions. They could choose their technology stack. It was all about pushing out and delivering the technology that they needed to push out for the organization, delivering that value as fast as possible. Over time though this created a huge challenge for the organization as

▶ 8:46 we had a fragmented technology stack which made software maintenance migration and any kind of change incredibly difficult. So the solution was to adopt golden technologies. These standards across the organization include things like libraries, frameworks, tools including known good versions. And so these standards make fleetwide changes possible. If you know what tools, languages, and frameworks your fleet is using, it makes it possible or at least so much easier to manage your fleet at scale. And this value is not just for owning teams of key platforms and services, but it's also for consuming teams as well. If you adopt Golden Tech, your software will be maintained for free by us. And ideally, you won't even need to know what's happening. To make this even easier, we

▶ 9:30 offer service templates with approved dependency versions to make it easy to build a service aligned with our golden technologies. So, I know there's probably some of you out there that love infrastructure. And as a platform engineer, I have a deep appreciation for when it works well or at least when I don't have to fix it. So, this is what we do at Spotify. Our infrastructure is defined in code, YAML files actually, and so teams can focus on the what instead of the how. This makes our infra and the way we deploy software standardized, versioned, and auditable. And it also enables programmatic changes at scale. And we'll talk about an example soon. So now that we've talked about the

▶ 10:14 building blocks and pillars of fleet management, let's talk a little bit about fleet shift and how this works, at least from a very high level. So when a shift is created and is rolled out, fleet shift creates a bunch of jobs for each of the target components and more specifically these are Kubernetes jobs. Each of these jobs clones the target repo, executes the transformations in a docker or OCI image and pushes these changes to a branch. Fleh shift watches for job completion and if the job completes successfully, it creates a pull request. Now let's dive a little deeper and talk about automerging. And I I don't just mean the box that you click in GitHub. Uh Fleh Shift interfaces with our VCS system, in our case, GitHub or really GH. And so it can clone a repo and

▶ 10:58 create a pull request. When a PR is created, it triggers a CI build of that repo via web hook. Our auto merger continually searches for eligible candidate PRs to merge. And for an eligible PR, we consider a number of different things. If all of our constraints are met, we automerge the PR. When a PR is merged, we firewatch the fleet, monitoring successful and failed deployments. And if a threshold is exceeded, we trigger an alert. And of course, when I say deployment, this can mean different things depending on the type of component. So with our software catalog integrated with backstage, we can know our fleet and using our internal check system, we can measure the compliance and health of our fleet as well as the eligibility for

▶ 11:41 certain types of fleet management. we can see improvements in our fleet and the impact of our fleetwide changes as they roll out. Now that we've talked about fleet shift, I would like to now drill a little bit more into fleet first changes, what they are, and what needs to be considered when making a fleetwide change and how we think about some different types of changes. First off, what exactly is a fleetwide change and how do I make one? A fleetwide change, simply put, is a transformation defined in an image that can run against your target fleet of software components to deliver a migration, maintenance, or some other solution you can roll out across your fleet. The owner of a given change takes the burden away from your engineering organization by rolling out this change

▶ 12:26 to the fleet directly. So, how would I go about making one of these changes? And what might I ask myself? Well, I want to know not just what the change is, but who is or should be affected. How do I create this solution? Not so I patch one single repo, but so I can automate this change to perform across any number of target components. And then I want to monitor and track success and failure and understand what this this is depending on the type of change I'm rolling out. So this leads me to thinking about what types of changes I'm trying to make. Uh there's more than one way to bucket these things, but two important ones are recurring or a one-time change. So a recurring change is something that should happen continuously or ongoing.

▶ 13:09 And a one-time change is something that you you rule out once and it's done. So some examples of these or a recurring change include dependency updates, uh security patches or security vulnerabilities, or some sort of config updates. An example of a one-time change is some sort of one-time large migration. uh some sort of platform upgrade or change or maybe some sort of deprecation. So let's now talk about rolling out a change. How do you target your fleet of components? So there are some different ways to do it. Some examples include searching your code to finding eligible components based off of some criteria. So for example, is this a Java service? Is it a website? Does it use our

▶ 13:52 internal services framework? We can use data endpoints such as in BigQuery that provide a list of components based off of some query or criteria. And of course, you can use an explicit list of components as well. We we now have an ecosystem of tools that have been created created and adopted to help perform changes on different types of components to make it easy as possible to get started. Of course, we now also have the prior art leveraging our catalog of current and completed shifts and the community of knowledge we've built over time. We can we also utilize test automation, golden standards and eligibility criteria for when we can merge and auto merge a PR. So maybe this is obvious or maybe it's not the funnest part, but another challenge

▶ 14:36 with migrations I find is with communication, continual monitoring and measuring as well as providing the context in one place. So, I don't know about any of you, but I deeply deeply appreciate when a pull request contains the context and resources I need to understand that PR and this is exactly what we do with a fleet first approach. We share the context in the pull request, including Slack, channels, docs, and anything else someone reading that PR might need to understand it. And then, of course, there's the code itself. So, you can see the change and how it is affecting your software component or components. And you have the record of that right there in your git history. As the change owner, I can see the impact of my changes by seeing the PRs merged into the fleet, the

▶ 15:20 success and failure of deployments, and if applicable, the version of some change across the fleet. So what really is the fleet first mindset and what is its impact? So it's treating our software ecosystem as a fleet of components and standardizing our fleet of components around our golden technologies. To me, it's thinking about technology problems through the lens of how do I solve this for all components instead of thinking how do I make this change in this one component or service. And so what is the impact that adopting a fleet first mindset has had. So for only 2025 and for only the first half of

▶ 16:03 2025, we've saved our engineers over 450,000 hours of effort through automated fleetwide refactoring. Over time, we've created over 2.4 million PRs, many of which have been automerged with a 3:1 bottom ratio of PRs created through automation. Let's now take a look at some real life examples of fleetwide changes from the past, ongoing, and the present, and what we're looking at now. and what the future of fleet management can be at Spotify. But here's just one quick thing I want to show you first. In backstage, not unlike our automated PRs, we try to have one place at least a start, you can go to as someone who owns components and

▶ 16:47 receives fleetwide changes or maybe you own some sort of fleetwide change. The latter of which you're probably in both groups in all reality because 3:1 bot to human ratio of created PRs. And of course uh we have a golden guide so you can understand best practices for making a migration the technical docs for the fleet shift platform and very important video resources for those of us who don't like to read everything in docs included. So let's now time travel and go back in the past. Let's say we all work at Spotify and in 2021 we wake up and we find out that our standard logging library across our fleet of Java services has a back door. That might

▶ 17:30 sound scary or unfortunate. Fortunately, we have fleet shift and a fleet first mindset and within 11 hours we were able to patch 80% of our fleet of services using log 4j. Sounds pretty good. Back to the present, we utilize the Java bomb or bill of materials which is versioned to roll out known good versions of dependencies across our fleet of Java services. This runs on a continual basis even today to keep our fleet up to date. And this is what you might see if you were looking at the Java bomber fleet shift. And next, here's an example PR from a really great fleet shift that updated database autoscalers across the fleet, optimizing and providing a technology

▶ 18:14 cost savings for the organization. The PR contains the context of who was making the change, how to reach out to the change owners, and the context for why the change is happening and more. All there in the PR description and your Git history for review today, tomorrow, and later. Here's an example of leveraging declarative infrastructure to make this change in a job in fleet shift to push that change straight to a branch. So now as we're looking into the future of what fleet management can be at Spotify and the evolution of what adopting a fleet first mindset is, let's consider a couple of different ways in which you can manage your software components. So with polyreos, your software components exist across many

▶ 18:59 many code repositories. And poly repos have some advantages, but monor repos also have some advantages and are great for making large scale fleetwide changes. So using what building blocks we have with fleet management and using this as a step toward monor repos. But before we could make this change, we needed to improve our fleet first. So we followed this journey. We first standardized our technology organization using golden technology. We have declarative infrastructure enabling us to make programmatic changes at scale and of course fleet shift allowing us to make fleetwide changes across our fleet of components removing the engineering toil and pain from our engineers performing fleetwide refactorings at

▶ 19:41 scale with monitoring and visibility. So now as we are in progress with this monorrebon migration across our engineering domains such as front-end clients, mobile clients and of course the back end, we have already seen some amazing benefits from making this migration. We have over 30% of our backend services migrated into a monor repo and we've already been able to do some things much faster than we could previously. We've safely updated Protobuff Java across the fleet confidently thanks to the full test suite of every service being run premerge in the monor repo which not only gives us confidence of a change like this for components in the monor repo but by proxy for components not yet migrated as well. We also rolled out Java 25 more aggressively than previous versions thanks to the increased

▶ 20:25 confidence from premerge test suites. So, as we look into the future of what it could be like performing a fleetwide change at scale, it makes sense to look at what fleet management has done well and possible opportunities. We saved our engineering organization countless hours. And since mid2024, around half our PRs have been created through fleet management. And this has had a huge impact on the organization, helping us save time, the organization a huge amount of time and effort maintaining our fleet of software. Like every platform or tool though, Fleet Shift has its opportunities and one of them has been performing more complex code refactorings. For example, manipulating a components abstract syntax tree or a using regular

▶ 21:09 expressions requires a high degree of specialized expertise in particular when you're trying to do this across your fleet of services. So a very popular topic I'm sure AI as it's becoming more capable of making complex changes presents an opportunity to try to do this. So we have built a small internal CLI that can among other things delegate executing a prompt to an agent. This is used within the existing fleet management tooling and it helps us generate the code declaration of our transformations but the rest of the tooling remains the same. So can we ask ourselves the question can we allow engineers to define and run fleetwide changes using natural language? And we're really excited for what the future of adopting a fleet first mindset can be using AI and of course other new

▶ 21:52 technologies. Now, what are other future considerations of ours? Well, we want to continue to improve our tooling to make rolling out impactful fleetwide changes easier to do and alongside this enable increasingly more complex changes. We want to push greater adoption, enabling more fleetwide changes to be automated, and continue to explore how things like AI, LLM, and other tools can be used in the context of making fleetwide changes and how this all fits into our fleet first approach. So uh we also have some additional great resources around these topics. Uh we have a blog post as well as a podcast or a couple of blog posts as well as a podcast episode available online

▶ 22:35 and of course in Spotify. Lastly, I wouldn't be speaking here today if it wasn't for my amazing colleagues. And so this is all of us. Thank you. I think we have a few minutes if there's any questions. >> It's a good question. Um, >> uh, yeah, of course. Uh, what kind of challenges we've experienced when standardizing our tooling? Yeah, internal tooling.

▶ 23:19 >> Internal tooling. Well, talking specifically about fleet management, I think there was an experience of a cultural change. So, we're now at a point in our evolution where teams come to us wanting new features and they want support and they're very, very eager of like, how can I do this with fleet management? But in the early days of fleet management, we worked very close with partner teams that had big opportunities for fleet management. So, things about things like updating Java dependencies, uh, keeping our data pipelines up to date and so forth. and we worked very closely with them to help try to build these shifts in the early days and then uh now we're virtually hands-off on all of those efforts. It's it's cool now because we have people coming to us about like hey can I do this with fleet management? Can we get these PRs auto merged and it's it's awesome. Does that answer the question?

▶ 24:02 Cool. >> Great talk. Um what do you do when your PRs get ignored by developers? >> Oo uh first off I love your shirt and thank you for the question. Uh so the question was what happens when our PRs are ignored by developers. So I think my the first answer to that question is when possible we have the PRs automerged. So when I talk about automerging we have a bunch of eligibility criteria that we rely on things like test automation things like that like is this PR like a good PR and we automerge them without needing the team review. and with our ability to monitor and see the health of a fleet as it rolls out and that like version or whatever that change is is deployed, we can understand like how safe and how successful that change is. So where possible automerge. I think the other

▶ 24:46 element comes in is just that clear communication and the culture that we've built that people actually want to receive these changes because truth is do you really want to update your Java dependencies? Wouldn't that be awesome if it could be done for you? So most of the time the benefit is not just for the person rolling out the change but they don't have to now make that change. So they can get that for free and they can focus on the more interesting things they want to solve. Does that answer the question? Cool. Great questions. >> Sure. Um well we have a lot of different

▶ 25:28 engineering domains within Spotify. So Java is the standard for backend services but uh Python is very popular for ML. We have Scala for data pipelines. Of course there's web which is generally TypeScript. Um all of the different clients iOS, Android and so forth. And so uh Java is one common and powerful example of this but the it's not the only domain that we have automated dependency management for. So there's ones for Python and for data pipelines and so forth. It's a good question service. How do you decide whether PR is safe? >> Yep. Uh it does depend on the situation. Uh generally speaking uh we have a set

▶ 26:11 of elig like constraints like is the PR green? So is it passing all the test automate? Is it passing all the test automation? And we also have other constraints like we try to automerge PRs when the owning team is in office. So we don't wake anyone up at 2 in the morning. Uh just because like we're in New York and someone else is in uh Europe, we don't want to do that. So we pay attention to things like that. Um if it's something that's more sensitive, uh we have a cohort approach. So the idea is that um just let's say it's the Java bomb and you're rolling out version 10. uh you might roll that out to a cohort of components that are lower risk, that are higher confidence, and you see what the success rate is by monitoring the deployments. So if all if those deployments are green and they're successful, then you can move on to the

▶ 26:54 next cohort. And so if you have some very important like essential services that you just you don't you can't have the risk of those going down, then you can see the success of that version rolling out across the fleet before uh like a higher risk service gets that change. Great question. Uh any other questions? >> What are the main assumptions for um this approach? I assume uh the presence of automated tests is one of them. Um can you summarize? >> Um the question was what are the assumptions of this approach like test automation? Um do you mean the context

▶ 27:35 of automerging or something else? Um what are the assumptions that must be present in the company to to start adopting this approach? >> Oh um >> like the requirement to have automated tests I assume is one of them. >> I I think it really depends on your organization and what you're building. So, for example, um I don't know if you worked for uh I'm just going to make up an example here, so this is totally off the fly, but let's say you work at a bank. Maybe you have really strict requirements for test automation. But maybe if you're a startup with three people and you don't have an MVP yet, uh but you've built a whole bunch of components with those two

▶ 28:19 or three people because you're using quad or some other AI tooling and now you want to manage the fleet. um maybe you can just roll out changes automatically and you don't have to worry about it. So it's I think it's sort of dependent on what your choice is. Does that answer the question? We we we have a we place a lot of value in test automation and uh using our internal check system we actually can see uh the compliance of our components in our fleet. So we have something in backstage that you can actually look at and say like oh hey these components they're eligible for fleet management for the Java backend because they're meeting uh the criteria for test coverage. So there's other tools that are playing into this as well. >> A great question. Thank you. >> Actually kind of had a followup

▶ 29:02 somewhere question. Um >> like would you introduce this if you imagine you're like a CTO of a new startup? would you introduce this from the beginning or do you think there's like a minimum complexity that needs to exist already for this for this you know to get adopted? >> Yeah, that's a great question. The question was uh if you were the CTO of a company like a new startup or something uh would you introduce this approach right away? Um yeah, I don't I don't know that I've thought of that personally. Um as a builder of fleet shift or fleet management uh I'm going to say yes, but maybe that maybe that's not the case. Maybe there is a certain scale where this is more valuable, but I don't I don't know that that scale is necessarily huge either. Cool question.

▶ 29:47 Uh any other questions? Uh are we I think we might be at time too uh any any final questions. >> Um okay, this is uh Andy, not Spotify talking. Uh I I personally would love to ship this uh but I'm not aware of any like imminent priority or like that we're doing that currently. I like the question though. All right. Uh thank you so much