Don't Start With 500 Clusters: Patterns in Scaling Multi-Cluster Kubernetes… N. Bora & M. Morrison¶

Playlists: KubeCon + CloudNativeCon Japan 2026

Speakers share patterns and lessons from scaling multi-cluster Kubernetes with Cluster API, arguing against starting with hundreds of clusters.

Categories: Autoscaling & performance, CI/CD & GitOps, AI & ML, Reliability & operations

Topics: cluster api, multi-cluster, fleet management, scaling, data sovereignty, gpu

Transcript¶

▶ 0:00 Thanks for waiting everyone. My name is Nived. >> I'm Matt. So thank you for coming to our talk. It's don't start with 500 clusters. We'll be talking about patterns in scaling multi-cluster Kubernetes using cluster API. >> So the the idea for this talk started literally like two years ago when we published a blog post on adopting cluster API. And someone on Reddit asked me like what what really is the business rationale for 500 clusters? Kind of got me thinking about like what are some of the reasons why like enterprise today like would scale to like hundreds or even thousands of clusters actually. Common ones are usually like scale ceilings. Today's Kubernetes clusters

▶ 0:44 can very easily scale to like 5,000 plus nodes. 150,000 pods. >> [clears throat] >> But with growing software footprint often enterprises would scale to multiple clusters. Various flavors of isolation as as obvious in many companies. Also data sovereignty, compliance, regulations, and kind of kind of things like that. Also like lately with AI adoption like specialized compute GPU is another reason why like companies scale to different a very large number of clusters. >> So what reasons might you or should you not scale out your cluster fleet? Well, if you haven't yet hit the single

▶ 1:26 cluster scaling ceiling then there's probably a your first indicator. A single Kubernetes cluster can scale to 5,000 nodes and 150,000 pods. If you find that you're already having scaling issues at say 200 node count then maybe it's more of a tuning problem than a scaling problem. Maybe you have a a chatty controller that has an unbounded watch loop hitting the API server and and bogging down the the the system. Another reason, namespace isolation is is often sufficient. If you have soft multi-tenancy requirements, then you know, we have our back, we have network policies, we have resource quotas, which

▶ 2:10 are all very capable for isolating your workloads. Don't try and solve organizational problems by scaling out your infrastructure. All you're really doing is copying your problems to multitudes of clusters that you then have to manage. And finally, if if managing your clusters is your small handful of clusters is already a maintenance burden involving a lot of human time, then definitely don't try and scale straight away until you've automated the processes. 10 clusters scaled out to 100 clusters is just going to be chaos if you're doing manual operations on these things. >> Cool. So, I'm going to kick off with the case study. So, I want to talk about

▶ 2:54 this company I used to work for many years ago, Atoms.co. It was previously known as Cloud Kitchens. Cloud Kitchens is a multi-product company. Started off in real estate >> [clears throat] >> and restaurant technologies, and today kind of expanded into physical AI, autonomous vehicles, etc. So, as you can imagine, like a single infrastructure software stack kind of supporting different products, the footprint is pretty large. What made us unique and what can actually force us to scale out was we were very we believe in Kubernetes strongly. So, like we ran all of our software on Kubernetes, including like all of our microservices, like our OLTP databases, our data warehouse solutions, observability, everything. So, we didn't

▶ 3:38 vendor in. We're also like big adopters of the Kubernetes controller pattern. So, we had about 160 plus custom controllers that kind of controlled our infrastructure control plane. And you know, as these grow, like it starts to put like pressure on the API server and etcd. Um, and also we tracked the total cost of ownership of infrastructure as a key business indicator. So, all of our clusters were heavily optimized for high utilization, essentially. Um, so before few quick stats, like we had about 100 plus clusters. This is 2024, previous 2024. Um, and about like 35,000 daily active pods in three three region multi-active.

▶ 4:21 Um, our before cluster API our stack was primarily a few touch points in Terraform for control plane and then a home grown operator that manages the node pool and life cycle as well. But, [clears throat] because of the overhead and coordination, um, this whole process of like setting up workload ready cluster would take about a week and a half. So, >> [clears throat] >> some of the things that like forced us to grow, scale ceiling obviously, like our number of microservices was growing, our business footprint in different countries was growing as well. Uh, so that was the biggest one. So, we were hitting like the thresholds of a single cluster. Edge and IoT, like we had a flavor of this. Essentially, like if because of third-party integrations, we had to do a

▶ 5:05 lot of like last stop engineering for network last stop engineering. Um, big like um, DMZ presence, pops in like different countries, etc., etc. Um, and then the the the final trigger that like pushed us to choose cluster API was our multi-cloud migration. So, essentially, we're running all of our Kubernetes clusters on a single cloud provider. And because of a contractual obligation, we were trying to migrate 50% of our footprint into a different cloud provider. Uh, and assuming that we'll run multi-cloud for that duration of like a year plus. >> [clears throat] >> So, uh, at that point the way we landed on cluster API was we're were using like operators and CRDs pretty heavily in our infrastructure control plane.

▶ 5:49 Um and cluster API with these CRs uh makes it very easy to adopt like new ones. It fits directly into our GitOps pipelines at the time. Um we also didn't want to invent new tooling, so we were adopting a lot of cloud native tooling at that time. We wanted something that would fit into our tool set, not force us to build out more tools. Uh and last thing was we wanted to like optimize the time it takes to bootstrap our clusters. Like essentially eliminate all human in the loop processes. Uh so we want to want to have like a single change cluster provisioning and a single change cluster life cycle change as well. Um and so this whole multi-cloud migration it essentially required us to provision like 60 plus clusters within like a uh not just provision them, but like get

▶ 6:33 them workload ready and like >> [clears throat] >> optimize them within like a 3-month timeframe. So once we adopted cluster API, what did our stack look like? What were What was the process for kind of building out like a workload ready cluster? So we the first step is we had this umbrella Helm chart that uh encapsulates the whole cluster definition, all the node pools that go into each of our clusters, and also like some 40 or 45 different add-ons that we had to install before workloads can uh be scheduled on them. So we use Git, uh commit this Helm chart to Git. Um at that time we had a custom CI pipeline, and the CI pipeline will deploy this Helm chart to our management cluster. This management cluster houses all of

▶ 7:16 the uh our entire infrastructure control plane and the controllers as well. Um we eventually switched over to Argo CD, but um at that time we were still using a custom CI pipeline. >> [clears throat] >> So once that Helm chart is deployed to the management cluster, uh it reconciles into the cluster API CRs. Uh when these gets rendered, and also we had a few other uh custom CRs as well. Once these are rendered, the cluster API providers would provision the clusters on cloud providers and specific individual cloud providers. Uh and these three steps together will create like a barebones Kubernetes cluster for us. And once we have those clusters, uh we had a few more custom controllers that will bootstrap uh essentially like

▶ 8:01 multi-cluster access as enabled and all the add-ons. Um since we're not using Argo CD, we had to build like a custom controller to install all these like uh Istio, um a fluent bit, uh and a cert manager, and so on to like kind of get the workload clusters workload ready. And the last step was um tenancy. So, um our teams were using like namespaces as like a tenancy boundaries. So, we wanted to ensure that all the secrets and configurations for each application was uniform across all data plane clusters, and this was also done using um custom controller. So, um >> [clears throat] >> Cluster API is like easy to use like you could use it off the shelf, but we hit a few roadblocks. Uh the first one for us

▶ 8:45 was the provider for Microsoft Azure at that time wasn't feature complete. Um >> [clears throat] >> luckily, we had good relationship with Azure and we were able to lean into our partnership there, collaborate in open source, um and get the features implemented that we needed to uh kind of essentially like start using it. Second thing was like ACR permissions. So, ACR permissions and like we all we're also using like auto scaler profiles to custom auto scaler profiles to um uh for for efficiency. These are usually outside of purview of Cluster API Cluster API's responsibility. So, we actually ended up building like a companion operator. So, and it's like interesting because like uh we have the cluster and the Azure

▶ 9:30 managed cluster CRs, and we just like tag onto that they reconcile into like Azure cluster additional configuration CR, and then we can add additional like any number of specific configurations to it. Um so, this [clears throat] was like a like a extension of the cluster API CRs that we did. The last thing was uh immutable node field node pool field. So, like um Azure's VMSS layer doesn't allow you to modify some of these fields. They might today, but actually at that time didn't. Like if you want to switch like machine type or like a disk size, uh you'll have to tear down a node pool and like rebuild it from ground up. And that was destruct- disruptive to our workloads, uh especially in the early stages of migration. We wanted to give like a very uh like opaque change um to our

▶ 10:15 stakeholder teams. So, we built out like a custom node pool controller, and essentially what it does is like a it creates like abstraction of a node pool and hides like any number of node pools underneath it. So, if you want to change a field, what it does is creates a brand new node pool, um drains all the workloads, like schedules on the new node pool, and once everything is drained like we delete the old node pool. So, it's kind of very We can make these changes as a platform team uh without actually disrupting workloads at all. Um so, once we adopted cluster API, I mean you guys already know like no no like um adoption or migration is complete without uh catastrophic failure. So, we had a we had a node provider ID bug. Um this is a classic first adopter risk.

▶ 10:59 So, because the provider for Azure was so new, there [clears throat] was a built bug that we didn't initially see. So, um essentially every cloud provider assigns like a unique identifier for each node. Um and it's like a long string. You can see the spec here. But, within the class uh provider sorry, cluster API provider for Azure code, it was interpreting the the last suffix index as like a unique identifier instead of the whole string. And as you can imagine, this is like a number incrementing number. So, like this was not unique across node pools and also across clusters. Um and this led to uh a sev one incident where we lost about 60% of our production nodes at

▶ 11:43 once on a cloud provider. Um luckily, we didn't have business impact because at this time we were operating um we were running clusters in both GCP and Azure, so a lot of the failures were absorbed into our uh GCP clusters and following this uh incident, we essentially maintained like a fork of uh cabzy for a few months uh so we can have for a while out fast uh hot fixes. And the hot fix was very simple here actually. Um so yeah, so adopting cluster API, we had a few big wins. So, time to provision for workload ready clusters went down from about a week and a half to under six hours. We still had a few like manual steps here and there that we had to coordinate, but like uh today this this number is like very low actually.

▶ 12:27 Um one of the unexpected results we saw was the incidents attributed to any kind of cluster operation down to zero from one per cycle. So, we used to have a lot of incidents during upgrade cycles. Um this is like uh essentially leaning into automation lend us reliability as a side effect. Um and then last but not least like we doubled the fleet size uh within like a six-month period. Well, not just doubled the fleet size but also like got workloads running on them uh but while keeping like the same number of platform engineers on the team. Should I go on? >> All right, thank you Nabeel. Um I would also like to talk about uh a second case study from uh Ditto, uh a company I worked at previously. >> [snorts] >> So,

▶ 13:12 um from the uh the the uh word cloud that we saw at the start of the slide presentation, what were the reasons that Ditto had to uh scale out? We had a fairly unique product that was uh meeting different or requirements than what described for case study one. So, we needed very good tenant isolation. We needed, bring your own cloud for, deploying into our customers accounts. We needed multi-cloud, cuz we had customers on different, cloud types. And, our customers wanted, data sovereignty as well. So, and finally, version life cycle was another key driver for us. I'll just dive into, um,

▶ 13:55 a couple of those. So, our data product that we were running on Kubernetes, was managed by our own internally developed operators. And, those operators were very CRD heavy in their, their approach. So, we had a large number of CRDs that were describing the state of the data platform running on clusters. And, as we know, CRDs are cluster scoped. And, so it's one global version per cluster. And, rolling out a new version is is pretty difficult on a multi-tenant system, or doing it safely at least, without impacting other tenants. Um, the second main driver for Dedo was, bring your own cloud. Our customers wanted this data platform running in their Amazon account, or their Google

▶ 14:40 project, so that they they had control over the data. And, also they could use their Amazon, buying power to reduce the cost. Uh, so isolation in this case was a product requirement, and not an ops preference, which is why we were forced to start scaling the number of clusters we were managing. So, before we embarked on this project, our cluster management approach was using, Kops and Terraform. And, it was really like a, a two to three day effort for an SRE to bring up a new cluster. And, that was in the best case scenario. If it was a bring your own cloud scenario, where we were deploying to our customers account, it could be a week or

▶ 15:24 two of effort. Uh Kops was running in um Terraform emit mode, so it was just generating Terraform config that then Terraform was applying. So, it was the engine for uh managing the infrastructure. Terraform's great for static resources, but it uh wasn't it's not great for dynamic environments and was not meeting our requirements so for and we we wanted to go. So, we wanted to uh explore using cluster API to enable dynamic deploys. Um and other goals we had was we wanted cluster creation to be self-service so for our customers and for our own internal engineers. So, a cluster can be deployed without um you know, getting an

▶ 16:06 SRE involved or a platform engineer. And so then creating a new cluster just meant feeding in an account, a region, and a few configuration fields, and click go, and uh some system uh kicks into life and and does the work. And we wanted clusters to be deployed in in minutes instead of days. So, once we sort of got the uh the cluster API project to a good state, this is what it looked like to build a cluster at uh at Dedo. We took a slightly unusual hybrid approach with our um GitOps. We We had most of the app and cluster definitions in Helm charts and application files and Helm values files in a GitOps repo, but

▶ 16:50 the actual source of truth of the cluster definitions came from a database. Um so, we had this system we called Valet, which was just a database with a REST API front end. And when a new a new cluster was requested, it would go into this database. Argo CD has these application sets which are generators for Argo CD applications. And it one of the generators that the the app sets offer is the uh plugin generator. So, it just knows how to talk to an API and ask for for some information and then it can generate application resources based on the responses. So, it would ask for the list of clusters and for each one, Argo CD would generate an application which would represent the root of what would become a rather large

▶ 17:35 tree of resources for that cluster. Of course, one of the most important pieces was the cluster API objects themselves. So, we had the cluster or an AWS cluster object on Amazon, GCP cluster on GCP and so on and all the node node pool definitions. And they were all captured in a Helm chart that was rendered by Argo CD into our management cluster. So, once those were in place in the management cluster, naturally the cluster API operators started to do what they're best at and build it build up the cloud infrastructure. Then we had a custom operator we developed that did a a bunch of companion tasks to to the cluster build, but one of its important jobs was to

▶ 18:19 watch for the state of the new cluster as it was coming up. It's looking for a healthy control plane and healthy running applications that represented the sort of the core add-ons that a cluster would need like EKS pod identity webhook and cert manager and so on. Once it detected those things were in a healthy state, it would actually write that back to our valet database and then through the same process that we'd seen before with Argo CD's reading from that database, we were able to trickle down the state of the cluster into the Argo CD application sets and ultimately they got written to labels on resources that Argo CD was watching. So, it then knew, "Okay, we can open the flood floodgates

▶ 19:03 and application sets would generate a suite of applications that represented all the rest of the apps that were to be deployed onto the cluster. So, without any human inter- interaction there, we've built up a cluster and then told Argo how to talk to it and said, "Go Go deploy." So, the outcome was a success. [clears throat] We reduced the cluster provisioning time from days to around about 15 minutes for a bare-bones cluster um that uh within a few more minutes for our actual app- application to be be deployed on top. We had operational parity across three public clouds, Amazon, Google, and actually Azure was a work in progress at the time.

▶ 19:48 Um we had the ability to provision across cloud account boundaries, which was a requirement for our bring your own cloud um offering that we had for our customers. So, we could deploy into our customers' account with the exact same process we would use for an internal cluster, provided they had um bootstrapped the IAM requirements that we would need. And most importantly, we had minimal human in the in the loop. So, apart from the initial data entry of cluster definition, the rest of the process was completely automated um from go to woe. But, with every uh new project, there are the the rough edges and the gotchas and the the firefighting. And we had a a doozy.

▶ 20:32 Um operators that are running on a reconcile loop make cluster automation very fast and predictable, but they can also make destruction very fast and uh and this is what happened to us. We were doing a uh sort of a routine upgrade of Argo CD and it had got stuck on a uh on a CRD upgrade for Argo CD itself. And having come across this before, one of our engineers thought, "I can just delete this Argo CD application CRD and it will get um recreated by the deployment process, the upgrade, and all will be fine. Unfortunately, this was not the case. Uh the Argo CD CRD, as soon as it was deleted, naturally uh

▶ 21:14 Kubernetes started to garbage collect all of the application uh objects that were on the cluster. Um Argo CD was still running at that this point um cuz it was actually upgrading itself. Um so, it naturally started to delete all of the child resources under the applications, which included our uh cluster API resources, and no no pools, and so on. Um and naturally, the cluster API operators also uh started to make that a reality and start deleting nodes, and uh so on. Fortunately, the uh engineering question uh recognized what was happening within 30 seconds, and immediately scaled down

▶ 21:58 Argo CD to mitigate the problem, uh and reduce the damage. Even so, um we lost uh some internal clusters, and uh a customer cluster was at least partially um damaged in in the few seconds that this was underway. The fix was uh a little bit complicated. We obviously had to get the resources on the management cluster into a state where it could continue to rebuild the damaged clusters, um and then the cluster the actually bringing those clusters back up was obviously not hard because we had cluster API doing all the work along with its operators. The real uh the long tail of the recovery, though, was um these were stateful workloads that

▶ 22:41 required a uh disaster recovery process, and we had to restore from backups, so which went on. But, the uh the mitigation that we should have had in place uh before this happened, and it had occurred to me that this could happen for different reasons, um was that we we added a a gatekeeper default deny policy to our cluster. So, we had gatekeeper admission webhook controller running. If there was a request to delete an application or a cluster or any of the sensitive resources, it would block that. Um we gave ourselves an escape hatch though so that uh if we did genuinely need to delete a cluster, we were able to. So, just recapping again, we had reduced our deploy time significantly. We had

▶ 23:23 multi-cloud support. Adding another account or region became just a trivial uh task of adding a parameter and this really enabled self-service deployments of these clusters. But, what did it cost? It was a complex project. I'll I will say that much. Um I haven't gone into a lot of detail on our our GitOps strategy, but what we ended up with was a rather large uh loosely connected sprawl of YAML uh that few engineers could follow. So, I perhaps would have leaned less uh heavily into the dynamic application sets um and and dynamic uh GitOps approach and maybe more of the recent trend towards um the rendered manifest patterns for Argo CD. And the deletion guardrail initially was was

▶ 24:07 made teardowns tedious and painful, although it did help me sleep a lot better. >> Cool. So, you guys [clears throat] have probably noticed by now that like two companies, two different times, two different eras. Uh we both used Cluster API to scale out our fleet management strategy. Um one of the companies the Adams was an internal developer platform and um at Ditto, it was more of like a uh SaaS offering where we had isolated environments per customer. But, the end-to-end pipeline kind of looks similar uh and this is like a we kind of summarize this into like a repeatable pattern that anyone could borrow to like scale their fleet operations today. Uh

▶ 24:51 we always start with a source of truth um represented as like Helm charts or CRs usually in Git. Um and then the whole management cluster of pattern where our controllers our control plane controllers reside in um in a management cluster use a delivery layer uh possibly Argo CD >> [laughter] >> to to deploy the source of truth into the management cluster. Um and then any kind of like business-specific customization should be codified into custom controllers. Um Kubernetes is very extensible, so like we should leverage that functionality. All of this um should be also reconciled in the management cluster. And then the data plane uh from there on could be uh provisioned

▶ 25:35 using the same delivery layer as well on any kind of cloud provider. Um with that like we want to summarize like our experience at two both of these companies into a few learnings and principles that you can take home. Um one is automate before you scale out. Um human-in-the-loop processes fundamentally do not scale. So like if stuck coordinating manual things or making changes in like with click ops or Terraform, that's not going to scale out to hundreds of clusters. Um the second one is like build for replacement not permanence. The the cloud-native ecosystem is um very active and is constantly evolving. So if you notice that like some aspect of your fleet management pipeline is not uh covered by an open source tool today,

▶ 26:18 it is likely to be covered tomorrow. Um build custom components for those things today and like just keep in mind that like they'll have you could easily swap it out like later at some point. Uh composition over invention. So this goes into the same previous principle where um often times when you have like a specific um like uh requirements of the business like not a you you cannot use an off-the-shelf solution to meet all of your needs. So that is not a good enough reason to build a custom pipeline end-to-end. So I think try to invent uh where needed, but like most of the time should be composing using like any off-the-shelf like open source cloud native solutions. Um last but not least, uh Kubernetes is not the end product. Uh Kubernetes is

▶ 27:02 actually a framework to build and customize your your platforms to the business specific needs that your stakeholder teams have. So, we should be It's extensible. It's composable. So, we should be leveraging Kubernetes to build custom platforms. Yeah. With that, uh we'll just leave this slide up for a little bit. Um here we go. >> Well, thanks very much for listening to our talk. Um Matt Morrison at Ravin is your agentic service desk for um improving your internal operations and support. >> Yeah, and uh I'm representing a open source collective called Clean Compute. Um there's a link to our blog there. Uh we're always looking for people to contribute war stories or success stories. Um just hit us up on LinkedIn.

▶ 27:45 Thanks again for listening to us. >> [applause] >> We might have a couple minutes for questions if anyone has a question. >> Um what are your thoughts on the Uh and my uh splitting Um cloud space on application environment. So, for example, uh dev >> Okay. Um what are your thoughts on um splitting um you know, uh Kubernetes clouds based on application environment. So, for example, dev, pre-prod, prod. >> You want to

▶ 28:29 >> Yeah, I I Every time I'm trying to manage an application, I would always lean into doing that for um this I mean Kubernetes is such a complex beast. You know, you want to upgrade some control plane component or cert manager or something. You need to be able to do that uh in a lower tier like a dev environment and then a pre-prod environment and and then eventually production. So, I I always lean into breaking it down like that. >> Yeah, I think I think to add to that like um at an internal development platform like it's even more important to like split production staging any kind of like pre-prod environments. Uh we actually one of the things we noticed that like as our platform evolved um we even though even if we had like separate clusters for staging and

▶ 29:14 production people were poking holes between the two environments. And at at at later stages we evolved our like uh essentially we separated staging production as different resource groups on Azure. Uh so, that kind of isolation like helps us like make sure there's we're not going down any um anti-patterns. >> Thank you very much. >> So, uh thanks for the talk. One question I have is the uh the control plane itself the the the cluster that's you know managing all the other clusters. Like how do you bootstrap that or how do you manage

▶ 29:58 disaster recovery for that? >> Yeah, it's a chicken and egg problem. Uh exactly. >> [laughter] >> So, um yeah, so I think the the the the nice thing is that the control plane clusters we don't have to like modify them like very frequently. Um as like I think the life cycle for data plane clusters are more frequent um operations than like the data plane. So, I think at at Adams we had different um uh strategies for managing failures. So, essentially for our data plane clusters it was multi-active. So, it was like a it already takes care of a lot of a lot of failures. But for the control plane clusters we did like a a failover. So, essentially we have to like build it from scratch if that region fails. Uh

▶ 30:40 Uh, then it's done through GitOps again. >> At yeah, at Dedoose we we bootstrapped our um, sort of our top level operations cluster using Terraform and cops, which had been the previous approach. And you know, it's nice it's a nice way to do it cuz it gives you some stability at the root, which is has the the broadest sort of a blast radius. Um, from there under that we actually would use that cluster to manage uh, further management clusters. So, using the cluster operation a cluster API, uh, we would create you know, management clusters for dev, staging, prod, and then those management clusters would manage child clusters that were where the workloads actually ran. >> Cool. We're out of time. Uh, we'll be

▶ 31:24 outside in the hallway if anyone has any more questions and also be in the solutions area. Thanks again. >> [applause]