Operating Enterprise Grade Kubernetes Clusters at Salesforce on Bare... Anubhav Dhoot & Mayank Kumar

case study · Salesforce · 31:30 · Watch on YouTube

Playlists: KubeCon + CloudNativeCon Europe 2020 - Virtual

Salesforce shares its architecture and four and a half years of war stories operating enterprise Kubernetes on bare metal.

Topics: bare metal, kubernetes operations, puppet, systemd, enterprise clusters

Transcript

▶ 0:00 all right welcome to the talk on operating enterprise great kubernetes clusters at salesforce on bare metal i'm anubhav dhoot senior director of engineering i manage a team that provides the kubernetes based platform for salesforce teams am joined in this talk with mayankumar who is the architect on the same team so who is this talk for this talk is for everyone who wants to operate their own kubernetes clusters especially if you're doing it on bare metal we will explain our own experience of doing it at salesforce this is really doing kubernetes the hard way as compared to the managed kubernetes engines out there we'll do a deep dive into our architecture and how we operate our

▶ 0:44 stack over the last four and a half years we have developed a bunch of war stories of issues and pitfalls that we face we would like to share with you hopefully you can learn from them as well at the end we'll talk about our current and future investments we hope at the end of this talk you will appreciate what does it take to run your own kubernetes clusters so let's dive in so here is the overview of our entire stack end to end as you can see we deploy using puppet on top of bare metal so we deploy kubernetes docker city all through puppet

▶ 1:26 on top of master and worker nodes puppet internally we configure it to rely on systemd systemd is a component on the box that is responsive to making sure the services keep running our customers or other salesforce teams who want to just focus on their business logic and not worry about infrastructure so what we provide to our customers is a github style deployment pipeline they check in their code into git along with that they also check in a simplified manifest that describes their application the ci pipeline runs it validates their application manifest packages it up and deploys it to all clusters as a goal

▶ 2:11 state the way we describe the goal state is in the form of a crd for each application as you're all probably familiar crds as a way to extend the kubernetes api we have extended the kubernetes api to add infrastructure integrations such as pki cert configuration or custom load balancing so when the goal state lands up in every cluster we have an operator that keeps listening for this goal state as soon as it gets a new goal state it'll go convert that crd into deployments so that gives us our customers ability to deploy across the world across all of our data centers

▶ 2:55 within minutes over the last four and a half years our git mono repo where they check in have already reached 100 000 commits last year alone we had 20 000 commands so it's highly used inside our infrastructure so how is the puppet module that this builds on implemented so we have built a puppet module that lets you deploy kubernetes in a very simple way the declaration as i showed with just this two lines you can say install kubernetes on my machine it takes in a flag which says do you want the master or the worker node behavior let's see what's inside this it's implemented in three layers from

▶ 3:43 right to left the right post layer uses rpm to install the binaries and as you said before it uses systemd configuration to manage the daemons so systemd is the component will restart your service if it fails that layer is then used by the node puppet class the node puppet class runs on every machine it installs docker cubelet q proxy flannel or our custom sdn and ha proxy why do we use ha proxy i'll cover that in the next slide the master profit class takes in the node puppet class so it installs all of those services in addition

▶ 4:27 it also installs the hcd as well as the kubernetes control plane that means api server controller manager and scheduler all as systemd services so what are the features that we implemented inside our puppet module first and foremost we completely automated the deployment of edge cd docker and kubernetes and we also added a bunch of configuration flags for our services important things like cubelet arguments and api server flags they're all configurable also we had to implement features just for hcd and kubernetes for example the hcd service it waits before

▶ 5:12 for quorum to be formed before it proceeds the service account support in kubernetes supports key rotation and there is one more detail that i did not show in the previous slide so we were forced to run flannel on docker but docker needs final running to start so how do we solve this so we ended up running two instances of docker in every machine the first docker instance is called docker bootstrap it runs flannel and sdn depending on the case and since flannel needs that city it also installs that city on that machine now since flannel is already running now the main docker can start and that's where we run kubernetes so

▶ 5:55 that's a neat way we solve this problem so to conclude how do we think about running kubernetes on puppet well the good things are it's completely automated it's also declarative the same thing that we like about pub kubernetes by using random start intervals on the puppet region on every machine we can get some amount of staggering across machines but what are the drawbacks this is not proper orchestration or any health mediation also the iteration cycles are very expensive on puppet like any small change it takes in a lot of effort to try it out

▶ 6:38 and lastly the way modules are composed in puppet any base module that you may not own maybe a common os library change can easily break your module and you will hear about this more in the subsequent slide so that concludes our deep dive into our puppet infrastructure now what did we have to do to operationalize this stack we knew from the beginning we have to run our control plane in high availability mode so we run multiple servers so our xcd runs in quora mode but kubernetes for example the cubelet cannot talk to multiple api servers so that's how we solve that was using

▶ 7:23 installing h a proxy on every machine so we configure cubelets to talk to the ha proxy as if it's talking to the api server and then the ha proxy will load balance across the api servers for security we hardened all our communication from service to service across machines using mtls based on our custom pki configuration also we isolated two workloads that should not be on the same machine by using node labels so we apply labels on the nodes to group them into what we call as minion pools and when we create a deployment we use a label selector to target one of those minion pools that lets us isolate which machines our deployment can run on

▶ 8:07 and conversely on the other way for each machine we apply our back that limits what name spaces they can access also we have to monitor every piece of our stack so we implemented what we call as watchdogs these are agents that monitor infrastructure and alert us we also implemented a sequel based monitoring pipeline that provides snapshot visibility and custom alerting and i'll go deeper in the next slide we also open source loop last year this is a project that provides historical visibility on kubernetes we believe there's no other tool available in the ecosystem that solves this unique problem we also automated as many of the

▶ 8:52 operations as possible for example we reboot machines when things go bad on them so now let's look into the monitoring stack for our infrastructure as i said we implement what is called are basically agents that monitor one piece of the infrastructure and emit an alert we have run watchdog itself on top of kubernetes and we implemented a framework that makes it very easy for us to add another watchdog this has features such as warm-up period so it waits for a failures to happen for a little bit of time before it alerts us so it takes care of momentary blips it also has cooldown periods so once it alerts us it waits for another

▶ 9:37 certain configurable period of time before alerting us again we also added snooze this is very helpful when you want the alerts to stop for a certain amount of time because you know you about to fix something and if you forget to fix that or it takes longer he'll come back and alert you again to remind you this need still needs to be fixed so watchdogs can alert us directly through pagerduty as shown on the top but they also emit crds these crds are coming from each of those data centers and we aggregate them using a kafka bus that lands those data into sql and with this central sql we can write sql query based alerts and metrics as well as provide dashboards for

▶ 10:20 visibility to our customers so that concludes a deep dive in our architecture next i'm going to hand it off to mayak who will cover some interesting war stories thanks anu so war stories are super fun so let's look at some of these war stories we have the first one is about perils of mounting host part this wall story dates back to the days of kubernetes 1.3 our first encounter with kubernetes itself we had made the decision to use it and we were still learning about the various knobs it offered one of those knobs was hostpath one fine day we found that every rolling

▶ 11:03 update of a deployment could result in parts getting stuck in container creating more debugging into cubelet logs showed teardown failures and volume setup failures for secrets and empty darwinians further along following a series of breadcrumbs in cubelet logs and running ls off searching through pop mount and using other linux tools we found that a partner team had mounted the root file system using hostpath inside their deployment yes you heard that right the same brute file system also has var lib cubelet which is the default path in which cubelet mounts a secret and empty disk that meant that any part that landed on

▶ 11:47 the same host as the partner pod mounting the root file system would not be able to terminate traceable so this is because that parts emptieder and secrets volume are also mounted inside the partner part and any attempt to tear down those volumes will results will result in resource busy errors we immediately ask the partner team to mount very specific paths they needed and went ahead and added additional validation the validation to allow only a limited set of paths using host paths we also went ahead and made read-only as the default way to mount host part while using hostpass is an anti-pattern we all realize now back in the days this was a lesson

▶ 12:31 learned that it can be dangerous to expose all knobs of a new tool that we all didn't understand properly using host paths can also lead to services that may work on one distribution of linux but not another so let's move on to the next war story so this is about bridge networking failures this story is also about our early days of kubernetes we saw a customer was complaining with intermittent connectivity failures with little more debugging we found that the failure only occurred when the client and server parts were residing on the same post after a lot of debugging reading through comments upon comments and kubernetes issues we learned about the new concept called

▶ 13:15 hairpin mode pretty sure you have heard of it helping mode is setting on the bridge interfaces that enables a packet received from an interface to be routed back on the same interface the help and mode did not apply in this specific case because the client and server pods were different so after some more debugging we found out about the ciscl kernel flag called net bridge bridge nf call ip tables you can see it on the screen on the slides this controls whether or not the packets traversing the bridge are sent to ip tables for processing someone had said this to zero as an optimization in the common puppet module and hence preventing the packet from

▶ 13:59 reaching high details since the client and server parts on the same host were talking to each other using cluster ip the packets needed to be processed by ip tables where the actual cluster ip magic happens setting this to one fixed issue but it was a long painful debugging we learned from this and added monitoring for this scenario in the form of a watchdog which would not only check the cctl kernel flag but also the hairpin mode for all we eats in the process we obviously gained a lot of iptable expertise so let's move on to our the third war story this is about quorum members i'm pretty sure you've heard of that word

▶ 14:42 the story is about an hcd outage we saw in one of our clusters hcd state was wiped out on two of the three fcd members and they restarted as a brand new cluster since the two fcd members were still in quorum they convinced the third xcd member to join this new cluster and replicate their empty state to their third member now the reason for why hcd got wiped out of its data is not clear but the reason for why third member joined the new cluster and lose all its state was that flag initial cluster state you can see it on the screen it was still set to new on all members if this flag

▶ 15:26 was set to existing the last hcd member would not have joined the new cluster since it was already part of an existing cluster and we could have some chance in recovering that data the xct cluster formation requires all members to be in new state initially but must be switched to existing later to avoid this scenario we use the puppet hiera mechanism to switch from new to existing in a month time frame when the new data center is ready to go live so people in the public cloud with managed cloud providers we will they will probably never get to experience this but on bare metal we need to expect this some

▶ 16:11 similar to how disk and network failures are part of any distributed systems and be prepared for it this incident showed how the flags are confusing and we started doing regular backups of our all our lcd data and also doing regular game day exercises to try such scenarios and make sure that our run books are ready for dealing with this so let's move on to our next war story and this war story is about memory lakes it's about the kubernetes cluster which was rapidly leaking pods you may ask how can you really leak pods one day our on-call reported that the apa server machines were going

▶ 16:55 unresponsive on looking at our graphs we found their memory usage steadily increasing until they would become unresponsive while debugging this we were manually restarting them to prevent the hc cluster from tipping over and lose score we decided that in the interim to we should put a systemd memory limit for our api servers while we continue to debug this further this at least stabilize the cluster since the systemd would restart the api server before it became unresponsive further debugging led to a mutating weapon that was incorrectly adding a json patch this json patch in trying to add a label dropped all other labels since the

▶ 17:39 labels were all gone the pod being created was not accepted by the deployment controller and hence the deployment controller kept on creating new pods to satisfy its desired goal state this continued until new pods filled up the entire api server host memory and crashed it so this bug in our jason patch it was his in it was hidden due to another kubernetes bulb that was re-merging the labels back and so we never saw this bag until it got exposed after we upgraded it to a new version of kubernetes we finally fixed the bug but there were a ton of learnings from this

▶ 18:22 and a lot of debugging for sure so mutating the books are a deadly weapon that must be constantly validated and can read in a cluster also adding limits is better than rendering the cluster unusable and this holds true for both hosts and parts so now we will move on to our last war story and and we have some more but i think this is all we have time for so this story is about our adventure in rolling out service accounts featuring kubernetes on bare metal infrastructure we had just enabled service accounts and puppet had begun rolling out this feature across our data centers

▶ 19:05 puppet uses a carrying mechanism for our bare metal hose based on their puppet roles for masters which includes api server controller manager and schedulers we can read the changes in one master before rolling it out to all masters one of our on calls noticed that the kubernetes parts were being deleted but were not coming up the error on those pods was no api token found for service account default we immediately recognize this as related to the service account feature further feature further debugging revealed that there was ongoing patching which was bringing down nodes one by one and hence the pods were getting evicted

▶ 19:49 and recreated at the same time the service account feature was being rolled out the puppet calorie mechanism had resulted in a situation where a master node carrying the latest puppet code got all the service account flags for api server and controller manager but the controller manager that was leader was running on an older node so the older controller manager did not ensure and you can see it on the screen so the older controller manager did not ensure that tokens for the reference service account in the pod being created exists and hence the port being created actually failed the admission check on the api server which you see on the left

▶ 20:34 this happened because the api server handling that request was a new api server in puppet can remote and was validating the existence of the token we fast forwarded that feature so that the puppet can remote is applied to all the hosts and that's all the issue we all learned that our channeling mechanism is not the best way to roll out new kubernetes features especially the ones which span across api servers and controller managers and more thought is needed so this is all the ro stories we just talked about now we are going to summarize

▶ 21:16 um to summarize rolling your own kubernetes on a bare metal non-vm infrastructure is hard and requires a lot of expertise and investments hopefully we have shared some recipes today that you can take home and apply to your own infrastructure we have invested in a fully automated highly available and secure deployment pipeline for docker at cd and kubernetes infrastructure across all our data centers which has evolved over the last 4.5 years the whole infrastructure has tight integrations with networking security and monitoring we have invested in watchdog monitoring alerting

▶ 22:00 and visibility pipeline you saw some of that in anu slide when he was diving deep into the monitoring watchdogs we have also invested in cost to serve visibility at container name space and team level lastly there is a robust on-call rotation and run books for all are on call which makes it easy for our on-call engineers to debug and resolve problems even at 3 am in the night we'd also like to mention some ongoing and future investments so we are also building a pass for containers to accelerate the adoption of public cloud and abstract enough of the infrastructure away for developers who don't need to fully

▶ 22:45 understand it we are also building a pass layer for accessing cloud resources in a cloud neutral way and we will continue to talk about them and open sources them when they are available we all earlier open source two of our projects one is generic site car injector which enables you to declaratively inject side cars meeting the need of a wide variety of our infrastructure teams this project is currently used in production by more than seven teams for critical side cars like pki secrets and kerberos and we have some interesting ideas in this space you will hear more about by the end of this year so we also open source loop last year which is our history visualization tool

▶ 23:30 for kubernetes resources and it helps in debugging and monitoring anu touched upon these in his previous slides as well if you want to help solve some really challenging problems come join us anu and myself will be here for 10 more minutes and happy to take any questions you might have thank you because you can hear us so we're going to try to answer all your questions so do we how do we do rolling update of our kubernetes clusters so as we said in the talk we do connecting we basically means we pick one server out of our five control plane

▶ 24:15 machines to run the new version if everything looks good after a certain amount of time interval then we start rolling the update to the other api servers that's why we run five api servers that gives us a way to test on one and the remaining four are still part of the quorum and if something happens to one it would not affect the others similarly for the minions we group them into a bunch of pools and then we upgrade them anything you add so i'll answer this one how do you use persistent volumes on bare metal

▶ 25:00 the answer is we don't there was i think persistent volumes are hard so most of our persistent infrastructure is not on kubernetes yet yeah i think the next one is this how do you do no recovery and bare metal it's hard i mean that's the biggest problem with bare metal is you have no way to quickly nuke a machine if something goes wrong so you really have to treat our api servers as almost like pets like if something happens to one of our api servers out of the five then we want to get it recovered very quickly similarly for our minions if we lose a bunch of minions that are affecting a customer we haven't really prioritized them that's some of the challenges running on them but i think to add to that uh we have a

▶ 25:47 node rebooter right which is basically in order to recover on bare metal we have an automated uh node rebuilder which based on some health signals can actually go and determine that the node is unhealthy and needs rebooting so that helps alleviate a lot of the challenges for diameter somebody else has to use terraform or ansible there are other teams have tried ansible nobody as we know has used it for kubernetes as you said in our talk like using puppet for kubernetes is a little challenging because of lack of orchestration so if you had to start over again we probably might not have picked puppet and might have looked at something else

▶ 26:31 that gave us orchestration across api servers if something bad happens we can stop the rolling update automatically let's see so do you use the same hardware spec for both master and workflows the answer is yes yeah it's exactly the same how do you use load balancer services so we run our custom software load balancing stack that does uh proxying across our pods so the custom load balancing stack can go talk to the api server grab the list of ips of actual pods and since we run sdn stack that gives us flat ips across pods and then metal it can redirect

▶ 27:16 traffic from the proxy layer of the load balancer directly to the parts and that's a much better way that we have found to run software load balancing on top of our infrastructure what is your solution for managing endpoints and service mesh so i don't know if by end points you mean the kubernetes endpoints um i don't know exactly what the question is but i think we we do have a homegrown service mesh solution and we are slowly migrating that to histo on first party bare metal and uh i mean istio uses endpoints and uh service kubernetes services rely on endpoints so if you can be more specific

▶ 28:00 on that question that would help and any more questions you can always come to slack after this talk ends with the slack channel hanging out there to answer the questions and how do you request for the question limit yeah it's basically we don't have a smart algorithm at the moment but we do plan to implement something in the future that auto automates our managing storage and bare metal as we said it's hard to do storage and then my room teams apartheid teams have built their

▶ 28:44 own custom orchestration using crds and controllers to manage that we do not use any dedicated storage for our cluster it's all on local disk on the developer cluster that's why you use ha then another question most of these so how long have you been running kubernetes on bare metal in production

▶ 29:28 that we are about to complete like probably five years now just started exactly around kubernetes 1.3 when kubernetes deployment was still an alpha feature and we were thinking about whether to bet it or bet on it or not but yeah now it's been around four and a half reaching five years yeah we were part of the first coupon it is nice to see how the community has grown over the last five years just exploded in front of our eyes and what else yes so you want to take this one why do you use bare metal at all why didn't you

▶ 30:12 just use stronger vms yeah we use bare metal for various reasons in our first party infrastructure i think using a vms might be much easier but we didn't have that option when we started and that's what we used what else i don't fully understand conversion hyper converged bare metals so we basically use puppet on top of that metal and it's we use a same skew for the entire data center that keeps it simple for us to build our first quality of the infrastructure we don't use any

▶ 30:56 specialized hardware and we don't use anything public cloud for our first party infrastructure [Laughter] thanks a lot for attending this talk it's different experience for us doing it live on this channel thank you