Scaling Organizations Using Kubernetes [I] - Richard Fliam, Comcast VIPER¶
case study · Comcast · 38:12 · Watch on YouTube
Playlists: KubeCon + CloudNativeCon Europe 2017 (Berlin)
Comcast VIPER shares lessons and mistakes from scaling its engineering organization and microservices with Kubernetes.
Categories: Autoscaling & performance, Cost & efficiency
Topics: organizational scaling, kubernetes, microservices, engineering efficiency, lessons learned
Transcript¶
▶ 0:00 alright put on them petting area um before we get started here I'm going to take a quick poll the audience who here works for an organization of 20 or fewer engineers okay what about people on about a 200 engineer organization all right 2000 okay what about 20,000 and if you're in Google you should raise your hand all right so I am Richard climb I lead the comcast I / engineering efficiency team and those of us who are lucky enough to be among the founders of
▶ 0:44 a really successful organization we all face a problem scaling software is hard and communities helps a scale software scaling organizations is actually harder and Cooper Nettie's can help you scale your organization as you grow too so seven years ago I joined a three engineer research team within comcast for IP video it is doubled in size every year since that organization called Viper is hundreds of engineers today we have doubled the number of micro services we support every year we served tens of millions of customers we sell our products to other providers we deployed tens of thousands of instances
▶ 1:29 into more than 50 data centers and if you can't read that architecture don't worry you're not supposed to like oh good documentation it's out of date anyways and I'm not Q today to tell you though about all the wonderful things we do because that would be very educational or entertaining instead I'm here to tell you all the wonderful ways you can screw up growing your organization's because growing organizations as I said it's hard really hard and hopefully I can share some insights on how to fix them I'm going to start a little bit with our story all numbers are approximate so when Viper released its first real customer application we followed a
▶ 2:13 fairly familiar pattern we installed Hardware in a data center set up a sophisticated virtualization platform created some VMS on it and started up copies of micro services it takes about five engineers no big deal so customer demand spikes for our product we rapidly set up more machines and more sites anyone who's done operations on hundreds of physical and thousands of virtual instances probably knows one thing something has broken all the time at every hour of the night and day so what do you do when your traditional technology organization you hire an ops team comcast has been operating
▶ 2:57 technology for 53 years now we know how to hire Ops teams you train them and you're off to the races so now you're at about 15 engineers take it to another year later for us customer demands spike again our ever industrious dev team has added two new products and another development team now we need thousands of instances running in tens of data centers deployment has become a huge problem with too many manual configurations many deployments have missed their targets and left things and broken and hard to debug ways configuration drift is a huge
▶ 3:39 issue so time to set up puppet hire a few people to manage it and run that and that's what we did so this is about a 50 engineer organization and Viper take it another year and you're seeing the effects of doubling the business has asked for a huge feature set so a very aggressive roadmap and a sophisticated and entirely new product line called cloud DVR we now serve millions of users every day our products include dynamic advertising in analytics pipeline client-side players and a home gateway at this deployment at this point
▶ 4:24 deployments have gotten risky even with an eye toward simplicity we have a very big set of products that are interconnected and many things in the stack rely on each other subtle changes in the scaling of our operate scaling or operations of our AP is mean outages only the developers themselves can troubleshoot which is hugely problematic so ensure we have a quality problem particularly a multi component integration quality problem so what do you do when you're a traditional technology organization you hire a dedicated to 18 / end and integration oh and you need to train them all and on what they'll be working on so we're not
▶ 5:08 about a hundred engineers including our QA folks one year later we're now so successful people in the industry are buying and deploying art components instead of others we need an Operations team just to handle our SAS customers not just our own internal infrastructure we now run tens of thousands of virtual instances in more than 50 data centers are internal and external customers are demanding new features and pushing hard on our ability to even scale or software the sales team is delaying onboarding new customers because we simply can't keep up with the pace necessary for deployment all products now have large
▶ 5:53 engineering QA and operations teams new hires must receive information about all technologies and products and that information about new technologies and products that needs to be disseminated to everybody in the organization for us to work effectively so now at about two hundred engineers so we're seeing a problem here it's hard to make 200 people as effective per person is 20 when they're all working on the same technology stack so why is that well innately three big thing this matter first communication is n by n second
▶ 6:37 organizational units will become boggled bottlenecks and third organizations become coupled so I'm gonna have some fun with math here I'm going to write a recurrence relation for organizational communication assuming a complete graph that's one in which all nodes are connected by an edge for those of us who haven't done this in a decade we're going to try to figure out the flat form of a recursive equation will call this update function U and its input n will be the number of people in the organization for my organization to stay in sync on changes with any people I need to communications with everyone
▶ 7:22 except for myself this adds the term you see on the right there or mobile and 2n minus 11 communication of the change and one back confirming the impact or the lack thereof so every other team when going through this change needs to do the same thing and this adds the recurrence term u of n minus 1 so we obtained the recursive relation for our organization update function u of n equals u of n minus 1 plus 2 n minus 1 and if you if you remember the master method and you apply it here you get n squared so changing anything in our org
▶ 8:07 and validated unit to impact is Big O of N squared and that's Big O because that's for a complete graph if we have a we have a less connected graph we can do better than that so why is that important because rapidly our ability to change and innovate becomes tied to our ability to communicate and trained members of our organization new features deep in the stack must be communicated and tested with other changes in operations and QA trained on its use so the more connected the graph the more difficult it becomes to change the more people you must work with and inform the
▶ 8:52 higher of the bandwidth of that communication traffic and this is a nonlinear relationship right it's N squared even if we do that successfully the volume of that information is going to exceed the ability of the individuals in the organization to process it which encourages us to bank high level release notes to batch changes and that significantly increases our risk which decreases our ability to deploy which decreases our ability to be chained to make change to be effective so now worse our velocity of now so awesome it's breaking QA they can't test component integrations when we are to
▶ 9:37 meeting three times a day put products with major features these cycles are going to run long and also is going to have trouble deploying new versions oh and the port infrastructure teams that support this everyone needs an EMS and hardware from them all of the time but those VMs when their Idol our pain to manage and keep true up huge chunks of resources and they must be continually managed by those teams even the lab team in this situation has trouble keeping up with developer demand this is because our organizations all rely on someone just as your software becomes coupled so
▶ 10:21 our do our organizational units the more complete that connection graph the harder it is to do anything an organization's ability to make a change is limited by the ability of others to change this decreases the ability of in unit to take effective action it's also worth noting that at no point or any of these choices poor when we made them look if you're running a single monolithic application on a few tens of boxes puppet and a small ops team makes tons of sense if you're integrating five or six components just hiring Q a friend and validation makes perfect sense
▶ 11:06 insertion sort is faster and really small data sets the xkcd timetable on automation explains this pretty well until you have a lot of people automating small tasks doesn't make sense now I could probably pick out a few a DevOps evangelist out there that are jumping about about to jump out of their seats and say developers should operate your software then you shouldn't have to train operations and fight deployments but we include local variance there are tens of thousands of channels being broadcast to literally millions of users all day
▶ 11:47 every day in the comcast footprint we do cloud DVR recordings in the tens of petabytes per day heck we have a team that's solely responsible for replacing failed hard drives and that's not a joke here's the thing if you want your developers to operate what they build they have to be even capable of it puppet modules and VMS doesn't make an operable infrastructure that's like running thousands of machines with your only capability being to edit in a dot B scripts don't get me wrong puppets a great product but it's not sufficient for that kind of scale it takes a
▶ 12:32 five-man team with an organization to make a component that millions of people tens of millions of people use they can't operate it without the right tools and this is why I'm talking here today you see the thing is when your organization really needs to scale itself you overhead a people communication of project management of training of integration of anything really escapes the ability of any human process to manage all these problems i just described are not just about scaling our software they're about scaling ourselves scaling the output of our work and scaling ourselves isn't about the amount of work we do if we double the amount of work we do that's
▶ 13:17 not that big of a deal when you do to scale the effectiveness of the work we do and simply being able to make big changes also is it enough you need to be free to do so we need to remove the limiters on our ability to act and increase the effect of those actions which is why M by M communications bottlenecks and organizational coupling are so bad they all limit our ability as engineers to act so coop can help us here the khoob website refers to itself as a container orchestrator I think of it as a distributed operating system it helps
▶ 14:01 abstract us from the hardware right it provides sophisticated multi user permissions especially with 1.6 so let's talk about solving that huge operational overhead the biggest reason we need a dev ops team for an application is simply the number of instances we need to care for no development team could interact with all the necessary machines in an efficient manner sure we could register them all and zookeeper provide necesita to do automated routes and pups have puppet enforcing their changes happen but that's all stuff cube provides to us for free cooper Nettie's abstracts those thousands of boxes and tens of thousands of instances of micro
▶ 14:45 services again however simply being able to direct those thousands of instances isn't it enough to enable our developers to operate their own software remember we're in like 50 data centers and actually quite literally hundreds of thousands of micros instances of micro services running they also need to be able to introspect the running state of that system in near real-time just like an operating system I need not only PS but top iostat and journal could help so in Cuba Nettie's PSN system cuddle functionality is effectually handled by
▶ 15:26 cube cuddle so what's iostat is griffin out or sis big or CC adviser there's tons of great stuff out there for this without journal cuddle that's your elk or splunk stack and they just keep saying cuddle on this slide so what about estrace that is something like open tracing for us or Zipkin but it's not enough that those tools exist they need to just work I don't need to modify my programs to use iostat it's part of the platform it's well documented and that's a key both up taken effectiveness any organization common easy-to-use tooling that's
▶ 16:10 available to everyone and just like UNIX I also need sophisticated permissions and process isolation I can't operate independently of others on a shared platform if I could accidentally let's go with accidentally forth on the system that means i need you guessed it resource limits and cuban the cloud native environment provides these it provides us a data center operating system okay so now you've given our developers a platform they can use to operate our system but what about deploying new ones what about meeting the ever-growing number of customers who use our software let's look at how to
▶ 16:53 solve this it Kiva's the Linux of our data center I need an RPM and young for deploying and managing installed software if your infrastructure of the service is extremely simplistic that's a really enormous task and trust me I've tried to do it if your infrastructure as a service has a complete well thought-out IEP I like you that's very doable so organizations sit down to write these for ourselves we needed a way to avoid concrete versions of a whole micro service system that can be tested upgraded and migrated online so we call those two items Geronimo and ticklish internal the comcast but we're
▶ 17:37 hoping to open source soon we have Geronimo it interacts with the queue API to store the versions of deployed software track across clusters what software is deployed and abstract environmental specific configs Geronimo via the cube API provides automated deployment and rollback hooking into arbitrary develop validators and those are defined in what we call our Geronimo file they can monitor production production metrics run smoke tests upgrade database and well anything that's what arbitrary means now that we have a way to package and deploy software in a repeatable manner dalinar p.m. we can vastly simplify our
▶ 18:20 QA and release process so let's be sized on that for a second the cube API lets us label deployed pods and software so in those labels the tool writes the version the system we expect you saw that in the screenshot here to verify that the things are what we expect them to be we store a cryptographic hash of all related JSON items like replication controllers endpoints etc in the system annotations which annotations are not mutable at runtime each system is kept in a Geronimo file which is a wrapper around the cube format it describes the environment independent format of the system remember I said we're in over 50
▶ 19:04 data centers and that has to do with pushing video very close to the edge an environment file has been combined with the system file and it contains environment specific configs things like replication counts secrets for that environment endpoints in that environment that are specific etc so when Geronimo goes to upgrade a system it stands up a new name space with the new version it didn't totally documents pods from the old namespace and add them in replication controllers to the new name system so this way we can roll whole develop micro or nano service systems at once in a repeatable and
▶ 19:48 testable fashion but what if we want to install whole groups of software that are dependent on each other again we need a young leveraging the cuban Nettie's api if we are developing to cushion to integrate these into digital systems in detestable services it like yummm resolves and installs all the tendencies of a system based on version indeed like yums group install it becomes possible to simply install a linear video line that consists of quite literally hundreds of components and then we'll also warn you statically as you build the system
▶ 20:28 if dependencies are not satisfied so to cushion begins with a component that's a whole artifact like a docker container it layers on this the concept of its system as in a geronimo system which is composed of those components that might be something like the set of micro services we use for cloud DVR recordings each system is assumed to be a flat working namespace and a has a methodology to deploy like Geronimo those systems are then linked together to form a service the service is something like video on demand and video
▶ 21:11 on demand is a really big thing it's advertising systems it's video back-end systems configuration systems a lot of things and those are managed by huge groups that are disparate from each other and usually often in separate namespaces with separate permissions so just like an RPM can provide and require certain files we allow systems to provide and require configurations and linkages this lets us statically determine the wiring of a system that's kind of like if dr. compose was applied to coover Nettie's namespaces we also allow it to link and compose arbitrary
▶ 21:53 services things like cameras for playback testing physical hardware databases that we can't get into cube yet and those kinds of items so notice what I've described here also enforces common best practices for configuration and deployment throughout the organization we no longer have 10 teams with their own puppet modules or even just their own handwritten configs new users to agro-based no longer need to learn all technologies from their peers or read wiki's with those out updated architecture diagrams the code base is becoming self-describing if you want you just grab the technician file for linear
▶ 22:38 and we'll describe to you not me how everything is deployed but how it's linked together for something like video on demand so the amount of necessary communication in our organization drops dramatically and as all the developers here I'm sure would love the amount of necessary documentation drops dramatically the system is self documenting with this I can also auto deploy from source code changes allowing QA to integrate with the exposed end points and write service level tests that describe the expected behavior of our system not just of our individual components and as I said we have hundreds of those which means i can
▶ 23:21 safely deploy my components as i finish work on them to QA and just assign the card on words for them to write integration tests for the end-to-end system which they'll see when all components involved in that new feature have finished their work our infrastructure team no longer has to work on the behest of each application came with cube to they deployed bare metal provisioning that provides us with the capability to bootstrap clusters in a single go our crew vanetti self deploy onto hardware as it's plugged in as infrastructure has changed rebooted or otherwise keeps ability automatically utilize or withstand failures utilize
▶ 24:04 the new equipment or withstand sailors what's these groups move quickly without necessarily notifying and working the teams which run on their stocks think about a AWS for those of you who use it had to notify you every time they reverted a note in this new world for us they're free to focus on owning and operating that infrastructure they develop an independent dev ops team from bare metal provisioning I'm going to deep dive on that per second boxes only need to have power and network cables plugged in they kick see boot send up a hardware config which returns a boot image and triggers work for a lot of
▶ 24:48 automation to auto join the clusters with the appropriate labels applied even better these cloths images are not stored there in memory that means they're immutable which is important because it means our test environments match production no point in testing something that doesn't match production and drift of the configuration of the underlying platform is much more difficult to have happen this process decouples our platform teams from our hardware deployment teams how would the point it doesn't even have to notify them when it's done HUD will just appears in our platform inventory as it's powered and cabled in think of the power of that compared to I plugged it
▶ 25:32 in please go install the OS and set up the network correctly in the DNS and all of that now we just come up so to D couple our platform teams from our software teams we need pop forum updates to be low risk and repeatable and frequent so how do you accomplish that cipd the same CIC d platforms the deployer software now deploy our platforms first deployment is done to test an environment the two a test environment that's running a production like config and production like applications as that deployment runs a sophisticated and aggression integration test suite runs it does things like verified doctor doesn't hang offer those
▶ 26:17 of you who've run into that networking works no performance degradation have happened logging still works etc so when that piece of continuous integration is done we begin the continuous deployment part rowing upgrades begin in production we again run that integration seat sweet as we deploy production and we also have can hook into production metrics and auto back out deploys that are failing or causing applications underneath us to die so that makes her platform teams capable of deploying freely and often and with confidence they've also provided features like user allocated high performance load balancers for our platforms for details on that you should
▶ 27:01 go to my colleague Kim's talk tomorrow in this new world teams need far less communication about the new releases the register contains all that information this makes what communication we do have far more efficient on group between the groups it's much more focused on the technical operation or that technical details rather than the operations of the components everything I have described here is about decoupling the operations of these teams at an organizational level and giving those teams a platform to manage their own large complicated distributed systems as easily as I can administer my local
▶ 27:44 linux box having also encoded the information about the current running state of the system in the system it becomes self discoverable and self-describing so these are the tenants the drive our ability to scale teams discovery and dissemination of actionable knowledge the ability to create impactful change and the safety and the freedom and the comfort to do so we want to empower people provide them with independence of action in unity of direction to do that for the organization operating a large distributed system we must have a complete platform a platform that is
▶ 28:28 easy to use and transparent in its operation a platform that automates as much work as possible from the users but lets them configure it as necessary from process management to metrics to deployment and upgrades through teams will also focus on decoupling from others acting is complete platforms themselves from other teams which was mentioned today in the keynotes as our infrastructure team has they can dramatically increase their effectiveness they no longer need to continually work and do that pole with everybody else around them and this is
▶ 29:11 why cube is such an effective tool for scaling and organization because it allows teams to operate independent of each other allows us to provide simple declarative definition nervous system and it's sophisticated yet simple api's make building tooling to meet your own needs very simple and that's that's very key the cube api's are a huge part of building a successful organization just as the Linux kernel api's are a huge part of building successful Linux application all right thank you for your time and any questions so most of the data centers are actually
▶ 30:13 used by all teams contests as I mentioned we have a lot of users and we also own the network so it's in our own best interest to push that video traffic as close to the users as we can so most of the data centers like actually all teams deploy to all data centers so we don't now there are what we call national data centers and we don't have obviously to keep 1.6 Khattak stuff deployed yet that's a that's a major portion underway we also have as I mentioned done deployment of tens of thousands of puppet VMs that's run by our operations team and we're slowly making the change to try to move as much
▶ 30:58 as that possible into cube those operators can focus on operating the system more like a whole system and less as a little services any other questions but the spoons of platform inventory would even joint will use so we don't that we don't use an inventory tool today for platform inventory to cushion acts like a young server in that it tells you what what components are available to be deployed as far as it also and this is where it's not like yum it actually is probably the piece that managers platform inventory the most
▶ 31:42 though those are usually individual components unfortunately most of our platform most of the platform inventory stuff out there wouldn't work with what we do like I mentioned via physical cameras that do playback testing we have transcoder hardware that's their unique and and actually hardware so we can't just say oh this is you know a sequel database or something like that any other questions yeah so we we always had likes plonker at spunk the big changes we use cystic internally a bunch the big change is
▶ 32:27 instead of going in setting up a vm which in a lot of our instances like we're running packaging on a single channel even the single core vm is way too thick for that so we would deploy tons of applications there and then trying to slice that out becomes very very difficult is when you change to a kind of cloud native environment where you can really focus on a specific pod doing a single operation in a single task setting up that monitoring and alarming as they were talking about today in the prometheus you know becomes much simpler the other part with logging on the other side of this though is it's
▶ 33:10 worth noting that you end up generating so much logs in these systems particularly from particularly from micro services that trace each other most people at that scale run some sort of sampling rather than rather than logging every operation because eventually the system generates more metrics than anything else and that's all you spend your time doing and I talked to folks at other very large companies who all say the same thing it's it's mostly just six statistical sampling at some point even even with things like Prometheus behind you I will
▶ 33:54 thank you for your presentation not everything can be put into containers and what do you think about it this is the first question and the second question are you using OpenStack in your infrastructure or something for what virtualization platform okay so first question not everything can be put in containers and that's absolutely true we have onions of products that don't fit in containers I mentioned transcoders and we tens of thousands of those they don't generally fit well in containers because they run on asics the way the way you deal with that is try
▶ 34:39 your best to make it look like containers which is what you do with puppet and VMS at scale anyways puppet enforces that it's a container it can't be changed and you have vm images that come up and come down in auto scale and deploy there's not much you can do beyond that and then try to make it look like containers for the stuff that can't be put in containers the other thing we we face as i mentioned collision you can't put an iPad in a container I need to physically watch the iPad and I can't like scrape the screen of what should be playing because that would break our drm and I need to make the ER dr m's working
▶ 35:21 so that that kind of stuff you write managers on top of again your own little ones and try as best as you can to integrate our infrastructure your second question was sorry OpenStack so comcast has three separate virtualization platforms internally and it depends on the application for the video stuff which is what Viper does we tried it's been as much of that on bare metal because that is all I owe bound and virtualization platforms not always the best at i/o bound tasks like the reason our initial deployment was tens of boxes
▶ 36:03 is because egress even in the smallest site was more than 10 gigabits a second so more than a single Nick would normally run in really large sites it's hundreds of gigabits per second so running that off of virtualization platforms is a huge overhead which is why we have so many bare metal provisioning and as I mentioned Kim's Park actually goes in a bit into depth on on what we do for load balancing and stuff for that we do have open stock and we do have these fear for virtualization of more kind of normal
▶ 36:45 applications and those are run by our clouds we have an internal cloud came as well we're very poly got I think we have least six or seven deployed languages and we don't actually were not like a lot of big organizations we don't declare official languages so people use whatever we have Scala we had Java we have go people out to go we have see we have c++ we have it's it's whatever those teams want to use one of our big focuses and this is part of my job
▶ 37:29 developer efficiency and effectiveness is trying to make sure our developers as happy as they can be and a lot of that's about giving them the initiative and the right to choose the right tool for the for the job and that really depends and it's like I can't dictate that down to them which is part of the reason containers are great because we don't I don't have to as long as the container runs I don't care any other question sounds like we're done if you guys have any questions you want feel free to approach me afterwards thanks for your time