Real-World Knative: Success Stories from Production Environments...¶

Playlists: KubeCon + CloudNativeCon NA 2023

Four guest speakers describe how their organizations use Knative in production.

Categories: Autoscaling & performance, AI & ML, Serverless & runtimes

Topics: knative, serverless, autoscaling, eventing, production use

Transcript¶

▶ 0:00 greetings everyone welcome to the project K native maintainers session where you are not going to hear from maintainers this time around how great K native is but hear about the stories how K native fars in the real world I am NAA Singh I am K native steering committee member and today I have with me four guest speakers who are going to talk about what how they are using K native in the world so if you attended one of my previous sessions you will think that I'm repeating myself but I think it is worth repeating that K native is more than sever less now what do I mean by that uh in kubernetes to create and deploy a

▶ 0:43 service you have a lot of constructs that you need to keep take care of to create to configure to deploy what K native does it takes away the cognitive toil that you need to have because you run a command on a container it gives you a ready to use URL with autot TLS and also provide you Auto scaling Based On Demand with Infamous scale back to zero and that makes it a serverless platform for kubernetes as well as a simplified kubernetes for application developers but that's not all it is also event driven platform for kubernetes because it provides you the Eventing infrastructure for your all your event driven needs and with one more what I

▶ 1:28 want to say is if I can reduce it to tagline it is K native by default and kubernetes when you must so today we will learn from our guest speakers their use cases what scenarios they are using K native for and why K native and with that I am going to give it to my first guest speaker to kick it off so I'm Andrew senar and um I work for cor weave who's a GPU Focus cloud provider so HPC rendering sort of things like that and we use K native for a managed service that allows our customers to run serverless style workloads most of our customers are using K native to serve

▶ 2:12 large language models and like image generation and one of the reasons they like to use these is because of the ability to scale up and down scale to zero these are very big features that they want some of them deploy a lot of fine tunes that don't get a lot of usage so scale Zer is pretty important so con of serving provides a very simplified deployment for them and it handles the scaling the Ingress everything all inone you can set this up and we have seen some customers do this manually with like kada autoscaler and whatnot but this is so much easier for them to manage and use you get the one crd so that's the really big driver for uh the K of usage there and we have a lot of clients using it we have thousands of K Services over with lots of

▶ 2:56 PODS so I'm going to talk about a few challenges we've had through our usage we've actually had a really overall good experience but there's a few things that we've kind of learned from it and I just kind want to highlight some of those so the activators for C of serving we've chosen to manually scal them versus using the HPA we found in our large cluster with our thousands of services that that extra kind of churn of the end points can cause some less than optimal Behavior we've seen this in the past from other speakers as well uh We've increased our activator capacity again per activator because we found a lot of our services have very cyclical views and that helps it churn Less on the end points we have a lot of dashboards to monitor the the health of our cluster there to make sure

▶ 3:40 it's performing well we run a dual Ingress with K native so we run both Courier and ISO uh career is great for what we need ISO was kind of used at the time before career existed it's a little heavy weight for our usage and we've seen better scaling of Courier um that was something that kind of came through you know we've had some bugs in the past with the activators not detecting pod Readiness and whatnot we've worked through those are patched now and you know occasionally very large clusters you can run into some slowness here and there uh that you just kind of have to manage uh there's some things we've done to improve that areas where there's a slowness but overall it's been very robust on the control plan side on

▶ 4:21 people using K Services um we've have a really good experience with K native but not all our customers are experts in deployment cand services so some frequent things we see is a lot of our customers especially with large language models you know they tend to have very large containers they tend to put their models in their containers they pull their models from hugging face and they expect all this to scale up and down quickly it doesn't so we've done a lot of training and whatnot with our customers helping them you know go from long container start times well over five minutes you know down to 30 second start times or less uh our INF structure set up to kind of optimize some things so we kind of steer them towards how we have things in place

▶ 5:04 there uh other things some again not every customer is an expert at K native or how to configure the service so we've seen like incorrect container concurrency llms a lot of times you have one GPU assigned to your container you can handle one request at a time if you set your container concurrency to zero or 40 in K native you will have problems because you'll cue in the Q proxy and a lot of your requests will time out and then you'll come us complaining why do we have all these failures and I'll say well you kind of misconfigured things uh so we've seen a lot of things there so sometimes a lot of the like challenges we face with K native aren't really challenges with K native the thing it's a lot of the understanding of our customers and whatnot so we've got a lot of docs for our internal support people

▶ 5:47 to help our customers through things as well as we have public docs on our doc site for best practices for inference and serving so that's kind of how that's where a lot of the focus has been with like I'd call issues and problems they're not really issues and problems they're just getting the right understanding on how to use the tool to do the right job so I'm gonna hand it over to the next guy yeah thank you so my name is nor sareno I work for SBA um a german-based um it infrastructure and consultancy Company and um yeah I'm an architect mainly dabling at currently in open Shi of kubernetes and distributed systems in the public sector and we are using cative mainly for the Eventing component

▶ 6:32 um because we have clients who have like multiple departments and they have many monoliths which they are now transforming into Cloud native applications and this provides a big um overhead and communication problem inside the organization so we have chosen to use conative Eventing to like decouple the applications and provide like a platform software as a service experience to the different departments that are residing in the client side um one problem that we of course face is that we cannot use something that is SAS and cloudbased because we have high compliance um standards so we have to choose something that is very um standardized has Open Standards and can

▶ 7:17 last maybe decades or we can think it can last decades at least that's what we hope for and we came to the conclusion that conative is here the best fit it uses like Open Standards um like cloud events um and the interfaces and the architecture is very stable and good design so the end users that we will deploy conative 2 they just have to interact with very simple and straightforward interfaces and can rely that they always work IR regardless if they having a development experience inside the cloud for just development and then going into highight where it's aired and secured um so the goal in essence is creating a big magic event mesh as a software as a service that we can deploy

▶ 8:00 on Prem and the users can interact with through the broken trigger model um on the right side on the on the picture you can see here it's like an high level overview of what we are doing so we have multiple kubernetes clusters and um we are uh we're putting a load balancer in front of these clusters and through the load balancer we address one of our um conative Brokers to then ingest the events that are coming to the system and through that conative broker we call them the clust the C native broker we then low balance these um or forward these events to the other clusters which we then call the followup clusters and from there the projects then get their name spaces they can apply for a name space or departments and they can just

▶ 8:44 publish their workload applications into their namespaces and then also consume the conative services so for example the broker trigger model which is backed by aacha Kafka in our case um we have some requirements in regards to that which are specific to the public sector which is like we have to we have some environments which are low side we have some environments which are high side so air environments and specific compliance standards we have to agree to firewalls are a huge issue custom certificate authorities and multiple clusters many stages different data centers which are also secured in a weird way which are not usual in the um open market so um this is all things we had to consider

▶ 9:28 when the stative at the customer side challenges we saw during the buildup and the implementation were mainly with the persistence so at least back then in the documentation uh the documentation and the details about how the persistence and EG for example kafco rabbit mq is implemented into K native wasn't as straightforward as it is now so the documentation is much more better now in that regard but we had to figure out a lot of things on the go um um and that's the second point about the SP docs there were there were some issues we had to find out through looking to the GitHub code but which is our now covered as well in the documentation and for our case for example is the custom certificate

▶ 10:12 Authority which we had to inject into all the pots that call like consumers because our services that we call our deployments they all have custom certificates included into their image um on the user side and user side one challenge is we are working in the public sector you can imagine pay is not that good so the engineers that are there are not really the engineers that would be confronted with Cloud native Computing to solutions they usually have like multiple years of experience at that client and they're used in their stack so it's maybe net Java and not much out of that because getting new tools into that environment is kind of

▶ 10:54 tricky so we the onboarding part was a big big um yeah effort on our side we had to redent everything we had to go through a lot of work workshops with the users and as well the architectural Parts like um item potency at least once delivery we had to explain those Concepts very much very often um in detail so that the users understand what they're really doing um we had to create a default grafana dashboard which is currently not included in the Upstream project because this then allows the um consumer to really see on one page what is happening in their system end to end and identify for example issues in their system and then go to jiga to trace them down and

▶ 11:38 on the G up side we had some issues especially in the administration part where we deploy a lot of like broker and triggers and when we update the cluster and all the Departments are already deploying their stuff while that is happening the reconciler sometimes gets some HIC herbs and uh yeah it leads to weird issues let's say like that so we are still working to finding out what exactly kind of issues we are getting but um um yeah that's something that we are still not sure what is the reason on that end um error messages as well are sometimes not that descriptive so especially in the controller level controll plane um there might be some error messages you might see if you use it in production where you really have

▶ 12:22 to drill down and sometimes also go to the GitHub go code to figure out what is happening here because you get the go code line that exception but the error itself is non descriptive something like context Deadlight exceeded doesn't tell you much um yeah and that's kind of the challenges we had we all um face those challenges and also found a fix to this but that's just something to keep in mind on the usage size conative is very easy to use and it has simple interfaces on the administrator side um you have to be very knowledgeable in all the systems you're using be it Kafka rabit mq conative itself kubernetes how kubernetes works the programming Lang languages that are being used into context of the customer so it's very I guess you have

▶ 13:07 to have a broad skill set if you want to deploy it as an admin a recommendation from my side is load and Chaos tests um your conative deployment very uh from the start so use something like chaos smesh um use something like k6 or what we used was a hyper foil I think we use it to test it and get some histograms to see how works in your system do some back of back of the envelope math to figure out what is your throughput with your Kafka so that you get a realistic estimate of the boundaries you were going to push um if you are in a regulated environment like public sector compliance think about that in the beginning not at the end or you will face problems um and um yeah consider self service so enable the users that they

▶ 13:52 can do it themselves and just help them on the knowledge part develop experience is key um that is what we found out so make it even easier than the normal docs and tools are doing it create something that is uh unique to the customer experience or the environment that you're working in um keep an issue lock about all the infrastructure things you're finding we have a structure for that we always document root cause assumption solution and ticket link um optimize the workflows if you have shared responsibilities and create a unified observability plane above all your services especially in the event part because you're then working with distributed systems so that is the recommendations I can give and uh yeah

▶ 14:35 give it a guy try try out K native um so far for our use case we are very happy with it and um we are surprised that we could set up something like this in a very highly regulated environment so yeah that's for my part thank you um there is a use case study on this K native Eventing on cncf website this is the QR code for that and I'll upload the slide deck later so do check it out all right so I guess I'm next so my name is Ricardo I'm a company engineer at CERN I'm also in the to and the newly formed technical advisor Board of the cncf I'll give like I I guess everyone

▶ 15:19 knows more or less what CERN is about but basically we are a large Physics laboratory and we have large requirements in terms of data storage uh and also analyzing processing the data so we we keep looking at all sorts of technologies that can help us uh today and in what's coming in a few years as well uh so we've been using K native for inference for quite a while um also via copf flow um we have uh strong requirements in in in terms of U machine learning that are uh speeding up pretty quickly I'm sure everyone has been here hearing about gen llms but actually there's a lot more around

▶ 16:02 machine learning that was there uh more than a year ago uh even more than 10 years ago so these things um are just becoming more relevant but uh but they've been there for a while so um the requirements we have for this sort of services that we we run are integration with gpus this has been a big thing since uh since a while uh and also the integration with uh better ways to improve GPU utilization and efficiency concurrency uh things like Nvidia time slicing and PS make all these sort of things and for us it's very important to have a platform that uh gives us uh easy access to to the sort of things like the Nvidia GPU

▶ 16:46 operator which we already use elsewhere as well so in the in the right side you see a a picture of uh multiple models being served uh the fact that we can actually use uh even different back ends for for the serving is quite quite relevant uh K native also gives us an easy way to to manage those Services versioning roll outs rolling back as well giving different endpoints to to people as well autoscaling and I'll talk a bit about that and this is what we've been using basically in production for for a while uh there are other use cases that pop up uh that are quite interesting so I'll talk about two of them the bottom left one uh is about the basically the key uh use casee have a turn which is when the data comes in we

▶ 17:31 get what we call Raw data and we store it in a back end in this case I put cfs3 but it can actually be uh the actual storage system is not today uh s in S3 but for this data but I give an example because this is the use case we have so we would push the raw data and we' rely on events on the S3 side to trigger um um the analysis the first step analysis that would generate something we call estd which is event summary data and this would generate output that gets pushed again into our back end storage this triggers another event that gives the next step which is the aod which is the analysis object data and this is what then gets pushed to the uh physics groups so this this sort of U uh way to

▶ 18:14 do workflows uh is actually quite interesting because we can manage the the processing part uh on the server side instead of having to like republish the software to run the workflows on large scale batch systems and give users the responsibility of maintaining those so there's there's a nice use case there to follow it is not in production it's something we keep uh trying and we still don't have it figured out then the other one is gitlab C uh there's a lot of requests for to do more than what you can do uh with continuous integration uh when it's integrated the repost uh and to use those web Hooks and triggers to to uh use manage services that can be again versioned and managed elsewhere so those are the two we're looking at uh regarding challenges so again we do

▶ 18:59 machine learning one of the big issues if you've I've given a couple of talks about this as well elsewhere but we have very large images um these are not necessarily U machine learning models but just the software we manage uh image distribution is a big thing for this sort of uh serving um um use cases it's even more important because you want the the starts to be pretty fast so the images have to be either pre-distributed but if you have several versions of software like this you're talking about about potentially terabytes of data that have to be on the nodes so we've been looking for a few years into this sort of lazy pulling uh where you just instead of pulling the full image and starting the service you actually start immediately and you pull the files you

▶ 19:42 actually need at runtime and there's some nice integration in continuity I put here an example the benefits are dramatic you can see the Improvement on pulling times from minutes to just a few seconds and then the execu the the Ingress is also very important we we move a lot less data around so so this this makes a big difference um The Challenge another challenge we have is that we actually don't have a big use case for service mesh elsewhere um so we had to learnto when we started looking at it um just for the specific use case so the knowledge about uh this tool is not necessarily spread across the teams uh which makes it a bit harder to to manage uh when when we have incidents uh the other challenge we have

▶ 20:27 is that the remote serving is often done in aircap environments close to the machines uh so this is not a use case we've explored completely but it means that we have to have some sort of offline replication of of the things that need to be served now I'll finish with the the needs that we see for for our use cases in the future so we do have a lot of uh ml coming uh bigger bigger models until now we actually didn't have that big models uh we start having a lot of bigger models but also so uh the some of the ml use cases they actually rely on the container to have the model but they rely on a lot of external data that is not inside the container image which means that all these benefits that I mentioned before of optimizing cold

▶ 21:11 start um they become an issue when actually you're pulling uh gigabytes of additional data at at at start so this becomes a quite a big issue it's not something that the traditional container or oci registry really handles so it's something that the community not only K native will have to figure out uh the other one is very large models uh we traditionally had pretty small models I would say some of them start being big enough to not fit in memory on a single GPU so something that also again this has to be figured out how how we'll handle this kind of use case properly uh the other one is we want to make the best use of GPU so um making sure that

▶ 21:54 they are not idle uh that when they are claimed by a uh serving component that actually we can run more than one workload maybe on the same GPU so that we don't have them idle when when things are just hanging hang hanging there waiting for new requests even before they are cleaned up for being idle uh so for gpus we've been doing things like M with partitioning slicing slicing but there are challenges because um um unless you do physical partition we make memory sharing is not obvious so uh we are looking into things like the uh Dr in kubernetes that is kind of promising for this uh the other one is multicluster so we are by Design multicluster on premises because we have

▶ 22:39 this requirement of different types of resources that are located in different areas uh but also we want to burst and scale out to public clouds and to give a good experience to our users uh while trying to do deployments that cross administrative boundaries is not obvious uh especially because kubernetes was not designed from the start to be multicluster so scheduling is kind of still a big issue that we have to figure out so it's not again a particular issue for K native but I think it's something that K will have to integrate properly to to serve this use cases and with this I pass to thanks for um hi everyone um my name is Ado

▶ 23:22 garciaa I am an open source engineer with chenard uh uh we are a supply chain security company for those are familiar with the supply chain security space what we do is we protect you from uh to make sure that software that you inest uh is safe to run and it's it can be safely used to build your applications on top of it um so obviously that involves lots of Open Source um and as part of that uh we use K native to power all of our sources all of our services so and so I'm going to give you a brief overview of one of our services which is

▶ 24:04 uh charden force uh chard enforces is our um supply chain security uh uh control plane if you want to uh see it like that uh chard enforce performs many functions um it uh the way it operates is that it uh observes or workloads in your cluster and uh it reports back what you're running and based on that information we perform a lot of uh uh it has a lot of features to act on that uh so it can do uh from security scans in just t boms uh it it will tell you what's running where it can it also does a multicluster um multi it has multicluster capabilities and uh and you're going to

▶ 24:47 see how we do it in a little bit so those Services all of them are powered by K native servers uh serving um the most uh we uh our changer enforce practically has no um Ro deployments uh or all of our services are uh deployed as uh using uh gr native uh serving um and this includes around 30 35 features that uh include all of those uh things like uh like like the uh like those me I I mentioned before um now uh we also use um

▶ 25:28 uh Eventing uh and Eventing is where to report back to the users uh all of the all of the events that we uh detect in your clusters so for example when we uh perform a vulnerability scan on your workloads uh we use Eventing to send the notifications uh back to the user and to the console uh we also um we also uh use Eventing for the life cycle notifications of our admission controller so we have enforce scan um has an admission controller that lets you uh admit or deny workloads based on uh policies that you define in your uh

▶ 26:12 that you define uh for each of your workloads uh and you will use Eventing to communicate all of those um now inside of your closer we this is optional but we can run um an agent and that agent is uh built it's the one that observes the running workloads on your closers and reports them back to uh to our SAS and that one is built using the K native controller framework um and so we discussed talking about uh some of the challenges um inside of chenard we have uh lots of conative expertise um this uh and uh I'm I try

▶ 26:57 I'm trying to collect what I heard from our Engineers um in one single slide um and this boils down to one thing um so in in the main problem that we have with gative when running it was uh making sure that the rabit mq broker was doing what we wanted it to do um so uh the main the main problem that we faced was that uh interacting with uh the project can be a little bit slow uh we we are proud to be serving back uh submitting back contributions to the project uh but uh

▶ 27:42 the the broker is in a I don't know in in need of help uh so when trying to find more documentation when trying to uh get advice from the maintainers it has been some a little bit of a problem um so some of it has to do with the fact that uh G native abstracts a lot of things for you but it also has to make sure that the rabit mq backend is running properly so it has to take care of a lot of things but when you have to find- tun the the features in the in the in the rabbit mq back end sometimes it's difficult you have to deal with annotations and uh it can be difficult to pass them to the back end so um at this point I would like to so one thing

▶ 28:26 I forgot to mention is I have uh this year I had the privilege to serve on the K native steam committee representing my company uh in the end user seat um together with Nina and um so I wanted to finish on this land because I would like to like open uh the call for participation in the project uh so if you are a k native user um you can do lots of things to help the project starting with uh helping struck you uh we're interested in hearing from you uh who's using it and how but more importantly try to approach and see some of the uh help some of the issues that are outstanding so while

▶ 29:10 things can work very well uh we need your help to make them easier to run um so with that I present it tonight thank you for me learning about that K native was serving AIML long before they were mainstream was was the key with certain use case um but I found those end users from every part of the world like Eventing use case serving use case and everything we do have some time if you have couple of questions to our end users if you have conative related questions please find us on cncf slack but you might not have access to the end user so if you have questions for them we can take them right now but I would ask you to come to that mic over there so that we can actually hear the

▶ 29:54 questions and record them so so [Laughter] [Applause] yes all right is this thing on yes uh my question is for Norris you mentioned that the reconciler has issues with uh changes to K native resources what were those and are those things that anybody that uses K native needs to worry about um depends on your deployment I guess uh so how many so I think we have a unique com combination we deploy and all the projects deploy all their stuff with ag CD so you can imagine we are talking about maybe hundreds maybe thousands of

▶ 30:39 custom resources um that might be changed during the reconciliation Loop and we specifically have that problem usually with the Kafka controller somewhere so we think it's pinpointed somewhere in the CF reconciliation when we change stuffff um but it's very hard to debug as I said the error message are not very descriptive and you have all this um layering so you have conative the Eventing core mechanism then you have the Kafka controller then you have Kafka itself so to pinpoint very clearly what happened where is kind of tricky um so we are still figuring out what really is the reason could also be our deployment or kubernetes configuration in the end but currently

▶ 31:22 we are sure it's more conative issue yeah and since there's no other questions I'll ask another one this is to anybody um what's often touted as a as a huge advantage of K native is the ability to scale to zero um but what I'm wondering is when you get into a production environment I understand that maybe in development and in early stages this is great because I can scale completely to zero but in a production environment I can see if I have like core weaves use case this might be super useful because a customer only wants to run an ml pipeline once a day or something um but uh in larger production environments where I'm constantly running uh or doing work um is scale to zero really that valuable yeah so like some of our customers scale to zero

▶ 32:06 doesn't do them anything they're they're setting their Min scale to you know five 50 nodes five nodes 100 nodes however many pods you know they need to run their back end other customers like I said they do fine tunes and fine tunes are very specific and so they have a tendency to allow scale to zero because they're okay with that startup time uh given that they have you know hundreds of these fine tunes sitting around they don't want to be paying for all those resources when no one's using them and so they've made the trade-off decision that like they're okay with the startup time these customers have generally done good with optimizing their startup time so we're not talking about five 10 minute startup times we're talking about sub 30 second startup times uh they're okay with that tradeoff and so there are

▶ 32:50 use cases that we see some customers in production utilize it a decent bit and other customers not at all so it's it's very use case uh centered I will say the functionality in K native that allows scale to zero and allows the queuing in the activator is also very useful for burst so while your containers are spinning up especially with long container start times if you get hit by a lot of requests the same thing that allows scale to zero handles burst and queuing so like that being built in like that functionality and that c functionality helps you out even if you're not using scale the zero uh I guess is what I'm saying so it does a really good job of handling long container start times where you need to buffer thousands of requests before you have containers ready as

▶ 33:34 well thank you I'll just add if you don't want in production cases you can always keep scale to one but that was a good do we have one more question yeah hi uh my question is for Ricardo um you had shown that you have a um multi-step physics models that you run um there are a number of different ways you can um do that um y k native and is are there just specific use cases you use it k for and um right so maybe I go back to that slide quickly uh so you're talking about the bottom left is that it

▶ 34:18 yeah so that's kind of a still experimentation we are evaluating for for this sort of use case uh this is not for physics models this for workflows to to do what we call event reconstruction which is when the data comes from the detectors it needs like we have the raw data and we need to do the Reconstruction to see what actually happens happened in the collisions so this is uh some workflows that we have for 20 years in place and the way we usually do this we is we deploy some sort of batch workflows on a dag but the software that is being run uh has to be managed by the physics groups uh by the people submitting the jobs they need to describe the jobs which versions of the the software should be running for those workflows all the steps have to be

▶ 35:02 defined so but the main motivation for this is that the components doing the data collection they will just store it and this will trigger events that trigger the Reconstruction steps the the proper ones and also the actual reconstruction software is now an endpoint an HTTP endpoint where the what is running there is managed by a system administrator and can be described in in G and we can do get ups for that we don't have to teach the users how to build their workflows and which versions to use and all these things we can do as we do for services we can actually offer them a workflow an easy way to implement workflows now if this will actually be used at scale we don't know because

▶ 35:45 people are already used to to to using large P systems but it's a very interesting use case and building complex workflows with this sort of Eventing model event and ser serving is actually quite easy and you can you can describe it in a declarative way so there there's some value there I think thank you and I love the energy though uh and we are out of time I'm just going to this is our QR code you know where to find us the cncf SL Google Groups and um we have lot of tutorials there and if you're still with us if you could leave uh feedback on our session that would be great thank you so much enjoy the rest

▶ 36:28 of the keepon and safe trip back home