Skip to content

From Chaos to Control: Cloud Native Governance with Kyverno!

case study · DE-CIX · 37:52 · Watch on YouTube

Playlists: KubeCon + CloudNativeCon Europe 2024

DE-CIX describes using Kyverno policies to govern its Kubernetes environment, plus recent additions and the project roadmap.

Categories: Security

Topics: kyverno, policy as code, governance, admission control, kubernetes

Transcript

▶ 0:00 thanks for coming uh thanks for coming thanks for your effort I appreciate that you came to probably one of the last session of this ccon Paris uh please be indulgent it's not easy to pass Friday at 400 p.m. uh we're all tired I hope you won't be sleeping during the presentation H I hope I won't be sleeping too so we are on stage we're going to talk about using Keo to control um and apply

▶ 0:42 policies in Cloud narrative projects uh using kop policies rul is going to start and Mariam will talk about uh recent additions in KERO and I will talk about the future of KERO what happened during the past year and how K could change in the next version so R yeah thank you all right so hi everyone my name is R Garcia I work for the diix one of the biggest uh internet Exchange in Germany and um yeah before we start with that uh let me

▶ 1:26 quickly introduce uh who diig is and and uh also show you about how the environment is looking exactly before we then move over how we have implemented keano and why exactly so diix is one of the leading operators of ips in the world to facilitate the interconnection of networks so that we make uh improvements into the efficiency of Internet services globally we offer a lot of different services like uh for peering or other interconnections like Cloud connects so the um yeah we basically play a crucial role in thank you a crucial role in the internet architecture for Network providers

▶ 2:10 CDN uh large Enterprise networks so that yeah all of these networks exchange traffic directly um so that basically we reduce latency increase bandwidth and uh yeah increase the reliability on the internet uh here on this map you can basically see where we are present and where you can basically interconnect with us how is the environment looking so we basically have an application platform team which um yeah is providing centralized services like a monitoring a long-term metric storage a logging stack uh we also provide oci image registry we have a secret store codre pository and

▶ 2:54 so on so basically we try to provide a holistic stack for all the teams and then we have different teams that get manage kubernetes clusters they get a monitoring stack they get a logging environment that we take care of we provide them with incest controller with Secret store access and so on and so forth but all in a way that they still are able to do selfservice so we want the teams to create their deployments by themselves to use kubernetes techniques and um yeah all of this however in a in a guidelin way so we don't allow the teams to do basically everything they want so for that is

▶ 3:38 exactly where keano comes into place and um yeah before we go in and look into some examples of how we use Keno let me quickly introduce Keno for those of you who have not gained experience so far with Keno so what is Keno Keno is an open source policy engine which is integrating natively into kubernetes it is uh it comes with a huge stack of functions that you can use to validate to mutate to generate or to clean up resources it also has a Poss possibility to do uh image signature verification next to it it also has a rich reporting functionality so that you can uh you get an overview with a user

▶ 4:22 interface to see which policies or which validations of policies are passing and which of them are failing uh you can also expose promus metrics which is pretty good so if you want to see in a long term how your teams are behaving and uh yeah last but not least there is a huge community in Keno existing so they have already 5,000 stars in GitHub and um yeah there is also a huge um amount of policy examples on keen.io policies where you basically find predefined policies for every use case all right let's have a look what we are using it for so at day KS since I mentioned like two slides before we

▶ 5:05 provide teams with managed kubernetes clusters we want them to deploy their stuff by themselves however we don't want them for example to um deploy privileged containers or also to deploy Services of type notep pod or load balancer so that they don't expose uh any kind of like uh traffic to the Internet by themselves which is not regulated by our Network team next to this we also prevent them by the on the management of resources so let's say we deploy resources to the clusters of the specific teams like the Ingress controller and so on we don't want them to be able to make modifications on the incest controller we don't want them to

▶ 5:50 be able to change the incest controller to change the image version and so on so we basically have a rule in place that mitigates that they make changes to the manage name spaces that are provided by us next to this um yeah we also have a validation policy in place that ensures that no one can deploy any blocking pot disruption budget I mean we have seen quite often the situation that teams deploy a deployment where they say replica is one and then someone goes and deploys a PO disruption budget that says me available is also one when it comes now to the cluster update the train of the node would be blocked because the

▶ 6:33 Min available and the replica is equal so the train would not be a the node would not be able to be trained in order to prevent this we have created a Keno policy that is preventing this and I will show you also an example of this on the next slide um yeah what else do we have we also have policies in place to mutate so basically we add to every pot that the team deploys a note selector so that every pot that the team deploys ends up on the correct note pool um yeah last but not least on the mutate side we also disable the automount service account token for the default service account so in case that any team says that they want to make any

▶ 7:18 calls from their port against the API server they have to create an independent service account for this specific service and last but not least we also use Keno to generate Road bindings for user created name spaces so in our environment users only have permission to do operations on Nam spaces that they create by themselves and we use keano to create the necessary Road bindings so that only the team members of the specific cluster get access to the name spaces that they create by themselves every other name space they don't have access to they can they can read in the namespace but they can any modification and this is all being handled by the RO Bindings that are

▶ 8:01 automatically being created using Keo all right let me quickly let me quickly show you an example so um what we see here on on the slide is one of our rules that we have in place to mitigate that uh we run into a a blocking pot disruption budget so what we see here online let me just make this bit bigger here so what we see yeah okay what we see here on line uh 58 is that we can that we are running this rule whenever a deployment scale command is being passed to the cluster so whenever you say Cube CTL scale deployment D- replica uh two for example this rule is

▶ 8:46 going to check if a deployment that is existing for this scale command so that you see for example here online 62 is being is existing and if it is existing then we basically go check the replica that is defined in this deployment and next then we go make another call with this uh Json match expression to obtain the Min available definition of the PO disruption budget that is linked to this deployment so once we got these two numbers we know how much is the Min available and how much is the replica value in the deployment and next then we compare them to each other so we check how many replicas have have we defined in the

▶ 9:30 deployment and how many min available have we defined in the PO disruption budget so should the replica in the deployment now be equal to the amount of Min available this policy is going to block the operation and same is also happening if the replica is going to be less than the amount of Min available in the PO disruption budget so that no user can create any blocking po disruption budget and I mean you can see this is only a short overview of what this policy is doing you can see there are uh some more rules defined um yeah and this is basically one of our use cases we yeah we have I think currently like 30

▶ 10:14 um rules in place in every cluster which we use to prevent the users from doing anything that uh yeah we will we do not allow on the platform and yeah by that I'm done and and give over to Mariam thank [Applause] you um hi everyone um I am Mariam fi uh I am a maintainer and um I'm working as a software engineer at nurata uh today I'm going to talk about kuet is validating admission policies and how keru uh provides a full policy

▶ 10:59 man management experience to validating admission policies so yeah no thank you so uh before digging into kubernetes validating admission policies I would like to explain first uh what problems do validating admission policies solve and why do they even exist so before validating admission policies custom policies were enforced through admiss web hooks so typically when you create um a new deployment an AI an API request is sent to the API server and then this API request gets

▶ 11:42 processed and you know it gets through different phases of execution uh the first stage of course is the authentication and the authorization where you want to know um who sent the API request and and whether uh he is applicable to perform that action or not um and the second phase is the most interesting part here it is uh the admission controllers um the thing is that kubernetes run a different built-in admission controller where um it is configured in the API server uh some of the admission controllers run directly

▶ 12:27 in the API server whereas some of them uh can run as an external system and uh this external system often is often used when you want to perform complex validations or you know complex mutations and in this case you use keru uh which will be responsible of receiving the API request from the API server and then responds back to that request with either allow or or deny and as you see you know uh this can cause some delay or some latency in the cetes cluster and of course the cluster administrators will take the

▶ 13:12 responsibility to make release plans for this admission controller and for that you know for the for these reasons uh validating admission policies comes in so yeah sorry so validating admission policies provides an alternative solution for admission web hooks specifically for the validating admission use cases uh you know in validating admission policies you re you write uh validation Expressions uh in cell language and these validation expressions are usually uh evaluated directly in the API server and um as you

▶ 13:57 see it's directly ated in the API server so there is no need for admission controllers at all and hence validating admission policies reduce the latency that comes you know as a result of the round trip between the API server and the admission controller and of course it also reduces the complexity of writing and maintaining a Val uh an admission controller um yeah so yeah with that said let's take you know a quick look a quick look on you know how to write a validating admission policy so first of all you

▶ 14:38 write a policy definition where uh you specify the resources that you want to check uh that you want to apply the policies to them so for the sake of this example in the left manifest you will see that we have a policy definition that matches deployments and you also specify validation Expressions those cell expressions are going to be checked against the incoming resource so in this example um it checks for the deployment replica it makes sure that any incoming deployment should be less than or equal five and of course this cell expression

▶ 15:21 will be evaluated to either true or false so if it is evaluated to true this means that the API server will allow the creation of such resource uh because it doesn't violate the policy otherwise if you know if if it if it is VI if it violates the policy then the cell expression will be evaluated to false and the API server will will have to take you know an action regarding this regarding uh the violation of the resource and uh this is um this is important and this is how the validating and Mission policy binding is important for us so the validating and Mission

▶ 16:05 policy binding as you see in you know the example in the right uh manifest uh you will see that U you can use the binding to specify the validation action that the API server will take in case the resource violates the policy and in this exam in this example it is set to deny which means that the API server will block any resource that violates the policy and another thing is that binding provides scoping to the resources that is mentioned in the policy definition itself uh what does that mean so as you see in the right

▶ 16:47 manifest uh The Binding is actually uh uh specifies match resources field where you know it specifies that it will be applied to to all resources whose namespace labels is set to environment as a key uh this means that together with the policy and The Binding it will be applied to deployments but not all deployments it will be applied to only the deployment whose name space labels H has you know the environment label yeah uh with that said uh validating and Mission policies um you

▶ 17:31 know is used to write policy definitions and apply them to resources but it lacks some important tools that provides a full policy management experience to users and that's where keru comes in so keru is a batteries included experience when it comes to uh managing validating ad Mission policies um one of the most important tools provided by keru is the keru CLI so the CLI can be used to apply validating admission policies to resources locally or against clusters and this is uh this makes it easier for

▶ 18:15 you to um you know uh simulate the policy enforcements without altering the cluster State and of course it makes it easier for policy authors to make adjustments to the policy before deployment and you know making sure that the policy achieve their the the desired outcome uh and of course users can use the K CLI to validate resources in cicd pipelines and there are many much resources uh sorry there are many much use cases that you can benefit from the K CLI um I prepared a quick demo uh you know on applying valid dating admission

▶ 18:59 policies to resources using the kerno playground so that you can see uh how how you can you know run validating admission policies without actually deploying them in the cluster so uh first of all um on the left side we have the policy definition and here it is you know the same example as uh the Manifest we have I have ched in the slides so this validating admission policy matches deployments and uh it checks that the deployment replica is less than or equal to and yeah sorry and we have also uh the validating

▶ 19:46 and Mission policy binding okay as you see here uh the validating and Mission policy binding is used to select deployments uh with a label with the app label so again this policy will not be applied to all deployments it will only be applied to the deployments who has you know the app label and the validation action is set to deny which means that the resource will be blocked in case it violates the in case it violates the policy okay with that said you will see on the right side uh sorry yeah on the right

▶ 20:26 side um um very is manifest we have um two deployments the first deployment you know is called busy box deployment and it has a replica of four uh and as you see here it has the label app and the second deployment is the engine X deployment one and um it has also uh the the app the label app but with a different value okay uh and you know it has a replica of four again so both deployments violates the policy but one of them uh uh uh the

▶ 21:09 policy will be applied to and the other the resource will be skipped because it doesn't match the policy in the first place okay if we run it yeah you will see that uh uh only the engine x deployments uh batches yeah H you will see that only only uh the deployments the engine X deployments that matches the policy so the policies appli to them one of them fail and the other pass uh this is because there is a third deployment that has uh a replica of two but yeah I I forgot about

▶ 21:54 it yeah anyways yeah here is it um anyway uh I provided a link to this example in the slides so that you can enter you know click on the link and you can manipulate with the with the resources further more and if you have any questions please reach out to me it is just a basic uh example uh if you want more examples yeah I'm glad to help if you have any specific use case Okay uh the last yeah uh the last thing that I want to talk about is um the reporting system so

▶ 22:36 in addition to keru CLI keru provides a reporting system that generates policy reports as a result of applying validating admission policies to resources and these reports you know provide valuable insights and information about the compliance and the state of the cluster as sorry and the state of the policies in the cluster and you know teams can analyze these reports and um identify for example the policy violations and then they can take appropriate actions to rectify them uh you you can also even have an existing resources um and then you and then you

▶ 23:20 for example deploy the policy so you want to um apply this new validating admission policy to the existing resources so this is extremely benefit when you want to you know have a background scanning for the existing resources and um uh there is some you know U manifest some part of the Manifest of the policy report uh it shows us the result of applying the validating admission policy which is called check deployment replica to the deployment uh whose name space is staging an s and the result is fail as you see and why does it fail because you

▶ 24:03 know it violates uh uh the the deployment replica it should be less than or equal five but it seems that this resource has you know uh a higher number of replica uh that's it if you have any questions yeah you know I will be glad to help um next we will talk about keru Jason and chalar will uh yeah with take the lead in this part thank you okay thanks Mariam so my name is Char I'm working at neata as a staff

▶ 24:46 engineer in the same t as Mariam so I started working at neata two years ago I did a lot of contributions to K during the first year and the past year I spent most of my time doing research and Innovation to see how we could do the same thing as Kau but outside of kubernetes cluster basically uh so we're going to talk about that and that's what the two infinity and beyond is about how do we take what we add with K know working well in in kubernetes

▶ 25:29 clusters and make this generic enough to apply that to other Technologies than kubernetes so to start with uh we studied the strengths and weaknesses of keano because um there's something important to realize is that Keno is a kubernetes native project being a kubernetes native project is great because uh it works well in the cloud native ecosystem it was built for kuties so um it's definitely extremely easy to to use CNO in a ketes cluster but when you try to

▶ 26:13 use the same tool outside of kubernetes what was an advantage at the beginning becomes a problem actually so we had to first ask the question what are the strength and what are the weaknesses of K so for those who know K it will be an Evidence and obvious for those who don't know kivano it will be good reasons to adopt it so kivano is simple uh it doesn't require any coding language and then it's really easy to adop and to start with it is kubernetes native so if

▶ 26:56 you're running kubernetes it's great but it makes it not portable and uh with those attributes of course kivano grew quickly and became widely adopted um it evolved fastly but also suffered a lot from the kubernetes influence uh we added features in kivano but we were always focusing on making those features work work well in kubernetes clusters so in the end the more we added features the more we were coupled with kubernetes and the more it

▶ 27:40 was going to be difficult to move outside of cuberes and to do that we probably wanted to do something different thank you uh uh but different but but the same actually and other slides will be same but different but yeah it's kind of similar so basically we would like to keep uh what we love in our toy but making it more uh modern and more adapted to the today needs so for example most of

▶ 28:24 our users wanted to be able to use CNO policies with infrastructure as code and they wanted to write policies for their tform code they wanted to write policies in the same way they do it for k no so with just basic caml definitions in a declarative way and that's what we wanted to do um for that uh we had to escape the kubernetes jail keano was logged in while keeping the simplicity while keeping the no coding requirement we wanted to erase the kubernetes native uh attribute and make

▶ 29:08 it portable so for that we had to look at the anatomy of a Keno policy what is a Keno policy actually what is doing a policy engine we can summarize the job of a policy engine like very simply given a object you have an object in your hand and you have a set of policies the policy Eng is going to take all the rules in your policy in your policies is going to first determine if the rule applies to the object if it doesn't apply it will move to the next policy and the next rule if it applies it's

▶ 29:52 going to evaluate the content of the rule against the object you provided and provide an outcome so is it matching the expected requirements or not so and at the end you have a bunch of outcomes corresponding to the result of evaluating every wule with the given object and you can make a decision I am going to validate this object or not um for that the structure we have here is perfectly adapted but if we look on the right there are a lot of things that are very kubernetes specific like

▶ 30:34 resource kinds labels name spaces cluster roles all this is completely kubernetes specific so taking this didn't make sense keeping the same structure made sense but without the kubernetes specific parts so we are somehow uh to replace that the question is are are we going to replace it with and we are uh going to go through a few example of how we could do that by comparing uh Keno policies and the keno Jon policies on the left we have a k policies for deployment so the match

▶ 31:19 statement says any resource that has the kind deployment we can do the same in uh K Json with some something quite similar uh we will match on any object that has a field called API version and the value is apps V1 and another field that has the name kind and the value is deployment so this time we don't have any dependency on the kubernetes schema uh instead of API version apps V1 it could be type S3 it's just the partial definition of an

▶ 32:01 object if the object we are provided matches the partial definition we have here uh it will be matching and same for extrud the second uh point is how can we for example uh check that the number of replicas of the deployment is greater than two so in a policy in AO policy we used uh patterns which represented comparison operations so we go in Spec then in replicas and we applied the greater greater or equal to two and this

▶ 32:44 was replaced in uh K oison with something a bit different but a lot more flexible so we are we replaced validate with asset and we have all check that are intermediary standard and we are checking that spec and this time we take replicas we apply a comparison operator so replicas greater or equal than two and we expect this to be true so it looks a bit different but also close to what we had in the in the first place but in the second case we would have

▶ 33:28 other conditions and things like this so it makes it a lot more flexible and the third one is about uh doing uh array processing differently so we introduced a way to process arrays uh a bit more elegantly and uh yeah a bit more elegantly and with more flexibility so we now have the til in front of the data that means that um K Jon will iterate over every item applying the same assertion to

▶ 34:10 every item in the array uh so basically this allowed us to write K policies for um terraform code and in this one we have a match this match is going to match on something that's completely different than uh kubernetes resource for this one it is an NCS task definition so if the payload we are receiving contains any ECS task definition is going to check that the resource AWS ACS task definition Network Mode is

▶ 34:51 AWS VPC so if it's different than that uh the resource will be considered violating the rule and will be reported so basically uh this is the new tool K Json this one is simple it has no coding required it is portable this time and it gives more freedom for policy authors allowing uh for example to use it as a goong library so one developing is Z Tool uh yes uh sorry uh so it can be used as

▶ 35:37 a CLI tool as a goong library or as an API and finally uh this is mostly a technology this is not really a complete project because when you are talking about terraform code for example you are probably talking about TF files those TF files are not directly expressed in Json so you probably have to convert that to Json before you can process them with k or Json um so this is more technology than product there's probably something to build on top of that uh for that we created two tools uh the first one is

▶ 36:20 Keno chains so it's a tool to run end to end tests that we used internally and that we open source and a lot of folks started adopting it and we are creating an Envoy plug-in to be able to perform authorization based on policies in service maches like ISO so this is really a project this time uh and this is really outside kubernetes clusters because uh we are going to use it for very different things than before and in the end the goal is to give that

▶ 37:02 back to kiano so that keano we should benefit those Evolutions in the in the future oops sorry and yes uh finally those two tools are working uh well so the next challenge is to get this back in kivano that's it and thank you so just to recap you can join the community we have a very active slack Channel we have weekly meetings you can contribute to our projects share your feedback and we hope you enjoyed the conference uh and they keep going this year