Skip to content

Deep Dive: Auth SIG - Mo Khan & Matt Rogers, Red Hat

concept · 39:08 · Watch on YouTube

Playlists: KubeCon + CloudNativeCon Europe 2019 (Barcelona)

SIG Auth deep dive on the Kubernetes authentication and authorization request flow and authenticator semantics.

Categories: Security

Topics: sig auth, authentication, authorization, rbac, request flow, audit

Transcript

▶ 0:00 all right thanks for coming everyone this is a cig off deep dive we're going to be covering the kubernetes authentication and authorization subsystem I did okay so who are we my name is Matt Rogers I'm the senior software engineer with Red Hat my co-presenter is makan and a software engineer at Red Hat and we both work on open shift on the auth team mo is also a sig auth co-chair okay to start with an

▶ 0:42 outline we are going to cover the request flow the request handler registration the the request context request metadata all of the items in the request flow we're going to go over the authentication piece we're going to look at some of the Authenticator types we're also going to cover the Authenticator Union semantics how the authenticators interact with each other and the same thing for the authorization of peace so this diagram shows the flow that an

▶ 1:27 incoming request goes through the HTTP request starts off by being converted into request info metadata and it's passed to the authentication layer the authentication layer authenticates the request and passes it to the audit layer where there might be a log generated somewhere about the request after the audit layer comes the impersonation layer this layer we're not going to cover this in detail but this layer checks if there was the desire to perform an action as some user if so then there's some check to see if the requester has permission to act as the user and assuming that that checks out

▶ 2:11 the request is modified so that the action is performed as the requested user so after impersonation is authorization here the request is authorized and resource rest is where the request finally hits the REST API or the business logic at the end and if there's a failure at any of these layers then the requests flow stops so the authentication authorization framework is implemented as a series of HTTP server handlers so if you're familiar with go you've most likely come across this code or written some variant

▶ 2:53 of it you define your handler function that might that might do something with the request and write to the response and then you register the hand the handler to handle under a specific path on the server and then you you start your server so since the request has to go through multiple layers as you saw from the earlier slide processing is implemented as a series of wrapped HTTP handlers the registration function which the signature here you takes an existing Handler and here's some configurable

▶ 3:37 data item that defines the behavior of the handler and then it wraps that existing Handler and returns a new handler so this slide shows some example code to display the ordering of these wrapped handlers during registration the handler that implements your business logic is wrap is wrapped first with an authorization Handler here and then an authentication handler right there so the last handler to be registered acts on the request first so in this example we have an authentication we have authentication that will act on the request first and then the next in line

▶ 4:22 would be authorization so it's it's a basically a stack the request comes up dude that way so these wrapped handlers are distinct and they don't communicate with each other directly only through the past in requests so during the request flow each layer communicates with the next by modifying the request context object at each layer with context is called to include the data specific for that layer the flow of information here is just one way down the chain delay they don't ever communicate back up to the previous layer so as you saw in the

▶ 5:06 previous slide each layer adds on to the context that moves through the request flow using with value and with context functions together adds data and replaces the current context with the new one while keeping a shallow copy of the old request so taking a look at the the context interface as a value method that takes some key and return something so empty interfaces there the width value function takes an existing context a key and a value and returns a context so here's the context implementation

▶ 5:54 returned by with value we're calling value follows the parent context in order to find the value under the key this highlights something about the key that we use here if the key is not guaranteed to be unique across all contexts then you could potentially have a collision and return the wrong data so as you and as you might have noticed that this context I mean this chain is effectively a linked list so we don't care about the key itself we just want to make it unique across the entire program so how we do that and go normally is the you define a private

▶ 6:37 type alias to int and making a constant of that alias is guaranteed to be unique as that's a private type for your package so even if a type with the same name is declared in a different package they won't compare as equal as you saw before there's a just a comparison there to see if that that key is is appropriate for that particular context in the in the list and here we use the with value function to store some data under the unique key so we have with the data type that's

▶ 7:20 relevant for your handler and and then it's added to D context with the width value call and under that unique key and similarly extracting the data in a handler involves using our unique key with the value method and casting it to our unique data type so bringing this together that you can see there's the example update of the request context and also later fetching the data so this slide shows an expanded example of the context after moving through the request

▶ 8:05 flow as you can see there are nested context with key value data and added at each layer at the deepest nesting we have the request info data which we'll look at look at that a little closer after that is the user info data here that's added by the authentication layer and finally at the at the highest level of nesting we have the audit data added by the audit layer so this slide shows the requests info from the first layer in more detail this object contains metadata about the

▶ 8:48 requests that'll be used by the rest of the stack the request info is especially important for the authorization layer it contains a minimal amount of request information that the author that a later handler can base and access decision on and it could feasibly cache this as opposed to doing something like caching though the entire HTTP request which even though it might be a request for the same path and everything the body would be different and that wouldn't work for for caching and you can expect later handlers to always like reach out

▶ 9:30 to to something or fetch any other data about the request so it prepares this for the whole stack so if you look at the default build handler chain function in the API server package there's a lot of coal a lot of code here hold on you can see how the standard handler chain is set up where the handlers are registered in an order where the last to be registered are the first to touch the request so you can see that with request info down here and it goes up authenticates the second step authentication audit impersonation and

▶ 10:13 authorization alright so this slide shows the with authentication wrapper which takes a handler and something that can authenticate a request if no Authenticator here is passed in it's an OAuth it returns the current handler otherwise we're wrapping the input handler with a hand with another handler function that first tries to authenticate the request here if there's an unsuccessful authentication then the handler returns otherwise the user is added to the existing context and the request with the new context is passed

▶ 10:57 on to the next handler in the chain so here we're starting to look at the authenticator interfaces so this is the request interface as you can see it has an authenticate request method that takes an HTTP request it returns the response object a boolean if you're authenticated or not and an error for there being a hard failure now to give a an example of the behavior of the bull in the error returned together so if the Authenticator for example compares a token in the request header against tokens in a

▶ 11:41 backing store and if there was no match that's an unsuccessful authentication and in that case the return would be false and nil error but if the Authenticator encountered like some kind of network air or database error reading the backing store something like that then that would return false and actually return an error so here's the response struct the response contains some user info about the authenticated user to be used by the later layers and

▶ 12:24 plus a list of audiences for who the authenticated token might be intended for so here's the user interface user info interface which describes a user that has been authenticated to the system contains methods for fetching the authenticated output so getname returns the name that uniquely identifies the user yet UID returns a unique value for a particular user that will change if the user is removed from the system and another user is added with the same name there's also the groups that the user is

▶ 13:07 a member of and an extra map that contains information that the Authenticator thought was interesting and one example of that is scopes on a token now a note about the name versus the UID so cube is generally named based in in this area so the use of the UID is not particularly common using the UID depends on a later handlers behavior so some may handle that and not okay so like with all with the are back

▶ 13:52 authorizer does not use the UID that'll use get name so that's kind of the canonical authorizer here with our back so here's the x.509 request Authenticator here the certificate is verified using a CA that the Authenticator is configured with and then the user information is extracted from the certificate subject and returned so if you look at the verify options struct in the corner this is configuration for the verify for the verifier where you can configure the

▶ 14:34 see a pool and any acceptable key usage for the certificate so for that Authenticator this is the default user conversion function that extracts the user from a certificate chain at the leaf certificate and and here is an example of the certificate conversion so in this case the subject the certificate subject of oh you foo and see in a bar gets the user info that gets extracted

▶ 15:16 is the name bar and groups food so there's also a header Authenticator that gathers the user information from specific request headers and that's all it does and use the loan it's completely insecure configuring this name headers group headers it could be an it could be like cube underscore or whatever whatever you defined here it'll just grab that information from the header and stick it into the user info so this

▶ 15:59 is the other variant of the x.509 request Authenticator so this one instead of just verifying and just returning the user conversion results it verifies the certificate and then checks the common name against a configured set of allowed common names so if the name checks out then it continues on to call the wrapped request header Authenticator that we showed in the previous slide since this Authenticator provides the right certificate authentication and then cause the header authenticate Authenticator right afterwards then it makes the use of the header Authenticator secure alright moving on

▶ 16:42 now we'll cover the token interface the authenticate token method takes a context and a token and returns response bull or air the same semantics from the the other Authenticator interface so this is an example of some of the kind of a glue code between between two authenticators so the bearer token authentic that the bearer token Authenticator here starts out as a request header Authenticator and but you can just see that that that extracts the bearer token from the authorization header and then passes that to the

▶ 17:25 actual token Authenticator here's the OID C Authenticator it's a common one to configure since you might use it with many other providers such as Google and get lab some of the configuration for IDC includes the server audiences which will be compared against audiences in the request if configured an issuer URL to verify against the issue or claim in the token and if these preliminary checks succeed we verify and then start to extract claims from the token the claim names are also configure configurable so you can think of that at that part of it as similar to the simple header

▶ 18:10 Authenticator that we showed earlier and this slide shows the Authenticator which is used to authenticate service account tokens the token is parsed claims are claims an audience is our process and then the token is validated and the user info is returned so we've talked about different kinds of token authenticators already so if you think about what authenticating a token might entail it could involve some expensive lookup remote lookup or an encryption operation in this case being able to wrap the token auth with a customizable caching

▶ 18:55 Authenticator allows for like a short-lived token cache for optimization so here's the Union Authenticator handler so this Authenticator is configured with a slice of authenticators and a fail on-air option so all authenticators are tried in a successful authentication from any of them returns an okay response if fail an error is set to false then a fatal error from one Authenticator can be over I did buy a successful return from a later Authenticator so if fail and error is true then an error from any Authenticator in the

▶ 19:37 slice short-circuits the rest of the authenticators and returns the air so this is the authenticate request version of the Union and there's also an authenticate token varying it the parts that are highlighted in blue is literally the only thing that changes in the code between requests and token unions right and I'm going to pass it off the mode to wrap it up with the full example ok so here we have like a visual representation of what we sort of talked so far so I'll walk you guys through it so I walk from the outside or the inside out first so pretend you have a service

▶ 20:21 account token Authenticator here right and we also want to honor Oh a DC tokens so we're gonna Union those two things together right so now we have a singular token Authenticator and these two things can be expensive you may have some lookups you do in verification so we go ahead and wrap it with and at that point you still can't honor requests because this is a token Authenticator another request Authenticator so we need to wrap it with a glue code that gives us and perhaps you also wanna honor nine search as was covered earlier and they kind of look at the order of these things x1i x.509 is a super cheap check you're just looking at metadata on the request so we put that one first

▶ 21:05 and similarly service accounts are super common right they're part of the core infrastructure whereas why do you see is users interacting with the system generally speaking right so we put that one first we want it to be tried first and all of this gets Union together so like let's walk a request coming in right so someone tries to authenticate and this is a request that does not have a client cert on it it just has a very quick time so it's gonna go through here it's gonna hit the x509 certifcate ER which will say there's nothing here for me so they'll keep going down well let's pretend that these unions are both don't fail on any errors it's going to pass things through the bear took an Authenticator will strip out the bearer token for us pass it down the cache is empty and keep passing it down we'll hit

▶ 21:49 the Union if it the service account authenticator let's say this was a user the service account open air often terrible error or otherwise say that this is not properly signed for me they'll let it go down to IDC hey let's pretend that gets verified correctly and as it comes back around the cash will report it and you're authenticated so let's say a few seconds later you could do another request you know you keep still get two more pots comes back in goes to the cache the cache is warm it knows who you are so as a actual example from the codebase for how we wire these things look together so this is the delegating hot

▶ 22:33 code so things like the cubelet inside should lie on this any sort of external component that wants to delegate it's off to the QA tester can use this type of code so now you guys said we might use this so we have the best I dropped indicator first client sir token all of this is getting Union together at the end so all the token off bits were you knew together with the cash as I showed earlier they get turned into a request Authenticator and then all of those authenticators get Union together if you're authenticated you need to have the system for authenticated group added to you automatically so we have just a little wrapper that does that and then that final Authenticator is what seems so if you remember back earlier the

▶ 23:16 widow's education bit takes a single Authenticator but it doesn't so without authorization is very similar to with authentication has that no operator at the top technically you could have an authorizer I don't recommend it well let's walk through this so it's gonna get some kind of authorization attributes from the complex we'll see in the next slide what that looks like then the authorizer is going to attempt an authorization and decisions based on those attributes and here we see sort of the most distinct difference between authentication authorization thank you instead of having a boolean state we actually have a tri-state of allowed in I in my opinion this allows an

▶ 24:01 authorizer to say that I explicitly know that this is a denied you don't get to keep going so the unions on this don't have a concept of balaam error or not it's a gonna get denied so let's look through the interface right so the authorizer take some attributes and give us a decision a reason for that decision and an arrow all right so the decision is pretty straightforward right did you explicitly denied this stop crossing them exclusively or is this a no opinion I don't know so maybe somebody else does let them have a shot at the attributes interface is just a combination of

▶ 24:43 request info and user info and a simple method that just checks if this were read only actions against the API so I didn't expand this out because you've already seen these two interfaces so all this authorizer does is it looks at the user on the request check cert groups and sees if that group intersects with a configured set of groups and think does it says listen aloud so we generally refer this to this as the system for masters magic authorizes it's effectively the way of saying that if you're a user with this group you can do anything otherwise you just say and more complex authorizer so this is

▶ 25:31 the web book authorizer right so this is a delegating authorizer in the sense that it delegates the cube api server to do some authorization decision so with the external component asking the qubit castle so we have an API call subject access review that helps us sort of do this let me walk through this there's a conversion from these attributes and to subject access it's really straightforward because subject access review is basically can some user perform some action which is exactly what attributes has within it right so a tiny little bit of detail in there right so there's some user groups all these other things here as well as well as what what are they trying to do they're trying to create a pod in the empty

▶ 26:14 string group that is the question you're going to ask the remote this is the way you do this is you you build up that subject access review and then you create it which is the remote call against a few API server and based on that result so you've asked the cube API server can this person actually so our Baptist's like the canonical example of an authorizer but we never fit on the slide and Jordan has done it far more justice and I be able to so please call us that so let's try to put some authorizers together so this is the Union for

▶ 26:58 authorizers so as you can see we're going to try all of the ones that we're configured to try but when we get down here we only short-circuit if there's an explicit allow or an explicit deny it's no opinion we keep going so this is the sort of distinct so let's look at the delegating authorized rules so as we covered we have the magic authorizer at the top we have some hats that we don't authorize for so it's for example like ten privileged information and here we have that subject access review based delegating osomatsu

▶ 27:42 we just Union those things together and we have one authorizer that with authorization let's talk about authentication and authorization together as a bit of leeway so we covered in the user info interface this extra so we can sort of think about this in certain situation as a contract between authentication and authorization doesn't have to be that but let's consider a specific example we in opens if we use this as a way of having scopes you can also see like this little bit right here the key is just a string so you do need to kind of name spaces so that other things don't get confused

▶ 28:25 when they see this so what does this look like right so we use this specific key as a way of carrying some valleys or slopes the exact semantics of the valleys don't matter too much here just I'll walk you guys to the authorizer you can kind of see what it does with these things so this a little bit of open code it's a scope authorizer so the takeaway I want you guys to get from this is that this isn't an actual explicit denial it never returns in allow it only ever denies or returns no opinion right so it checks their user extra and sees it there's some scopes on this request there's not it doesn't have anything to do you to the bone otherwise it's going to take the scope so that were passed

▶ 29:10 them and it's going to convert into a set of our back rules so we didn't cover our back but the general gist is very similar to like the subject actually bit sourcing right there's some verbs some resources right so can you get so your scope gets converted into a set of rules and if those rules cover the action you're trying to actually do then it's no opinion it's not allowed there's no opinion so it's basically saying your scopes did not deny this request otherwise they don't cover it so does anyone here use open ship for anyone run the install them any demos

▶ 29:50 obviously you work on so if you if you guys have used it there's this user that you can log in as initially in the web console called Q : admin so when I initially wrote the code for this I made a mistake which is on the slide so let's kind of walk through this right this user has authenticated to the system it's given the system : master to break this is a bootstrap user can do anything is how do you manage the cluster Buddha possible in the beginning so it kind of makes sense and let's say it went through a flow of that game with the scope right so if you have ever used open ship and logged into Jenkins be OAuth we disk Ravana that's what scope that component gets so like you're finding it's a token for you

▶ 30:34 that has no scope on it it's perfectly safe if this token can't be doing and as a very trim down example of how authorizers looking Novacek let's pretend you have a magic authorizer your scope authorizer anybody tell me why this cannot work yes correct these are good yeah so basically the mistake I made when I wrote this was I was thinking in my mind well this user can do anything so I should give them the script I don't have to check anything else except what

▶ 31:17 happened is when they would get a scope token scopes would never get a chance to deny would always just approve and so certainly you could try to reorder the authorizers but you really want this one for super cheap and a lot of things use it and you don't want all the bad connections to start having random authorizers so what I ended up doing is less I change this out to system polling questions which is normal that's just a sign you are back the cluster at so this gets skipped this goes to nice expect yes

▶ 32:05 [Applause] so one thing that's kind of annoying is that I can't have Bruce for my service accounts right so there's an implicit group for the namespace there's a group for all service accounts but there's no I want this one being are you guys going to try to address that so I'm pretty sure Clayton's gonna make me address it at some point let's be very fair but yeah I have thought about it certainly I was tempted at some point of happy

▶ 32:48 decided that was a waste of my time but yeah I I've thought about what that would kind of look like I assume exist in the namespace live in the namespace yeah anybody else don't be shy do you have any plans to extend our back to people do or like label-based

▶ 33:31 authorization of resources so like this is a very common feature request but so like if we go back and look at our interface [Music] I guess I guess I really want them I'll get there right yeah you have no access to the body whatsoever so authorizers cannot make a decision about the shape of an object admission is the soul sort of curated

▶ 34:12 there also we we want authorisation to be extremely fast and using the cache and having decisions that are based on the body of objects we did have some of this this stuff over our back it was yeah I'm aware it's kind of messy some oak line stuff read there

▶ 35:06 it's not so so one other thing I've been looking for so one particular problem solved actually by implementing some these interfaces is authentication with firebase kind of a weird use case there I wasn't able to use OID C even though it's an OID c2 token because they don't have to discover URLs is initially I was looking to find webhook implementations or like scaffolding but really couldn't find it other than inside existing API

▶ 35:50 server is there a plan with the cig to maybe provide a framework or add something in like key builder or some of the other associated ecosystem projects so you couldn't use OID C because they don't provide you with the public keys it doesn't have like the well-known girl can you just host one but it's like you should have you had the public keys somewhere right that somebody has that great so you're not minting the token with yourself right you just want to educate them so I mean

▶ 36:32 so there are plans in general like if you've been following the new token request API for service accounts if we want those identities to be able to assert it outside of the cluster like inside of like your cloud provider and so there are plans on making it so that the cube API server it's so exerts metadata about tokens but on to your specific question that like it's very like I would expect they're like you should be able to technically postal on URL to pass to our o IDC educator attended a verification because as long as the issuer URL is correct and all the public key business practice should be able

▶ 37:16 otherwise I'm trying to think about we moved out of court so the delegated opcode is relatively separate now but I don't know it's like super easy to consume certainly I would Oh like we're like self-hosting on the bus yes

▶ 38:09 I mean the the general desire for that stuff have always been that like like sort of they're all so I would think it we sort of thing but it it should all just be a pretty small you shouldn't have accepted my delegating authorization to that code - it's possible now what time is it are we done

▶ 38:55 we way past thank you guys for coming Oh [Applause]