Mutual TLS Adoption Made Simple, Safe and Secure - Lizan Zhou, Tetrate & Jianfei Hu, Google¶

Playlists: KubeCon + CloudNativeCon Europe 2020 - Virtual

Maintainers explain why mutual TLS is hard to adopt and how Istio API improvements make adoption simpler and safer.

Topics: mtls, istio, envoy, service mesh, authentication

Transcript¶

▶ 0:00 hi everyone today we are going to talk about mutual tears adoption so here is the introduction for ourselves hello my name is rizzan jo i work at tetroid i'm an envoy maintainer and also in istio networking and security working group hi my name is jim fay and i'm working at google my work is also mostly focused on is

▶ 0:38 your security and working group so we are going to talk about what is mtrs and why mutual tiers is very hard and what customer challenge is going to face um and based on that challenges we are going to explain some of our approaches to how to optimize and improve user experience and in the end we are going to talk about the uh summary and also some lessons we learned during this uh api improvement for mts option journey and last section

▶ 1:23 when we have time you're gonna have a q a session so um first we're gonna explain why what is mts and why it matters um so mutual kiosk is a extension of the tis and it's essentially a security check on both client and the server side the client is checking the server's uh identity by a x509 certificate and the server is also asking for a certificate from a client

▶ 2:06 the these we are these steps is secured to service to service authentication and the transport layer also based on the x5509 certificate information the server can derive some client information like namespace and services account to do some further authoritation check so if mkrs is good why what makes it very hard to adopt let's take this as an example on the left side uh we have client and a server a full service and the traffic are in plain text after the customer adopts the

▶ 2:50 site car by injecting the envoy to the application we can now establish a mutual ts by adding this in between the server and client envoy so this way you don't have to change your client and the server application code at all you get mts for free everything is transparently but in the real world things are not that simple so if you look at this diagram you can see on the right hand side the bar service actually has v1 and v2 uh two versions and the v1 version is a legacy release for example and it doesn't have the envoy cycle

▶ 3:34 rolled out so if the client envoy is only sending the mutual ds traffic for the bar service it's gonna fail as they need to reach out to the bar v1 port endpoint and similarly on the counter part of the client side story we have a similar situation the client one does not have the invoice card and the client too does have when the server full service envoy only accepts the mts then the plaintext traffic request from the client one will start to fail so to make things even more complicated

▶ 4:19 you can see both things can happen at the same time so we have a partial sidecar brought on both clients in the server side and you can see two of the requests kind of situation out of the four just be fail so how can we solve this problem one way to think about is well if you cannot accept the mutual test between the client and server and way unless envoys are everywhere why not we just wait right on the um we start from the plain text by not doing any of the muted at all on the left side

▶ 5:03 where you start to ramp up the invoice injection in the your service mesh and he only starts to do mutual tears once you finish the um cycle route on everywhere and at that time you track this progress the then flip the entirely from the plain text to mutual tiers and julio all at once however in the real world this is almost impossible to do we all know the complexity of a enterprise deployment that you always have some kind of corner case where some team's service would not be able to adopt new technology sidecar can just not be put there you

▶ 5:47 cannot uh this will never happen basically so um ucl is doing initially is doing some initial api improvements on this in this example you can see the full service both its client and server application have already finished the envy cycle route so um in that case we don't have to wait for bar clients to be finished we can just adopt mutual gs for full service so we have a two apis on the eq side while it's called destination rule which is essentially a api configure the client envy cluster behavior for example like load balancing

▶ 6:31 like a circuit breaking and also eco mutual tiers can be configured here we have a in this case we specify the service host as full and the ts mode as issue mutual to let the client envoy sending mutual tears traffic on the server side we have an authentication policy and this policy also specified the service as full and a single boolean flag has to enable the mutual tears uh this here is our very basic approach our baseline of how to solve this problem now lisa will start to talk about how istio

▶ 7:12 and envoy make some improvements so in the server-side improvement this is in the case of the client-side car is partially deployed we do the ti sniffing that detects the tos and plain text traffic so in this case server will be able to serve both case whether the clan have unvoiced sidecar or not in the istio api this is expressed in permissive

▶ 7:55 mode and compared to the strict mode the server only accepts mutual ts config in envoy config this is shown in the snippets in the slide here we have the tos inspector listener filter and we have to define two filter chain with different filter match one matches to the tos traffic and another matches to the plain text traffic so how it works basically the ts um sniffing the ds inspector take a look on the first packet that sent from client

▶ 8:40 to check whether it's a clan halo in tis if that's the case the tos inspector will extract the some tos metadata search whether it's the os what alpn it is us used uh what the sni in the client hello message there's some drawback of this approach for example for some of the server first protocol such as mysql this case the first packet won't arrive when ts inspector is working but the traffic is not forward to

▶ 9:23 upstream server so in this case we introduced some timeouts in the listener filter but it still adds a timeout to the such protocol when timeout happens the server side avoid forward the traffic as plain text and then server respond with the first message back to grant it keeps clients such as mysql works but add latency to the all the other to the time timeout so in this case um we we address the two different needs

▶ 10:07 one is permissive one is strict that is the ins once you're done with the all row out on the client side you can switch the tos policy to strict mode in that case the server won't do the ts won't detect plain text traffic that means you are now fully secured in the full service that won't get plain text traffic so in istio site configuration this is this express as destination rule and authentication policy so once server side rollout finished

▶ 10:50 the clan destination rule should be still mutual and the permissive should be changed to strict this also addressed some config complexity in the previous like the manual config approach since the install destination rule is an override base in this example if you have some fine-tuned connection timeout in full service and you apply the cluster while it is still mutual it won't apply to full service so it will break there as well you have to do something like

▶ 11:34 the bar service to have the traffic policy and also you have to apply that to the other subsets as well this is an uh complexity of the istio destination rule itself but this makes mutual ts much harder because you have to know about the odd override and applying them correctly so in the client side this is for the case that server doesn't have the fully uh sidecar deployed this is based on the new feature we

▶ 12:18 introduced in envoy called endpoint labeling uh transport matches was added to the cluster config which defines a can defined multiple transport socket which basically allows you to specify multiple tos config or plain text config and in the end points that delivered for the for that cluster just a label um called mutual ts ready so when it is true it matches to the transport socket defined in the cluster to initiate the tis connection to those server workload with

▶ 13:00 sidecar deployed for those without server envoy deployed it still uses the raw buffer means it uses the plain text communication so combine those two server side and client side approach you can see we addressed all the situation here that whether client have envoy or not whether it's partially deployed or fully deployed all the server and client can talk to each other

▶ 13:42 for the for the legacy case like both side doesn't have envoy deployed it can still communicate um of course so we start so as a journey of the adoption we start with you can start with premise save mode by default and then this happens automatically when you start deploying sidecars to your server and client once your rollout is done on this on the service file side you should be able to switch that to strict mode then you are fully secured on that

▶ 14:27 service now i'm going to uh pass back to genfe to tell more about the lesson we learned from this thanks lisa for explaining what kind of the details and the mechanisms for improving the issue mutual trs adoption so now i'm going to talk about some lessons when we learn go through this journey ourselves the first of the first that service mesh sidecar raw doesn't take all at once it takes time people try new technology gradually and there will be quite some periods when your service mesh consists of both workers visa and vis outside car

▶ 15:11 and this could take quite some time we need to have good user experience in terms of the configuration amount you need to do for you to adopt mutual deals that brings up to the second point we one way to look at this problem is that customer can solve this problem on their own if they just reading to write more kind of configuration for example one of the alternatives as our compared to our approach is that you can create some subset in the destination rule and the one subset corresponds to the server workloads without any sidecar and one that has the mvos icon you can also specify the

▶ 15:55 percentage weight between this subset when doing the load balancing and of course this is very manual and introduces a lot of config toy problem and the next point is that don't forget about security even you just want to focus the transition and make user experience good uh that's why we realize there's two modes in the beginning you need to need permissive and make things very smooth but once you're done with the second route since you want to be fully secure only strict mts is what you want um in the end one last point is that the real world traffic is really complicated

▶ 16:40 and we hit all kind of corner cases when we design these mechanisms for example on the server side then we are doing the permissive by tr sniffy we found sometimes the legacy client can send a https request to the server and if we are just doing the transport circuit match transport tier sniffing based on the transport protocol as trs that will match to the wrong filter chain and trying to terminate a https traffic with eastern mutual configuration and of course since we start to fail um on the client side we have here some similar issues

▶ 17:23 um the endpoint labeling a solution only works for the endpoint discovery typed moa clusters and and we have other kind of clusters like original destination and sorry this is the type of on the slides that we also have dns resolved kind of clusters those kind of clusters doesn't go through the end endpoint labeling selection code pass in the workflow so you cannot have this automatically detect endpoint mtrs ready behavior next is our just to summarize or we present today um we solved this

▶ 18:05 mts adoption problem by two parts the first is the server envoy we make it both support mtrs and the plain text this is done by the envoy level ts sniffing in as a network layer filter and on the easter api side we introduce permissive and strict for two kinds of needs on the client side we introduce the make the client envoy able to send both plaintext and the mutual ds traffic based based on the server endpoints labeling results the outcome for these improvements is that we remove the client side

▶ 18:48 destination rule configuration totally if you just want to get mts done and these two things combined together is that you start from the permissive and the auto materials in the beginning all you need to do is one switch from permissive to strict for your entire mts journey which is much much more simplified all right and today that is our today's presentation uh here now is the q a sessions feel free to ask any questions thanks um i um my name is gandhi

▶ 19:40 i see a question of posted on the room that is asking is it easy to renew the mta certificate in the whole cluster in istio's case that we have the entire public key infrastructure to responsible for key rotation and deliver it from the control plane to this from the control plane to the envoy sidecar so the key certificate of the workloads are rotated every for a short period of time and i see another question supposedly

▶ 20:45 the original question is if some of the apps use the mts already and then some rely on anyway what is the easiest way to support this traffic pattern plain text will not allow during transition um so uh on the server side that we uh for the workers that have the envoy sidecar we have uh the aipm match with instill dash our own erpm so that that won't conflict with user their own mps that offer you a support both

▶ 21:27 the application that has speaking the its own mps versus that we are trying to terminate nts but uh we cannot control the workload that does not have the mts uh with a car go ahead so yeah i can answer the next question um can use of mutual tears lead to lack of visibility since the traffic is now

▶ 22:12 encrypted there's no way to do traffic inspection on application traffic is there so yeah there's a debugging techniques still can be used with envoy to inspect application traffic using android debug feature or you can still try to see what the traffic in plain text between side car and the uh and the application by inspecting the local traffic traffic but by inspecting local host traffic okay i think we

▶ 23:35 okay there's a follow-up for that one can traffic inspection be done for encryption application traffic um if if your application encrypts the traffic itself you should you need to somehow by using application specific logic to export the trs keys i mean the encryption keys there's some way to do that and having the chat like um tcp down for any traffic capturing stuff and combine them you should be able to do the inspection so yes it will be more steps to inspect the traffic when things are

▶ 24:19 encrypted okay we will um on slack for uh more question if you have more um we'll monitor slack for more questions thank you