Paint the Picture! - Detecting Suspicious Data Patterns in E... Natalia Reka Ivanko & John Fastabend¶

Playlists: KubeCon + CloudNativeCon NA 2023

Shows how Tetragon uses eBPF and kTLS to detect suspicious data patterns in encrypted traffic.

Categories: Observability, Security, Networking

Topics: tetragon, ebpf, ktls, encrypted traffic, l7 observability, runtime security

Transcript¶

▶ 0:00 all right hello thanks for coming uh we're here to talk to you today about detecting suspicious data patterns in encrypted traffic with ebpf and ktls and I just want to say I'm excited to talk to everyone about this today because I've been working on ebpf and TLS for close to 10 years now so seeing all the pieces come together is exciting um my name is John fastan I work on tetron I'm the lead there I'm also Main also a colel maintainer um and I work at is surveillant um and Natalia is with me as well and um she also is at at um is surveillant and she's the product lead um for I think everything we're going to be talking

▶ 0:42 about today um so here's a quick agenda we're going to talk about tetron just to give you a just a quick um brief about what our background is like how we approach problems like our lens for looking at this and then we're going to dive into L7 observability and security and use these cases so let's get going because we have 34 minutes now so first of all what is tetron and the reason um this talk isn't about tetragon we did have a talk on Monday so go back to the conon if you really want to get a deep dive on tetragon but I think it helps to understand what we kind of problems we think about and how we how we go about addressing them so tetragon is a security observability and runtime

▶ 1:26 enforcement agent it's written in BPF they really heavy focus on overhead try to keep that performance really low we do that with BPF in the kernel we move a lot of the business logic that is usually in user space we move it down into the kernel so all your filters um are are down there we keep a a state of all the execs that have happened in a system all of the sockets so we have a map from sockets to execution we have a map from DNS to sockets back to the execution and we can build kind of a map of what's going on in your platform for observability reasons first and foremost and then we follow that up with enforcement so you can say things like if um if this pod or binary even is

▶ 2:09 talking to another binary that's allowed if it's doing a DNS query out to a new DNS server that's not allowed so you can build these fine grain binary level policies um and uh there were actually uh Thomas graph gave a talk Monday about this so go to the symcon if you want to know more about that um but that's really our focus is the security deep observability low overhead so that's where we're coming from both from a networking side but also from a runtime so think about file Integrity monitoring think about um building signatures to match CIS calls and so on so that's our um that's kind of our lens you know um we want to make sure we do that transparently we don't want to get into the application world where you have to modify your applications to get this data we want to have a single agent you

▶ 2:53 deploy it and then all of a sudden you get this visibility you get this really good enforcement property as well um and we're not just at the CIS call level we're getting down into DNS requests HTTP request TLS which we'll talk about um you know what files are being opened kind of deep deep inspection down in the kernel mapping that back to binaries user space kubernetes labels that's the kind of tetragon um Viewpoint of the world um and now we want to talk about layer seven okay so one thing that we do um at ice surveillance and I'm sure this isn't too foreign to most people I know a lot of different people do this kind of observability is you get a nice

▶ 3:36 dashboard like this okay you got your um htttp observability you can build a service map it'll say this service is talking to this service you can get latency histograms um you can get uh how many bytes are going over to this URI or that URI so a lot of different platforms build this HTTP observability platform and give you this visibility okay and in tetragon we do something very similar um and but we do it at the binary level so you can pin it back all the way to the binaries inside the containers um so this is great we um you know we endorse this kind of view of the world um you know you can even do things like this where you get uh a process

▶ 4:18 execution Trace you see inside a pod there somebody executed node server.js who are they talking to they reached out to API Twitter apparently a few other things elastic search and maybe a malicious bucket that you want to detect um in your system so this is all really great um like I said we provide this along with a lot of other folks um but there's a there's sort of a a problem with this when you go to zero trust and you start encrypting everything right uh your observability tools cannot look inside a TLS session um directly and so that's that's what we want to talk about

▶ 5:02 today um if you're going to build these tools there's a bunch of different places you could conceivably hook into the system and this is a a kind of a picture here um in the upper left you have the Pod and you can see there's a couple options up there that you could hook you know Library instrumentation is a is an option right you can uh instrument your library with observability hooks extract data U probes we put up there that's BPF ability to hook user space very similar to a library you hook into the user side of the program your BPF program runs and you get some observability out then we have side cars still inside the Pod okay

▶ 5:46 basically what that means is you take traffic from the containers route it over to the side car the side car can inspect traffic so these are all hooks for for layer 7even observability um you know another option that's silly provides is a host proxy so take that proxy put it on the host bring it outside of the Pod then when traffic is sent from the Pod to the network you route it through that proxy either through ebpf or if you have a more um kind of Legacy system or an older system you might use IP tables as well um then finally if you're inside of a like a virtual machine there's even Hardware hooks for this so if you're a big cloud provider you might have a a hook on the the dpu or the fpga on the network side

▶ 6:29 okay so traffic comes out of the system um you get hooked there uh you that's an interesting one if you have the ability to play with Hardware most of our environments we don't have a ability to drop down fpgas on the system okay so but we we put back on our tetragon lens because we are a security tool we don't actually trust the Pod so we get rid of a whole bunch of observability points that would makes that could work but don't work inside of our trust model so if you look up there you know clearly we can't hook the library anymore because we don't trust the application that's the thing we're trying to observe the thing we're trying to secure from a malicious user if that thing is the malicious untrusted thing we can't trust it to give us data right um and then if

▶ 7:13 we also look at the sidecar um model inside the Pod from a security standpoint uh you know we don't want to trust that entire pod that's our boundary we don't want to inject something into the Pod and then have to trust it okay so we we rule those out which really leaves us with two hook points that are viable either the host proxy on the left which is an agent that we redirect to or something we're going to call sock map L7 BPF and what the L7 BPF is is it's a way for BPF to hook directly the socket so any application that sends data we get a BPF event with the data so you do a send

▶ 7:57 you do a send message uh even kind of a receive message for example will'll get that data in that BPF program um what's really interesting about hello oh we're back all right um what's really interesting about that is the this program cannot be controlled by the Pod at all even though we can attach to the sockets the user of that socket the owner in the Pod cannot detach or somehow avoid the hook we will get the event um this lets us put a BPF hook at the socket layer read all the data um those are our viable hooks um you

▶ 8:42 might have noticed I put TC BPF there um TC BPF is a way to hook at the software in the network it's a great hook we use it in syum a lot unfortunately it's after the TCP stack so if your mindset is L7 you really need to be at the socket layer um because because if you're not you have to worry about TCP retransmits which will throw off any parser building the packets back together um it's just you end up writing an entire new TCP stack in BPF we don't want to do that so what's the flow let's think about um let's think about a curl here at ebpf do0 what happens first there's a DNS okay then we connect then we do a

▶ 9:25 send message if we're using the uh pod visib or the Pod prox host proxy here sorry um what'll happen is that packet will be sent out of the Pod it'll be redirected to the host proxy you'll get your event and then the host proxy will forward the packet along so kind of a bump in the wire on the host to do your observability all good again if for HTTP this is how the model Works um how does it work for the sock map stuff very similar still need to do your DNS you do a connect to the IP address and then you do your send same as before except for right after you do

▶ 10:08 the send we get the BPF hook We R uh we run our BPF program over the data we parse your HTTP headers uh your CFA whatever we're looking at TLS handshakes and post that event to our observability stack and then send the data along the advantage here is we don't have to do any read directs were not even in the routing stack at all or purely a BPF hooked directly to the data source okay again works really well for HTTP and unencrypted traffic so as I mentioned before this is really great but what happens when we want to encrypt stuff for zero trust service

▶ 10:53 smash all the words that we keep hearing about everybody says let's encrypt so we do do that and then our dashboards do this right everything's encrypted so I don't know anything about HTTP anymore because it's all wrapped in TLS so everything's broken we need to find a solution the first one that uh that psyllium supports as well is same flow again curl but this time the curl is to https okay uh sends it out it's encrypted no data right so that's that's

▶ 11:35 kind of the flow again it's broken what do we do you know BPF doesn't really save us here same idea curl they because the encrypt happens in user space before they even send the message same problem okay so we need to think up uh kind of another solution right uh let's talk a a little bit about what those are what are our options how do we resolve the the encrypted data problem here because of our security model like one option is to don't encrypt unfortunately not an option the other one I've seen used quite a bit is U probes you could in fact stick a u

▶ 12:19 probes in front of the op SSL library to see the encrypt process um so basically in that mode you take your U probes and you attach it to somewhere in a SSL Library whether it's open SSL or boring SSL or any of these SSL libraries you hook in before they do the encrypt you get your BPF hook the only trouble with that from a tetragon standpoint and kind of our Viewpoint as a security problem is that that memory is still owned by user space and we're kind of at the mercy of a the user space calling our library and B um a malicious user could in fact change the data after we've read it so we have a security problem um from using U probes um otherwise it would

▶ 13:02 have been a great solution but unfortunately we cannot use it so we really have TLS termination which is the psyllium option and then we're going to talk about ktls which I think is is an option that we've been investigating um so it's back down to to the same kind of flow again that we've talked about curl but this time to https U probe we throwing that out so that's basically before the encrypt like I said and then you do the send message um and you're you're good to go so let's see here sorry I got my slid slightly mixed up there we go so this is the next one about the termination this

▶ 13:46 is one I wanted basically you do the send you do the redirect the redirect goes to the host proxy host proxy does the TLs termination because you're sending it with encrypted you then terminate at the host proxy observe the data because now you have it unencrypted and then you reencrypt it and send it along to its destination the um the complex part about this is really what this slide is from the docks in the cium side because now you need to inject your CTS into the encrypt because you're going to be decrypting you need the application to do an encryption in in a way that you can decrypt right if it just encrypted it arbitrarily you wouldn't be able to do that so you need to do a kind of a

▶ 14:30 certificate management process shown here um not going to dive into all the all the nitty-gritty details of this but the basic idea is you inject your CT the application uses that use that to decrypt and then on the um on the other side you reencrypt and send it out and so you basically interject yourself into that flow and so how do we do this with B BPF um so BPF we don't do this this kind of CT interjection instead what we do is we're using something called ktls and ktls was designed as a performance optimization

▶ 15:15 for TLS the original use case and it's in production and widely used was to avoid doing an extra copy every time I need to encrypt data off of a dis so you can imagine if you have a dis image it's a video perhaps or something like this times there we go I'm back um Netflix has talked about this in in a couple papers and talks basically the idea is you have a video or something on a file and you don't want to copy it to user space encrypt it and then send it back in you want to be able to send it directly from the disk to the um to the network and by by moving the crypto into the kernel you're you can do that process

▶ 15:58 all in one operation copy from disk crypto send without ever going to user space so that was the origins of ktls that's why we we added it to Linux um a handful of years back uh that's why it's been in FreeBSD for years um that's the origin of it um and if you're curious a little bit the breakdown is to do the handshake in all the control plane in user space and just do the data path operations in the kernel um and you can see performance wins from doing that then what we did uh after that is we then hooked a BPF in the kernel can hook in front of that encryption and so when you send the data you're sending the plane text the BPF program runs and then it's encrypted after the fact which lets

▶ 16:42 us read the text in plain text but the application is allowed to keep the keys and and do the entire control flow and we don't have to interject ourselves on the certificate side um this is the flow here labeled out the the one trick is uh the one piece to enable it is the application does need to do a set socket option to turn it on um we've done work in op SSL and a few other libraries um to add support for this um so that it's mostly um transparent to users they don't need to opt in it can just be used if the Kel supports it the um the main thing is that you would need an OP SSL 3 some of the new newer

▶ 17:25 libraries to have this but it is um available in production libraries already um so that's the basic idea and then with by the by doing that then we can get our dashboards back and then that's the big win um and then if you look kind of at what does this look like we have a a CLI here that we show for a lot of things when you do the crypto basically the process gets um created it gets the connect we see the TLs and this is actually important because if you wanted to um if you want to make sure that you're enforcing this Behavior we can actually observe whether or not the user has set the socket option to use ktls and then we can do things like alert or

▶ 18:09 we can you know drop the traffic and we're in a very strict environment or at least alert to folks that this is not being used and so we're not going to see the next line which is the HTTP so this was um basically some code we added to tetragon to do the httv parsing in the kernel do the TLs handshake parsing and then um print it all out uh and again we didn't have to manage any of the certificates we didn't have to do this kind of stuff we're directly in the BPF at the send at the send side um and with that I will let um natal talk yeah so now that we got the asan visibility back bya ktls so what would be the end case use cases here so how for example an information security or SEC Ops Team

▶ 18:53 would use this so I brought two examples like both of them are very cool and interesting the first one is like data exfiltration via token so the main security challeng is like let's imagine an Enterprise like many Enterprise use like a s service to store data like data dog t or Sentry and then in that Enterprise each team has its own account and then an Associated token so for example how could you spot if you have a moish employee in a team and then it just creates its own account and the token and then just start to ship like data to that

▶ 19:35 account so with traditional network monitoring tools you only see that there is like encrypted data or encrypted traffic going to that like s Service you don't really have visibility into what token is used so this is just like a very simple example you have the organization you have the application team platform team each team has its own account with a token you have the s service and then basically you only see that traffic is encrypted over TLS and then you have this malicious employee created the account with a new token so one solution is like let's use ktls and then internal HTTP parsing and then basically you would have the full

▶ 20:19 visibility into an L7 Network flow so you would have visibility into HTTP header and then the body so you can imagine like you can identify what the what was the token and then for example you can compare it um You can compare it with other tokens on the white list so you could actually spot that um there was a new account with a new token and then the dashboards it's actually the HTTP traffic um in a graph on a dashboard exported by tetragon of course it's like kuber identity a so you could see like kuber name space or kubernetes workload but was the binary that actually um initiated that request you

▶ 21:02 could see like the method um you would have visibility into the URI so that's where the token would leave and then you could see for example the HTTP version content length um Response Code and so on so this is also another screenshot from the dashboards um here you could see like all the ues that were for example queried by a workload and then you could see like how many times and then you could do for example a sum like per workload or for example per binary the second example is actually like detecting sensitive data patterns

▶ 21:44 from the HTTP body so let's imagine like it's actually like a real problem that many Enterprises struggle with achieving compliance standards for example to not lock sensitive data like credit card numbers Social sec cre numbers IDs so with ktls and HTTP parsing you would have full visibility also into the ASM Network Pro so you could actually create signatures to detect sensitive data patterns or even like payloads so very simple example you would see here here is the header and then here you would have visibility into the payload so it's actually just like a very simple post um request to like a

▶ 22:29 payment service so you could see the web server information you could see for example the user information it was a go agent and then you you would have visibility into the payload as well so you in in case if there was like a payment you could see like okay um that was the email address the credit card number expired expired date and CCV so you could you could actually detect those and you could even um create like signatures um to detect like more micia payloads so that's what we all wanted to talk about I do a little bit of w up so we talked about like what is tetragon um

▶ 23:12 abpf based uh runtime security observability and enforcement project we talked about S7 observability from different views architectures trust models they talked about like what happen if you introduce thls how it breaks the observable ability and then how could you still have observability if you um have encryption and we mentioned some use cases so if you're interested um in a more in-depth demo just come to our boost um celum boost or isovalent boost and then we could show you or if you're interested in tetragon just check out the GitHub repository it's open source we have documentation website if you

▶ 23:56 have any new idea um um create feature request um people are actively working on it we have many quick starts user guides um and tell your user experience and if you're interested in more in ebpf we will have a documentary film starting at 615 so you could check it out as well so with that I will open up for Q&A and thank you yeah so if you have any questions just come to the mic I can hear you and if there are no questions we can

▶ 24:41 go to oh there we go perfect um thank you very much just um curious you said the grafana those were an exporter from tetron is that available to us the graun dashboards I don't think I don't think we have any we have some in the docks I think there's some actually in the repo but we haven't really added them to the docks at this point I have to go back and check I mean so the from the tetragon project side um we just released 1.0 last week um and we got a lot of the quick start docs I the I think a few of the graphon dashboards are still buried in the code somewhere we'll try to get them in the docks here

▶ 25:23 pretty quick um there's no reason not to have them visible I it's just a matter of making sure we get them in the docks and and visible for people to pull down perfect thank you yeah yeah no worries yeah neat talk really I've never seen that before um KT with ktls I have to have a version of op SSL or whatever is doing the encrypt encryption that supports ktls does that mean that I am maintaining my own main containers that have specific versions or is that pretty widely so supported yeah you know if you have op SSL 3.0 then you have ktls support because we um we went you know

▶ 26:08 whenever SS this was two or three years ago we went through and added ktls support to open SSL um and because it it was a major feature we can't get it in open SSL 2.0 because it would require back porting a feature but if you have open SSL 3 you know which will become more and more widely available hopefully over time um then you do have ktls support and and that's actually independent of ebpf um we use it with ebpf kind of layering the Technologies together to be able to look into um Data before it's been encrypted but you can use ktls just as a performance optimization even without ebpf um and so it exists there um your applications will have to opt in with that socket option so that was kind of where we landed on that is you know

▶ 26:51 we didn't want to just default everybody yet you know to every case being KT so you really have to do that the application will have to do the set socket option to turn it on um once that happens then um then if you're an application using open cell then there's nothing um nothing else you need to do like under the uh the actual Library will'll just handle the handshake offload it all to the kernel um and you're good to go uh you know there's a there's a goang sell Library there's a PR open to do ktls there um I believe if there's somebody who's really um up to speed on what boring SSL is doing they maybe correct me but I believe there was something there I believe Facebook has talked about using this as well so like there's definitely some rather big

▶ 27:35 organizations that are using ktls today in production um kind of you know in use so awesome thanks cool any uh any other questions we still have a few moments otherwise okay cool um yeah if you guys want to find me later or Natalia can talk to you more if you're interested um we definitely we have a demo too we can if you come by the booth we can show you a demo if you want um just come find us uh you didn't do it here just from swapping screens around it would looked a little bit complicated I've seen a few demos that tried to reach out to the web and then the web didn't exist so um you know come find us we can show you a demo if you're interested all right thank you

▶ 28:19 so much