Testing Kubernetes Without Kubernetes: A Networking Deep Dive - John Howard, Solo.io¶

Playlists: KubeCon + CloudNativeCon NA 2024 - Salt Lake City

John Howard shows how to test Kubernetes networking components without a real cluster for faster, easier-to-debug tests.

Categories: Networking

Topics: istio, ztunnel, testing, networking, kubernetes-free testing

Transcript¶

▶ 0:01 Hey everyone. I'm John Howard. I'm a senior architect at solo.io and I'm here to talk about testing without Kubernetes. So before we get into things, I want to get a quick poll of the audience to see how many people are running tests where they actually have to deploy something on Kubernetes as part of the test. All right, and how many of you guys enjoy that experience? No complaints. We've got one. They're not sure though. I I wouldn't be so sure either. So in my experience like testing on Kubernetes, we do it but it's really rough, right? It gives us a really hard time for all sorts of reasons. The biggest one for me is that it's quite slow. I'm not a very patient guy and in order to

▶ 0:44 set up these tests, we have to go create a Kubernetes cluster. We've got to build some images. We've got to push some summer. We've got to create some deployment services, configuration, have those pull in the the cluster. They spin up, maybe they take a while to pass the readiness probe. All these things and by the time we actually get to our test logic, we've got like 10 minutes down the drain. We've already finished our coffee and then the actual test maybe only takes 10 seconds, right? It's a huge waste of time. It's also going to be very hard to debug. Aside from the fact that there's now all these moving parts, it's also not so easy to just go attach a debugger to our process and just go see put breakpoints somewhere and figure out what's going wrong. It can be done. I know I'm sure there's plenty of tools out there that help us do it but it's still nowhere near as easy as a basic unit

▶ 1:29 test where you can just you know, click the button in your IDE. It's also hard to onboard. Like I've been running tests in Kubernetes and that's part of projects I've been working on for many years now, so I've learned a few tricks to speed things up and I know all the commands to run and the magic scripts that you need to touch. But we often see really smart people trying to onboard the project and it takes them many hours or even days to figure out just how to run the test and that's time that they're not contributing value and they get really frustrated. Sometimes they leave and they never come back, right? And it can be expensive and unreliable. And depends on how you run things. We often use this project called kind Kubernetes and Docker which helps keeps our cost down. But we at one point were running all of our tests on a real cloud Kubernetes

▶ 2:13 cluster and we were racking up the cloud bills like crazy. Like you would not believe the price we were paying just to run tests which is not great and those are unreliable too. Even the local ones we see you know, every once in a while we spin up the cluster and it just doesn't work for some reason or it times out or you know, that image registry is down for a little bit so all of a sudden all your tests are failing. And it just it really just gets at you and breaks your productivity. So if we have all these problems, then why did almost everyone here raise their hand and say they're testing on Kubernetes? At least for me and I have to imagine for a lot of you, it's because we feel like it's necessary, right? We feel like the way to actually get value out of our tests and cover the functionality is we need Kubernetes. Whether that's because

▶ 2:57 we're writing like a controller that actually needs to run on Kubernetes and takes advantage of specific Kubernetes features or that's just the only way to run our application or for various other reasons. So it is rough but maybe it's necessary, right? Or maybe not which is maybe some foreshadowing why everyone's here and what we're here to talk about. Um So I want to give some background on what I do just briefly to set the stage. So I work on Istio which is a service mesh. If you don't know what that is and you're interested, there was plenty of talks this week that you can go catch the replays of. Um you don't need to know the details though. But what you do need to know is that we've been working on a kind of next generation service mesh deployment model which gave us a unique opportunity to also rethink our

▶ 3:40 testing strategy from the ground up as we were building this new architecture from the ground up. Um one of the tricky things about this though is that the new model actually even more deeply integrates with details of Kubernetes which made it even obvious to just go run this test on Kubernetes to make sure that we were actually testing the functionality, right? Um so very very briefly, I just want to explain the architecture of our system to make sure things make sense here. Um ultimately what a service mesh does is it takes your traffic and runs it through a smart proxy that can do various functionalities. Um The new architecture is deploying a single one of those per node rather than what we previously did by running it inside of the pod as what we call a sidecar proxy. Um and this node proxy is

▶ 4:25 actually handling networking for each pod on the same node. Um what this means is that we're hooking into all sorts of nitty-gritty Kubernetes details in order to modify the networking stack of these pods, right? Um and so again, like it was super obvious. Let's just run this on Kubernetes because that's like we can't even run this proxy originally outside of Kubernetes. It had to be on Kubernetes. Um but we took the opportunity to think how can we do this without Kubernetes? So I want to go over a bit like what I view as like the ideal test and what I wanted to achieve with our test. So ultimately what we want is tests that can run quickly. I'd say a second per test but even that's a pretty loose goal. I'd have much rather have all the

▶ 5:08 tests run in one second, right? That's a great iteration time compared to the tens of minutes that we have with the Kubernetes style test. It'd also be great to have no test setup. No like special test setup so that someone can just come into the repo and run go test and things work or whatever the equivalent is in their language. It'd also be great to be easy to debug with standard tooling. Like I don't know about anyone else but I like to just go and click the debug test function in my IDE and I can just go set breakpoints anywhere. I would love for that experience of unit tests to apply to these integration tests as well. Um and these are kind of a bit redundant but in order to meet those three goals, we really need this run without Kubernetes for some of the reasons I mentioned without Docker as well and

▶ 5:51 potentially without root as well cuz if you have to require root, then we've kind of violated some of these no setup steps. Um and we'll get into some more info there a bit. So with that being said, we before we get into testing, we need to understand a bit of how Kubernetes works because if we're going to run a software that typically runs on Kubernetes outside of Kubernetes, we're going to need to appropriately simulate Kubernetes so that we're actually testing something that's close enough to Kubernetes to give us value but not so much like Kubernetes that it's really slow and has all these other problems, right? So a common perception with Kubernetes or even containers is that these pods or containers are these really strong atomic units of isolation and like you

▶ 6:34 have a pod, you can't break it down further than that. Or at least you have a container and you can't break it down further than that. Um and this is true in honestly a lot of cases but it's not really how like the reality of the system works. If you look at Linux, there's no such thing as a container. What they have is actually these concept called namespaces. Not Kubernetes namespaces, this is different Linux namespaces. And a container is just a nice way to package a bunch of different namespaces and a few other things together into one nice package which is really like what the innovation of Docker was years ago. Was taking this obscure Linux feature and giving a great UX around it. Um so if we look at the reality, what we actually see as you know, just this pod is really a collection of a network namespace, a mount namespace, a user

▶ 7:18 namespace, a process namespace and even a few others that I I didn't really mention here. Even this is a bit of an oversimplification too. Um namespaces are a lot more fluid than just like these boxes around process, right? You can have a process join a namespace and leave a namespace and you can have this kind of mix and match of namespaces to processes. So here I show an example of we have kind of two applications that are sharing the same mount namespace within a network namespace and there's another process in the same network namespace but not in that mount namespace. And we can even do crazy things as well like have something in a different network namespace also in that mount namespace and really any combination that you can dream of. So Kubernetes doesn't expose all this complexity for the vast majority of

▶ 8:03 cases, you really don't need to do any of this crazy stuff. I just want to illustrate that these things are a lot more flexible than we are typically exposed to because we're using the higher level abstractions. What you do see in Kubernetes instead is just really the ability to opt out of namespaces with things like the host network field, the host pid field and some other fields on pod which really just say I don't need my own namespace but they don't let you do this weird sharing of things. So next thing we need to need to understand is a bit about how Kubernetes does networking. Um So if we look at a typical cluster, I'm just showing two nodes here but if you had more it'd be the same. We kind of have nodes and each node has some networking device that lets it reach the internet, right? And the internet is bridging all our nodes

▶ 8:47 together so that we can reach them. This could be the public internet or some VPN or you know, any other way to connect a bunch of nodes. That's not really relevant to this talk. Um but we still have this issue we have the pods that are within the nodes. So how do we get the network in and out of those? Um so you can see that I've listed on the nodes these devices that I call eth0 which is common name for the ethernet device, right? What we see on the pods is these things called a virtual ethernet device. So just like a physical ethernet cable connects a node to the internet a virtual ethernet device is as if we stuck an ethernet cable in one end in the pod and on the other end on the host and it can allow you know, packets to flow to and from the pod. Um so with all these devices, we then

▶ 9:31 just have a bunch of rules on the node that are way outside the scope of this that just routes traffic through these devices through the public internet and gives us the Kubernetes networking experience we know and love or maybe don't love. So now comes the question like we're building this networking proxy we need to hook into this whole complex system. How do we actually do that? We're certainly going to go build two machines, get our ethernet cable, hook them up, or something just to run a test, right? Uh and while there are some advances already that don't require like real physical nodes to do these tests like virtual machines, of course, or uh Kubernetes and Docker, it's still a lot of pain even with those tools to get a

▶ 10:15 full Kubernetes set up. So, what we want to build is actually a setup where we have a networking like from a networking view it looks identical, but without the actual physical infrastructure there. So, this is a picture of like kind of my ideal network namespace and device setup where we don't have anything physically running here, but it looks exactly like a real Kubernetes cluster in terms of networking. So, you can see this picture is basically identical to the previous one, but instead of real nodes as like actual machines, we've replaced these but just a network namespace that kind of simulates the node. Instead of an actual pod running a real container, again, we simulate that with just a network namespace. And instead of like the public internet as this thing that is routing our traffic, we again

▶ 11:00 have a network namespace. Uh this is going to be a common trend of namespaces and namespaces and namespaces uh throughout this talk, so uh be prepared. Uh and then the only other difference is the devices. So, while we use a veth device in pods uh in Kubernetes, typically the node would have an actual ethernet device, right? We've also replaced that one with uh virtual ethernet device as well. And then this last thing a bit nitty-gritty, but we also have a bridge device which acts kind of as like the public internet would act uh in the real Kubernetes case. Um so, I'm I'm not going to go into like line by line how we set this up. It's fairly complicated, but I will just give a very very brief overview. I don't even worry about reading like the lines too. That's not the point I'm trying to convey. All I'm trying to show is that setting

▶ 11:45 this up is just a matter of finding the magic, and believe me, it is quite hard to figure it out, uh but set of commands that do the various things that these up. So, you can see for example like we set up up some devices, we add some routes, we assign IP addresses to things, uh and we just do just fairly standard uh network setup um to simulate this. And the nice thing as well is that when Kubernetes is setting up the cluster, there's this thing called the container network interface uh CNI, and they have a variety of plugins that do exactly this. Um and so like literally you can see the comment there that says like this script was derived from this CNI PDP plugin. Um so, this exact logic is how real pods

▶ 12:29 are set up in real Kubernetes clusters. Uh and if you were doing this, you could even actually run those binaries like the exact same ones that a cluster runs uh set these up. We didn't do that because I didn't think of it until it was too late. Instead, we went down this more complex route, but if I was to do it again, that would probably be a pretty good option so we'd have to reinvent the wheel here. So, now what? We set up this whole crazy like Rube Goldberg set of namespaces and devices. Now, what do we actually do with this, right? We We wanted to do testing and all of a sudden we're in like the depths of Linux networking. Um So, the last missing piece of this is how do we actually go and take those namespaces and run our tests on it, right? And the missing piece there is this convenient Linux syscall setns, which

▶ 13:13 conveniently lets us reassociate a thread with a namespace, exactly what we want, right? We can enter one of these pods, run some test workload, and it's just like as if it was running on a pod, right? There's a big problem here though, this word thread. So, most people here I'm guessing are running Go typically, um or even if not, most languages these days are not typically directly interacting with threads. Instead, we're dealing with goroutines, right? And a goroutine is assigned by the Go runtime onto a thread. And it can move which thread it's running on, and multiple goroutines can run on the same thread. So, if we were to just reassociate a thread with a namespace, we are going to be in trouble, right? Cuz our goroutine is going to move somewhere else, and

▶ 13:56 suddenly our code is not in that namespace anymore, and some other guy's code is going to go if on our thread, and suddenly they're in this namespace, and things are going haywire, right? So, this is not going to work for us right out of the box. There is an approach I've seen in a lot of Go projects. In fact, we actually have some Go code that looks quite a bit like this tries to and in each simulated pod, we

▶ 14:43 want to run real software like a real server, a real HTTP client, a real proxy. And all of these things are using many goroutines, right? So, this is not going to fly for us. Uh and by the way, if you want to do something like this in Go, you can do it, but don't copy this code. There's a lot more robust error handling and things that are needed to do this safely. Now, fortunately, I've been showing Go because it's convenient, uh but our proxy is actually written in Rust, and Rust gives us a bit more flexibility to solve these problems. So, unlike in Go, Rust doesn't have like this global runtime that you set up and that messes with these threads for us. Instead, we actually have more control over when we create a runtime, and actually the ability to create multiple runtimes. So, the equivalent function in Rust

▶ 15:26 looks a bit like this, uh which basically says that when we run around a function, we're going to spawn a new thread, and on that thread, we're going to create a new runtime dedicated to this thread, which can run our function to completion. And within that, we can have as many tasks, which are the equivalent of goroutines, and run kind of arbitrarily complex software still scoped to this network namespace. So, I want to show like when we put all this together, what does it look like to actually run a test? Um this is or to write a test, I suppose. This is going to be an example of a real test that I copy and pasted from our code uh that is actually like exactly what our tests look like with just a small abstraction we have to kind of help build these simulated pod environments. So, first we're going to spin up a

▶ 16:11 deploy ztunnel. Ztunnel is the name of our proxy. And so, what this will do is tell our test framework to go set up a fake node because we requested this is going to run on the default node. Uh go spin up a fake pod inside of that node, assign an IP address, set up all the routing so they can do standard Kubernetes networking, and then run a our proxy workload inside of that pod. Uh and again, all of this is just happening within a single process. Uh so, we're not actually of course spinning up pods or anything. Debugging is quite simple because everything's in a one single process. Next, we can also deploy say a server workload, which again just creates another pod, simulated pod, I should say, and then we can run a our simulated workload within that pod. Uh and again, same thing. Go run a

▶ 16:55 client that can connect to the server, presumably through a proxy if our test is working properly, uh and connect to the server, and then we can do some various assertions like we would do do things like check that metrics work, check that the request was encrypted as we expected, uh and all sorts of other things. Uh one small cool thing here is that all these fake pods are getting assigned IP addresses just like they would in Kubernetes, uh but we of course don't want to write our tests based on these random auto-allocated IP addresses. Um so, this resolve thing is just a neat little like tiny DNS thing essentially uh that just lets us look up from a name the IP address that was allocated to it. Um so, it's not literally DNS, but it kind of serves the same role. So, with this test, if we want to run

▶ 17:38 it, we can just use the standard tooling. Um if you're not familiar with Rust, Cargo is basically the same as the Go command. So, we just run Cargo test, and the test will run. Or or maybe not. Because after all that, we forgot that every single operation we did like in the past 15 minutes we've been talking, every single one of those requires privileges, right? So, we've just blown through and built all this, and it does not meet all of our goals, right? Now, we can just do sudo Cargo test. It works fine. We actually shipped this for about a year. Um so, it's not, you know, the worst thing in the world. Uh but you know, it's still kind of a pain. Like if you want to run in your IDE, you have to go and like configure some weird settings to make it run it under sudo,

▶ 18:23 and we actually made it so if you weren't root, like it would skip it automatically. So, all the time I'd be pushing code and like, "It works locally. It fails in the CI. What's wrong?" And it was just the test wasn't even running, right? Cuz I forgot to run sudo. So, we want to do better than that. We want to drop root. So, how are we going to do that? It is of course more Linux namespaces, right? So, just a refresher, I already showed basically the same diagram, but this is what our testing environment looks like, right? We have basically just namespaces on namespaces on namespaces. So, what we're going to do, and I'll explain this in a few seconds, is add some more namespaces on top of that. A user namespace and a mount namespace.

▶ 19:06 So, what a user namespace is is it basically lets us map a user within the namespace to a user outside the namespace. And so, we can say that within the namespace, root is actually equivalent to John in the host namespace. And this is very cool because now we can do operations that seem privileged as long as we're within that namespace. Now, if you're sitting there wondering how is that not just like the most obvious security vulnerability in Linux, it's because it doesn't actually let us escalate privileges beyond that namespace. So, if we were to try and read a file that's owned by the actual root, for example, it would deny that. Even though we're the user zero, it is aware of like the real user mapping and knows that this is not allowed.

▶ 19:51 This is a problem for us because a lot of the tools we run rely on files on the file system that are owned by root. Um so here's an example one xtables very nitty-gritty detail, but it's a file used by uh some of the networking setup we do. So again, like we just need more namespaces and we can surely solve this. So that's where the mount namespace comes in, which essentially lets us mount files within our namespace that's scoped only to us. That lets us essentially in our case fake files. So we can say that instead of this file or that file representing like the real file on the host file system, it can actually point to another file that we just set up like a temporary directory for each test. That way that's fully owned by the actual user and we have access to it

▶ 20:35 within our namespace. So I'm going to go back a bit. If we So now that we have this user namespace with the mount namespace set up, we're actually effectively running our tests as root for all intents and purposes from the test they are user zero. They have access to everything they need to set up all these networking constructs, but the actual uh like you don't need to run sudo anymore, right? It's all run as a standard unprivileged process. Uh so this makes running the test quite a bit easier and doesn't have some of the limitations we talked about. Uh but we still need a way to actually do this at the beginning of the process, right? We talked about if we're moving threads, we're leaving uh namespaces, right? And we know that for every one of

▶ 21:19 these simulated pods, we're definitely spinning up new threads. So we need to make sure that we get this uh this user like faking set up as early as possible. So if you're familiar with Go, um you you'd probably think, "Okay, that's fine. We'll just run it in an init function. It's going to run our code before anything else. We'll set up our namespaces there and we'll be good to go." Uh it's not quite as simple, unfortunately. The problem is that when a process starts up in Go, first we do like the basic Linux process initialization, then the Go runtime initializes itself, and only after that is our user code actually called. So while we like to think of it as like this very first code that's called, uh

▶ 22:03 it's typically it's good enough. It's not quite as early in the process as you might think. So the big problem is that the threads are created as the Go runtime initializes, which means that we actually need to hook in before the Go runtime. And with Go, there's not really a way to run your code before Go starts up itself. So again, we obviously just need more namespaces, right? That's the running trend. No, that's not true. This time we just need a little bit of C. Usually not my favorite, but it will get the job done here. So while we can't run Go code before Go starts up, um I think they would be mad if I even suggested the feature, uh we can run C code and link that into Go with feature called CGo.

▶ 22:48 Uh and there's this cool feature of the linker called the constructor, which basically lets us run our code at I don't know exactly when, but very very early in the process initialization done by the linker. So you can see we just define this simple function called enter net NS, uh which calls unshare. It's a really weird name, but it basically just means create a new network namespace. And we annotate this as a constructor. Um uh the usage of this is really awkward because it's not actually a function we call. It's called by the linker during the process initialization. So instead, we really just need to link this into our code by doing an import. Um so a little bit weird, but it does get the job done. Um so you can see here we import a few packages that sets up a user namespace, a network namespace, and a mount namespace.

▶ 23:32 Um our proxy is actually written in Rust, but we do use some of this for some of our testing in Go. Um so this is a real library if you are interested in using it. In the actual Rust version, it's basically the same, but we can write uh just normal Rust code and then annotate that as a constructor. Ctor is the library that that does this in Rust. So very similar concept. So now that we've gotten all this complexity, quite a bit of complexity, right? A lot of nitty-gritty Linux details. Did we actually meet our goals or did we just go off on this crazy tangent? Um So the first most important one is is it quick? Uh here's the snippet uh the test output from running all of our integration tests in uh the Ztunnel proxy.

▶ 24:16 Uh and you can see we've got 21 tests here and the total runtime is uh 0.8 seconds. Uh which is quite good, I'd say. Um Fortunately, all the tests are also able to run in parallel, which is not something we would have been able to do in Kubernetes either, at least for the type of test we're running. Uh because all these are isolated, we can go easily spin up 20 of these in parallel and run the tests um like that. Uh which is quite nice because each of these tests is taking about 250 milliseconds. So if they were run in sequence, it would take a few seconds. 5 seconds not so bad, uh but under a second even better. So I'd say that's uh mission accomplished there. We also saw from the the previous snippet uh that there was no setup required. I suppose I could have done

▶ 25:01 setup and not shown it, but I I promise I did not. Um and again, because everything's running in a single process, you and there's all standard tooling, we can just attach a debugger just like normal in the IDE and everything's there in a single process to kind of debug, put breakpoints wherever we need, and see things flow through the system. Um and again, as there was no setup, we didn't need Kubernetes, we didn't need Docker, and as we spent a lot of effort on, we also didn't need root. Um so I would say we did accomplish all of our goals. We've been using this in our real code for I don't know, a while now, maybe a year. Um and I would say well, it was a very complex and long setup, uh it did pay off quite a bit and helped us achieve our goals of

▶ 25:44 making running tests easier, faster, simple, uh and easier to develop. Um one thing I do uh want to talk about is a bit on the runtime. Um like I said, I'm really impatient, so 0.2 seconds is pretty quick, but also it's 0.2 seconds I'm waiting and I would rather not, right? So I was always wondering like why is it that it takes that long? Um and the reason is that there's just a lot of stuff that's not that fast. So here's an example of benchmarking uh just creating the network namespace and user namespace, um which is with this cool benchmarking tool. And you can see it takes 6 milliseconds to set one up. 6 milliseconds is is pretty quick, but we set up a lot of namespaces as you saw, so it adds up over time as we're creating, you know, if we create five

▶ 26:27 simulated pods, it's like I don't know, 20, 50 namespaces that we're setting up for these tests. Uh and even worse, it doesn't actually parallelize well. So here's basically running a different benchmark, but we're running 256 of these in parallel, and you can see it's averaging 1.2 seconds each. So unfortunately, a lot of these calls uh in Linux are blocking and they take kind of a global lock on the system. Um so this kind of ties into like real-world Kubernetes. I've often wondered like I asked for 200 pods, why did it take a whole minute for you to give me those 200 pods? Um and the answer kind of is that there's even at the very lowest base layer of what it takes to run a pod, these operations are not instant. And then on top of that, there's of course even more and more abstractions in terms of scheduling the

▶ 27:10 pod, creating endpoints, and all these different things. So uh when you consider that, it's actually quite a wonder that it's able to schedule pods so quickly and run them. Um so with that, I want to thank you all for coming out on a Friday to talk about testing. I really appreciate it, and we do have 7 minutes. Uh we have some mics here if anyone has questions, uh and I'll of course also be outside afterwards if you want to talk more. Hey. Hey. Um thanks for the talk. Um

▶ 27:53 So in the Rust code, um you used I guess it's assuming that's using Tokyo for the runtime, and that works fine, but does that preclude spawning actual OS threads? Will that break out of the namespace? Yeah, if you were to spawn a thread, uh it would. You could potentially set up a system such that you, because you're in more in control than in Go, you could set up a system where you spawn a thread and then make sure you enter the actually take that back. When you spawn a thread, it should inherit the namespace of the parent. And so that I think that would actually be fine. Got you. Uh we're not actually doing that in our code, I believe. Uh so it hasn't been an issue, but it would work fine there. Okay. It's more of an issue in Go because you have absolutely no control over the actual threads. Yeah.

▶ 28:35 Yeah. Thanks. Yep. What? You have a mic here? Uh I don't think so. You probably have to go there. Yeah, I just kind of a similar question like he with the Go example, if I if if the code's been up a routine in Go, so would it break? Uh that's part of the tricky thing is honestly, a lot of times it probably wouldn't break because Go may schedule it to the same thread, uh which is almost worse because it may appear to work fine. Maybe it works fine on one machine that only has one core or two

▶ 29:20 cores makes it less likely, and then you go to deploy it in production and suddenly it doesn't work. Um so the short answer is no, it doesn't really work. Um so there is a lot of code both in Istio and other projects like a lot of the like, you know, networking stuff in Kubernetes is doing things like this, and I think they often just make sure that the code that they're executing in the namespace uh is very small, careful not to run goroutines, uh and take cautions like that. So. Got it. And and for the 40 rust example, so why did you so I remember you showed it go example and then you move to the rust example, but then why did we need to do the unshare guess there's more slides than I thought sorry. Okay,

▶ 30:03 so if you go on one slide back. So then you said okay like we go. We cannot enforce but then you go you went to the rust example which you can but I didn't understand why we did the unshare link. Like we to see go in the rust as well. Yeah, so the unshare stuff with the C go in the initialization that was for a separate purpose. So this code on its own is perfectly fine for entering a namespace and doing an operation and leaving it. What they part about the constructors and the username spaces was about getting us an environment where the entire test process is basically root, but it's not the real root which allows us to do privileged operations. So one example outside of like the proxy. Is an issue we set up IP

▶ 30:47 tables, right? And we use this approach with the username space mapping to run the test as this route and actually run real IP tables commands against the real kernel and actually apply them and test them that way without requiring running as a root and that's something we do with the go code specifically. So I see it so so you needed to be forwarded runtime initialization just to yeah, because otherwise we we could enter. You can enter it the username space just for one function call but then you have to you know then you have the constraint that you can't spawn go routines and you have to make sure every test calls that. So for us at least for our use case we needed it so that the entire process had that kind of simulated privilege access. Got it.

▶ 31:31 Appreciate it. Thanks. Yeah, actually before like in in rust we initially weren't using the constructor approach and we just made sure every test like wrapped the entire test with that setup. We moved to the constructor approach cuz. Just easier and you didn't have as many steps. In rust like we were initially not doing the constructor approach. We were just having each test individually set that up and we moved at at the very least because then you don't have to call a function on every code and I think there was also something about it making clean up easier but I don't entirely remember. Thank you for great talk. So you want to send correctly the virtual ethernet adapters and bridges so this stuff goes

▶ 32:16 L3 over L4 networking did you had any chance to measure what kind of performance application does it have? I mean to set up you shown but any idea how so we're not doing like we're not running our our like benchmarks against this. So I want to say that we've tested it. I would expect it to be fairly small. Like there is a like in the real world in kubernetes. There is a cost of a pod traversing the the virtual ethernet device that I've seen a bit about. So there is some cost to it but for us it hasn't been a big deal because losing 10% performance on our test is is not a big deal for us. It's about 10% yeah, I mean I would if I had to pick a

▶ 32:59 number I'd say 10% but it's a wild guess really. This was an excellent talk. One of one question that I had was you kind of alluded to namespace creation in the Linux kernel being like a blocking action that was pretty variable in its timing. Was there anything that you all picked up in doing this work where you're like in order to further improve kubernetes or istio or something we need to start making changes. At a more fundamental level in the system to make things as performant as we need them to be or like

▶ 33:44 future looking to like enable kubernetes to like hit more spin up pods faster that kind of stuff. Yeah, I mean I I looked into a bit like I did like did a flame graph or whatever found there was some lock somewhere in realized it was like deep in 2000 lines of code in the Linux kernel and called it a day. So is there stuff that could be done probably I had assumed that. There's a lot of really smart people that are working on this code that they've probably optimized it hopefully I hadn't explored much there but potentially there is more low-hanging fruit not low enough for me to grab though. All right, I think that's everything and

▶ 34:28 we're out of time. So thank you everyone and hope you have a good rest of your cubecon.