Container Runtime and Image Format Standards - What it Means to be “OCI-Certified” [I] - Jeff Borek¶

Playlists: KubeCon + CloudNativeCon North America 2017 (Austin)

Update on OCI runtime and image format standards and what OCI certification means.

Topics: oci, container runtime, image format, certification, standards

Transcript¶

▶ 0:00 all right well good morning and welcome everyone I hope you're here for the container runtime an image format standards update I want to start by just welcoming you all here and I'm going to turn it over to my esteemed colleague Stephen Walley and who I'm in awe of Stephens a sage open source professional been doing this longer than me but not by much I'm not hot I told you I could break anything like that mic up now I'm hot there you go thank you I'd better step in before Jeff gets too

▶ 0:42 carried away here my name's Steve Wally I work in the azure engineering team and I've just recently gone back to work for Microsoft after a dozen years away just to kind of get a sense of folks in the audience to folks know what OCI is show hands that's good and developers also good operations operators folks like that excellent thank you and lastly managers and other folks trying to figure out what this is all about excellent thank you for your honesty okay I'm gonna turn this back over to Jeff to get us going great oh well thanks and you've got to to

▶ 1:26 Northwest Nate at alight residents generally speaking any other Northwest folks in the audience I wore my Seahawks shirt just we won't do the chant though I'm gonna go ahead and get down to business so we want this to be very interactive I'm going to cover the kind of the first half in a little bit of the background and Steven's going to jump in on the organization and what's going on with certification and then we're going to have time for Q&A and with that how many have heard of a little startup called docker raise your hands that ought to be everybody in the room cuz it was really interesting if you turn the clock back to around 2014 in the spring I was at an

▶ 2:08 openstack conference and there they weren't presenting at that point but the buzz in the room was all about you know hey this company docker and have you seen what they've done with container technology and so I got plugged in pretty quickly but the point I want to make on this first slide is that containers have been around for a while and I don't represent this slide is exhaustively complete right so if I've left off something that's a favorite of yours I want to apologize right out of the gate but you can see that you know freest freebsd started with the concept of jails early on

▶ 2:52 you had linux get involved sun was certainly a player with what they did with their zones technology Google started leveraging the heck out of containers back in the oh six time frame and obviously did a tremendous amount with that technology to be able to scale out their global ad platform Red Hat and yours truly IBM got involved with container technology and that all of that led to the point where if you've seen Solomon Cakes talking about this you know he's used the phrase in the past standing on the shoulders of giants

▶ 3:34 right because while hats off to what docker did in terms of enabling to make containers more approachable and usable by the typical developer it really is a cumulative effect that's got us to where we are today so when I met the docker team back in 14 it was still early days and yet there was a little bit of confusion in the ecosystem if you will because you had this very viral open source project around containers called docker and then you had the company docker as well and so is it an open

▶ 4:17 source project is it a company how do you sort of sort through that and I got involved early on and started working with a number of other people in the community around this concept of moving towards open governance and ultimately that led to an announcement made by docker at docker con back in 15 the middle of the year around this idea of the OCI or the open container initiative a single open container specification that everyone could collaborate on and everyone could agree that would help create some of the foundation of this technology and you can see the concepts around not being

▶ 5:01 bound to a high-level construct not tightly associated with any particular commercial vendor or project but a multi vendor open governance structure and the intent for this was to help enable portability across multiple platforms that principles associated with getting this spun up when we worked on the original documentation for it content agnostic infrastructure agnostic designed for automation and industrial grade delivery so there were quite a few companies that got involved early on and one of the big players in addition to

▶ 5:45 docker early on was the guys at core OS and the folks there had a solid team that was working on this collaboratively and they also had a vision of how to execute they came up with the rocket project the last thing that the echo system really wanted though was to see multiple competing solution alternatives so the OCI worked to help bring all of that into a greater sense of alignment and you can see the logos of the other companies involved with the effort this has been picked up tremendously beyond the initial activities by docker and

▶ 6:28 core OS maysa the CN CF with kubernetes cloud foundry all of them look to the OCI as a fundamental in of the basic technology and in in terms of who's been involved to date the contributors hats off to Red Hat no pun intended they've been a major participant in helping to develop the initiative Huawei Fujitsu core OS suzay those are some of the top contributors IBM itself is in the top ten and that's actually when I got to know Steven better he was doing consulting work for a docker on this project

▶ 7:11 and we started to collaborate not just on the OCI but starting to think about well where we gonna go when this becomes firm how are we going to try and create some sort of a certification process around this and what was that was that like a year and a half ago time flies when you're having fun absolutely so we've got a high level representation of how this is structured and Steven when you first got involved when did you first when did the OCI first hit your radar so so I was actually at peal at Packard when this was all happening and it was one of those things where you ended up

▶ 7:55 in a situation where containers and containerized workloads were gonna make such a difference compared to the kind of infrastructure arguments that folks were having in the OpenStack community that's working for Hewlett Packard I thought we should be involved in the the frightening thing was it was all based on a single company's technology at the time docker really made the container echo system usable and so you know I was pushing to get HP involved at the initial bootstrap and Jeff and I have both done enough standards work in our careers that sooner or later it's going to come down to a certification debate that's where it doesn't matter how long the standard takes or anything else sooner or later it's going to become a

▶ 8:38 certification debate and so as I stepped out of HP cloud I ended up going to work for docker for a year so yeah that's we've overlapped almost since the beginning of this yeah the in fact how many you have heard of the docker governance vizor II born six or seven one of the best-kept secrets into the industry all kidding aside in the early days docker was sort of conflicted because yeah they they wanted to share they obviously put that out an open source but they also were starting to think you know how are we going to pivot from the start up to more of a going concern or an enterprise

▶ 9:19 and it's it's great when docker came to that realization that they needed to make this contribution to help to kick off this project in a more meaningful way the timeline associated with this you can see that this is taken two years and in the era of open source that's a long time but in the era of standards it's really kind of a bit of an eyeblink because getting collaborative cooperation and getting consensus amongst such a large ecosystem on such a foundational technology was very very important and I think that that's a really important point here what the thing that's been different about the

▶ 10:03 open container initiative from a lot of traditional standards organizations is it's a group of collaborative open-source engineers and they're trying to build a specification and those two things are almost the antithesis of engineers collaborating with engineered standards you know you kind of have to get it right the this idea of doing things on Internet time is based on a foundation of a group of engineers thirty years ago took the time to get it right so we can be moving it in at a time at the same time when you're living in an open-source world and as we've watched these projects evolve and well run projects are on this cadence of

▶ 10:46 releasing regularly and frequently that's a different kind of engineering collaboration so open container initiatives worked really hard to try and find that balance the fact that they kept the target lean they started with enormous contributions from docker for the base technology both for the reference implementation as well as the the overall runtime it's it's allowed the group to make really good quick 2-year delivery of those first two specifications and so just a bit on how the OCI is organized there's a technical developer community and

▶ 11:30 there's the organizational membership and we're not going to go into great detail on the org structure but we can get into that a bit during Q&A if you like the important part is that we really wanted to a have this be a level playing field right right very much so and actually if you bring up the next build out you know this is an organization that was deliberately designed to get out of the engineers ways so that the engineers could be doing the collaboration to build these specifications Jeff and I are not those engineers Steve and day in the front row is one of those engineers so it's you know it's one of those things where we were trying to keep a very lightweight organization that's also one of the

▶ 12:15 reasons that this didn't go to a more traditional standards organization those processes can be very heavyweight and so it's trying to find that balance of creating useful specifications and at the same time letting the engineers get it done quickly or as quickly as possible yeah if you're an engineer that's worked on the OCI raise your hand I've got Steve Oh up in front there's Michael Brown we've got looks like a half dozen of some of the usual suspects here in the room with us again hats off to them it's it's taken a lot of hard work to get where to where we've gotten to at this point so this is a good time to start to pivot and get into the details associated with the certification itself

▶ 12:58 and with that I'm going to stand somewhat quietly and turn the mic over to Steven so this is the dangerous part because Jeff built these wonderful slides and I start to disagree with that's the joy of the process right essentially there's two kinds of certifications that there's two specifications and so there's this idea of a no CI certified runtime and kind of the portability promise that that would make to folks that are building containers and then there's that idea of a no CI certified image and that is so that this this image that's been created is this is that the entity of portability if you will part of that lightweight structure that was created under the auspices the Linux

▶ 13:42 Foundation allowed us to do all the trademark tracking so there's a collection of trademarks that are listed there these are trademarks that the Linux Foundation will defend you know for on behalf of the OCI project so this is one of those things where you can't call yourself basically an open container sorry a OCI certified runtime or an OC is certified image unless you've survived the certification program this this is the point of this is to actually provide the marketplace information about where we're to look for things of value that said we're also in this place

▶ 14:23 that as we just start to step in there is a draft of a process it's a process that you would imagine it kind of looks the way you'd expect a process to look like and this if anything was the easy part it's coming up with this because you know really we're looking at a process that's normal for these kinds of things you start off with the application you basically this is this is who I am this is what I want to do I want - I want this trademark you take it through and at that point we've sketched in kind of the basic information that we think we want to have and the next thing

▶ 15:08 you do is you test it so we're gonna go through and test the products against these two things I'm sorry I'm having a hard time because I'm just getting used to having readers for the first time so I need them to look at the slides but then I have all of you and I look up and I think what happened to the audience it's very disconcerting right now after you've run whatever the testing environment is of course that's when you publish your results and you have to what the thinking up till now has been is you're not just publishing your results you're publishing all of the framework around that so that anybody else could repeat this and that that's the intent of this is to make sure that this is a highly repeatable system and that can be checked and validated by

▶ 15:52 anybody else and that's kind of the peer review is to finally get that certification you'd somebody within the rest of the organization as a collection of peers will validate that yes I was able to repeat this and at that part you know you you can actually claim the certification so that's the easy bit so to speak that framework feels like probably everything else you've seen in many different guises and you know on the back end of that is this whole idea that you'd you know then want to promote your brands that you've earned to the marketplace some of some of the where the debate still lies though is there's two places one of them is the test

▶ 16:35 harnesses work is still ongoing on the test harnesses and it's there's good work has been done and it's evolving right now so these are we aren't quite ready to announce things yet but we're close the interesting debate that I still bring and and I am happy for opinions from the audience when we open this up for questions having a certified runtime I think is kind of the base entry into the game having a certified image I worry about and I worry about this because I've seen this situation in a past life and I'm not going to use the p-word today I've seen the situation in

▶ 17:17 a past life where folks were so excited about having certified operating systems which made sense to support a portability model but then they weren't they wanted to be the organization that was running this set and then we'll have certified applications and that's starting to feel like certified images and the problem when you look at it from an application perspective is if you're the vendor you're probably not going to certify if that's a promise to the marketplace that you'll obviously run on all of these places you haven't tested because that's the reality of being an ISV you're not gonna promise your customers

▶ 18:01 regardless of what the the the platform vendors have told you you're not gonna promise your customers that it's gonna run until you've tested it there and I think that that tension needs to be resolved here so that's part of the debates that we're still having ongoing inside of the certification work but it's that doesn't mean that you couldn't the world wide web has you know for w3c as validators so you know this is a valid page that's a that's a great piece of information and having that kind of toolset come out of the certification work so that you know that you have a valid image and you should have confidence that this will run on a certified platform is a good thing so I'm not saying that that tooling doesn't

▶ 18:46 have value and that we shouldn't have that kind of validation I'm just saying we need to be careful as you step into the nuances of a brand program about exactly how you what promise is being made to customers on the end it really can be hard to strike that right balance because obviously if the collective group came up with this draconian certification process have a lot of overhead have a lot of conflict and a lot of stress about how to resolve that conflict on the other hand if it's so lightweight and it's toothless then you're you're not solving a problem for the marketplace right I think the the the great thing that has been accomplished in the two years up to

▶ 19:31 the release of those specifications is it was the the collaborative effort all the engineers from all the platforms were in a room together having having that debate you know it's the fear of the certification program that's coming to ensure that they're actually aligning the technology and you know writing it down and agreeing to it so I think even though we you know there's been this gap from when the first specifications were out the door were still in a place where enormous value has been brought to the market in that collaboration and we're still working on the details on how we'll

▶ 20:09 demonstrate that in a broader context so there's still time to get involved these slides going to be available up on the Linux Foundation website so I'll post them up just shortly after we finished today but there's the link for the open container site as well as for the developer communities and with that it's time to start to transition to QA and whilst you guys are thinking of your first original question I'm gonna go to Steven and ask him what was the biggest

▶ 20:52 surprise that he thought he ran into you over the last 15 16 months in this effort we've been under way and I'm not allowed to use the P word Emily yes you are no I'm not yes you are he's talking about POSIX yeah I I was part of the the UNIX wars back in the 90s and so that that's where I got my gray hair from what I was impressed with is having having been through that effort to see the good work that was done in this organization in a short span of time there it was funny because it was a group of really knowledgeable engineers that are used to collaborating in an open-source context and they weren't

▶ 21:37 used to building standards and so there was there was always that tension if they wanted to be moving faster and at the same time the specifications that have come out are very useful specifications and had that right effect in the industry of ensuring that there was a central point of agreement for things going forward as containers become more important for the way people start deploying software so I think the fact that they were able to without necessarily going through the whole learning curve idea build a standard to actually come through that learning curve quickly and deliver on these first two specifications I think was a normal so helpful it looks like we have a question over there and while

▶ 22:20 he's getting the the mic - Mike I people run a just say that I think the other thing that we tried to help Shepherd the engineers along with on this was that how do you try and structure this in such a way that you don't stifle future innovation right I mean the other thing about this o CI process that was of a concern is that you know if this locks it down you know in such a way that it prevents future innovation that's not a good thing for the industry as well question four yes all you're gonna certify the runtime processing tools that container runtimes that kind of stuff and format maybe images but it was more of the format right but to me the

▶ 23:05 image format should barely be verified at the tooling area as opposed to you know each image itself for example you know a build tool that builds these images you might want to make sure that it does build OCI images correctly right yep good feedback other questions 20 and while that questions getting teed up pull out that smartphone and follow Steve Wally on Twitter I'm kind of been following the standardization process at one remove from the time when I worked at core OS I was in the Pro Services Department so I wasn't like a developer or anything I have not read through the spec but I was wondering does the spec

▶ 23:48 say things that container runtimes or images must not do or if it doesn't is that something that you've ever considered or thought about are there things that we should say no you should never do this that's a good point Mike you wanna yep yeah and and and that's really important because the experience that I certainly went through I I got involved in POSIX late and so when that first speck came out you know there was just things that were left unsaid and there was no flags around that to say you know really we aren't saying anything here and so it's

▶ 24:33 actually very helpful when when you start in a specifications organization that you're able to do that the marketplace couldn't have allowed it in that situation because there was just when you're trying to bring together dozens of products in a marketplace worth billions sometimes you have to leave things unsaid whereas in this situation because this is still a relatively new space we were in a good position to actually build up and and make the claims you know don't do this other questions well that one's getting teed up how many have heard of container D you'll want to stay for the next session my esteemed colleague Phil

▶ 25:17 Estes who's a docker captain who works for IBM is going to be doing a deep dive on container D in this very room in about 15 minutes so I'm a software engineer DEP Sarah I have looked at the runtime expect for awhile and it only includes what you should specify it to run your image like CPU and other stuff but it does not describe anything about the API itself like how the runtime should expose its API and do you have anything in mind for including API also or is just gonna be just how you specify the how do you run the image yeah we had a philosophy of

▶ 26:01 kind of crawl walk run and we didn't you know collectively companies that were involved in supporting this to varying degrees were worried about over driving our headlights so we tried intentionally to start with the very basics and there are next version topics that I think are getting teed up and that's certainly one of them distribution the other things right the other thing that we really are also trying to do with the OCI is think of the you know if you think of the Asian yin-yang symbol of black and white and synergy we're trying to keep the OCI focused on the fundamental container

▶ 26:44 technology where we're not trying to take it you know across the line if you will into the orchestration in space we're trying to be very complementary obviously to what the CNC F is looking to do with container orchestration Stephen yeah and that question is it's frustrating for me because that would have made certifications so much easier if we had a had that and and so that that exact question has been an active debate for a long time but at the end of the day there was also the we we were the engineers building the spec were in a position that they could actually claim victory at that point and that was a good thing to do at the two-year mark

▶ 27:29 so that's why it's still an active discussion there's proposals underway that will allow us to kind of do that have another question over there yeah have you guys discussed image distribution as far as like you know standardizing maybe the registry API is or something like that or do you leave it up to the runtime to pull images however it sees fit actually allow me to take this place here oh no you clapped you clapped and you're one of the engineers helping build the specification so I think you should

▶ 28:12 share an answer here the top the topic has been like it was just said a second ago up for debate since even before OSI I was a thing and the very first steps were the the image the actual image format something that could be distributed and the runtime so now that the v1 is out for the image and the runtime spec and a light sabbatical for all the people that were involved for the last two years everybody's teeing up what we're gonna be working on for 2018 there is an OSI I face to face tomorrow so besides like just updates in current events the two items on that agenda right now are a

▶ 28:57 distribution API and a way to a common way to build OCI images so it is very very much demanded and needed because there's plenty of de facto stuff out there but it needs to be iron dome and thank you Pinson bats from Red Hat oh hang on hang on Luka Jo Mike damnit Steve what steve-o steve-o if you can distribute a content addressable blob you can distribute an OCI image so it's it's all about figuring out how we distribute those like it like once you have the content IDs you can distribute it over any medium that can do those content IDs and figuring out how to do

▶ 29:41 that is the problem so does that mean we should have an distribution spec probably but also right now it's pretty open and how you can distribute an image because once you get to that ID space you can be really creative and that were the kinds of things that we get to debate in this and so that that's why I like the work is they are starting to ramp up again for what the effort will look like for the next phase of specifications in 2018 so if you're interested in your company has a perspective or you want to share your expertise I would encourage you to participate see I mean just a question like that Steven if he was the dog Stevens tail would be wagging because he

▶ 30:25 he just likes this stuff like is a different kind of work maybe too strong we've got time for one or two more questions before we wrap up and Steven as we start to wrap up I'll have you come up here for one last event but a couple of closing remarks I have a thing or two you want to make it nope that was really it is I would encourage you to come and participate if this is something that affects the way you want to start thinking about things at scale it's not just a game for vendors there's lots of bleeding edge users out there that have real experience with how

▶ 31:10 they're doing things like deployment today and so your opinions are valuable my initial step into the standards world a very long time ago I was on the user side of the house I was working for EDS in a General Motors Factory and we cared about the standard and thought we'd go find out you know what it was all about and that was a very fruitful number of years in my career so from even just a understanding how specifications happen is always good in this industry and if you'll indulge us we have to go on posterior record for this so if everyone in the audience that enjoyed today's session will raise their hand

▶ 31:54 say yay thanks for that seriously how many got something out of or enjoyed today's session just a quick show of hands if all of you would take a moment provide that feedback to the Linx foundation Stephen and I would greatly appreciate it if you found it a little lackluster well you've got better things to do with your time than to respond to those surveys so oh great really do you appreciate it thanks everyone for attending do stay for the container D update following in this breakout

▶ 32:37 session and without a big round of applause for my partner in crime thanks very much folks thank you