Deep Dive: CNCF Serverless WG/CloudEvents - Clemens Vasters & Vlad Ionescu¶

Playlists: KubeCon + CloudNativeCon Europe 2019 (Barcelona)

CNCF Serverless WG deep dive with a CloudEvents-based deployment pipeline demo and architectural considerations.

Topics: cloudevents, serverless, deployment pipeline, eventing, cncf serverless wg

Transcript¶

▶ 0:00 hello welcome to the cube con so you're in the dip-dyed session for the CN CF server last working group cloud events was the first big project that we actually had so we have two items for you today there's gonna be a demo a deployment pipeline presented by me and then Clemens yes I will be talking about some architectural considerations and things that we haven't done won't - and might do later so let's get into the demo okay first of all I build this demo for Amazons they do video conversion in the cloud instead of having video be

▶ 0:41 transcoded and a movie just in hours or days is being done in minutes cloud very nice they also have another product which is a straining delivering video to your end users everybody's building their own streaming service right now so this is helping a lot with that and they just announced the frame DNA which is AI for video if you have multiple versions say a version that sensor does version then does not censor this helps a lot with managing this owns owns was and is a start and as a start-up they the biggest focus is delivering a lot of features fast to get feedback from their customers to improve and to validate

▶ 1:24 your market use fortunately they've had great success and that means that now they do have a paying customers they have to focus on reliability so they need to do the whole switch from yes while building fast we were breaking stuff - okay well building fast but we are focusing on reliable they were focusing on the customer experience so to help with this what we thought we would do is a deployment pipeline that's something that was missing we kind of have something right now which is working but it is not ideal where we want to go is we want to have a fully observable deployment pipeline so if something fails at any stage in the

▶ 2:07 pipeline we want to know about it we don't want to spend hours looking through log the switching contact between github circle see our other tools to see where a build fellow where a deploy is down also working with video they need to adhere to a lot of compliance standards MPAA is an example and those are very strict so the whole deployment pipeline needs to be very very auditable and it needs to have steps like okay somebody needs to approve this deployment going to a client environment manually also from a development perspective this needs to be fast and it needs to be scalable building developer environments at the start of every day they're obviously going to be a lot of

▶ 2:51 deploys and at night just a couple and having this kill for this would be absolutely perfect as we don't want it to be slow in any way as that would impact velocity so we started thinking about okay how should we build this obviously the first step is getting data from github all our social our source code is hosted in github and github centralized feature which is called records and they send you a web hook a big JSON containing everything that's happening in your repo say you just push the commit do you have this be as full torrent as possible and as reliable as possible we don't want to have just a

▶ 3:36 single point that we ingest events from what we have fear if I get this point to work github attending the event to an SNS which is a message gif for all my dahlias and then other kids subscribe and duplicate the messages they duplicate these events and they can each work independently they can only subscribe to some events that they care about I only care about to issue comments I don't care about commits because I'm a bot that does github issue automation stuff it's pretty much the same thing for circle CI when a build is done where a test has failed I want to know I might be interested in different events in different types of events

▶ 4:19 sorry so this is what we came up for here the proof of concept and the demo that we'll be presenting today there are the SNS is I just spoke about the circle CIN give happiness is they all send their events to a lambda which of course them in a dynamodb table this dynamodb stream if you're not familiar with it think of it as tour procedures but more powerful each time there is a line change in DynamoDB you get notified and you can run lambda you can forward it to wherever you want to do anything with it this is our github event is gonna be

▶ 5:05 first it's gonna go to lambda that's gonna put it in dynamodb an event is gonna go from dynamodb to you the other lambda and this is Nancy okay this is not ready to be promoted this is the promoter lambda because we don't have an image yet a little while after the circle CI event is going to get triggered is then I'm going to die nobody be into the stream the lambdas no I see okay I have an image I can deploy this and then I put the a message with height please deploy this for me into another SNS which is going to be picked up by lambda and deploy to ETS using how so let's talk a bit about this this is the first step and it's github this is

▶ 5:48 what we would be interested in github we want to know the repple what app is this happening for you won't know what Brent we're on master branch might have different promotional requirements as compared to say a release branch you wanted to commit we're going to for tracing we want to have full visibility we found out that the best tool for this is honeycombed honeycomb needs to propagate a trace so give up the call and I make you happy event you can do to land that which is going to put all this data into dynamo dB sounds good the

▶ 6:31 circle see I even then is gonna come and it's gonna mark the build as true the build was successful the build was done if the doesn't ever come okay it's clear the bill failed we can see that very clearly in the trace we can see where the issue was obviously these is just this is just a demo so there are just two data sources Circle CI and github you can add as many as you want what what a promoter lender does it checks if they are - you can do whatever you want here so you can check hi do I have approval for illegal do I have the static code analysis do I have the security check do I have everything I need in here there's a sponsor version there are a lot of other

▶ 7:16 tools you can integrate in that as a start-up especially you want to be able to integrate easily fast and to be able to plug them in and take them out as it's needed okay so I guess now we know what we want to build we know like what dataflow will be what events will be generated and how we'll be acting on those events so now we now we got to do even design we want to see what events are flowing to the system and how we

▶ 8:00 design them so what do we need we need the even type we need the source is it give up where is it Circle CI we need a branch and we need a git commit job okay but we also want a header and it's a trace other which is terrible it needs to be serialize it needs to be serialize it needs to be taken care of and as soon as we got to this place we were like okay building our own JSON our own data structure might be a bit too much we don't want to do that but hey we know about clergyman's cloud events has the exact structure and it enabled us to very easily put our data and not be

▶ 8:44 concerned about it cloud events is not making any assumptions about your actual and data we just metadata that goes are around even you can put whatever you want in your data so you can go as well as you can this is just helping with routing so do you want to know if there was build success you can just yeah subscribe to this event Janna alert or have a special automation for fail bills you can just subscribe to it you can just subscribe to the events you're interested in so let's go through the demo first of all special thanks to Elena who actually wrote a lot of the

▶ 9:25 code let's go through the demon is gonna die okay the internet was not helpful we have a simple Python app which take you can Europe which is very nice but this is not your this is Keep Calm + clone 84 so let's change that and we just commit to this what happens right now is we're going through that whole pipeline that I show before we're going through that whole level github we sent a web hook to the lambda the lambda is forwarding it circle CI has been

▶ 10:09 triggered and is building this if I go to circle CI and if the internet works I can see that the build is right meanwhile let's see that what's happening with the traces okay so we can see that with our event got into the github automation part we did validate the github signature we did validate the secret this is a genuine github event we generated the cloud event we can even see it we've attached it as contacts it's basically four encoded but it is

▶ 10:54 there and then we want into the handle github event part we read the previous load event we were generated we created our dynamodb table which was the one above and you can see some little here circle see I build on these fools is not ready we wrote that dynamodb and the promoter lambda got him and shut check for promotion and we have our f-type promotion failed why Circle T I build not that so the circle via build is still happening looks like it is done so less I'm not gonna refresh that by hand because I know it's acting weird open

▶ 11:32 okay what there is a missing span somewhere I don't see it but okay so in here we can see the full trace why spoke early about the level of the of absorbability that we are very very much interested in so this is the span that's missing okay so we're at the close if we got what is this apologies it looks like some events did not get they did the lambda flushes the events at the end and most likely it would it was still deploying the whole chart so

▶ 12:14 that's why it was and ready so let's look through all this now that we have any so automation promoter check for promote promotion promotion failed this is where we were before it waited it waited the circle CA even came in the promoter function ran again based on the DynamoDB stream it checked for promotion it's all add that's ok so it created and some with the cloud even further away and then we got into the held install automation there's also ready vent it did some preparing we are in a lambda using home we are authenticating to e KS which is the I am a WSI Authenticator so

▶ 12:58 we need to do quite a lot of stuff for that we install the Authenticator we validate that installer we have access to it so if there's any issue with our lambda we do want to know about it we're preparing for help will anything we'll we are adding our Apple which is using the hub s3 plug-in which is amazing and I highly recommend we even get the output so if there is an issue say we take only change the bucket policy and this lambda is no longer allowed to access it we will be able to see that here and then we did install so we have the whole output and if we go to the coral that has been running in the background you can see hey Keep Calm

▶ 13:42 Plus cloud native corn Europe which is exactly the new current and the pods that are running any case are all around 2 minutes also this all happened between the demo the deploy was fully automated it was very visible it was clear and it was very very fast and scaling super fast this also allowed us and encouraged us this is a la hives in server last years as when you're doing lambdas and several eyes you are forced to invest heavily in observability you cannot s si Jin you cannot do that you can get well I'm not SSH back to you but don't do that and being forced to properly instrument

▶ 14:26 the applications to properly get visibility into it is exactly what just wants to add to its all products and what we've been doing for the last month and this has been extremely extremely helpful because it's been a safe place to play around there now let's get back to the slides slideshow play from across floor okay so up sliding hey stuff didn't work so about the external events experience this greatly simplified or design process we didn't have to do any encoding our souls and decoding we have as decades in most languages some

▶ 15:11 lenders were written in golang somewhere in Python and this allows us exactly that flexibility it also comes with same defaults what really helped was exactly the event I which is a top-level property so we can filter by it and other automations can very quickly say yeah I'm not interested in that event just give me commit just give me issue comments and again flow defense does not restrict what you put in data and all it just adding same default same values to help you around that as you've seen I haven't thought that much about cloud events they're here they're boring you can just use them because don't design your own JSON though design your own data structures because you're gonna get

▶ 15:55 issues and now we actually have something that's focusing on interoperability that said there are some best practices and there are some things you should consider and for that you have Clarence thank you very much you landed exactly on time that is really great okay so is it not working of course no it's not working okay next so okay you will be my clicky everybody go back to that side great so my name is Clemens I work for Microsoft it's not a startup and I work in the messaging and the

▶ 16:37 venting team we own service boss event great event hubs relay and we also own standardization in that space which means were engaged in C&C of cloud events will also engage in several industrial vertical standards and were and I'm the technical and the chair of the Technical Committee for mqp and Oasis so a few architectural considerations one of the things that you should like to take away from the demo if I can say that is standardization helps because it lets you build an application one application based on heterogeneous event sources and

▶ 17:18 lets you handle this play and and filter events in a common way or you only need one infrastructure if those if you have these events in different formats you have to have a handler and filter etc for each of the event types and here we all make them the same we have a effectively simple model we have the the minimal set of things to treat all those events alike and that's where standardization makes sense so next slide so let's go in and think briefly and this come it guess it goes into the best practices section what events are and and how they are different for messages because that's a very important architectural concern one

▶ 18:03 of the things that is there's a slippery slope here and you will just want to make you aware of that events are not messages in the sense of telling someone something to do and they're not RPC it's not the same thing we're designing for a very particular subset of messages so a event carries a fact it carries something that just has happened what you just saw in that demo is in github you did something you committed something and now actions happen based on that that just happened they propagate through a system the great thing about those facts is because they prescribed the describe a thing that has already occurred you

▶ 18:47 can go and fan them out to everybody and you can go and write them down you can put them into a log and you can go and replay them that's different from the intent from a message which you sent to someone with a job of do something particular so we're we're just reporting things out so there's some principles for events or principles for pub/sub and that's also what we have in mind when we design these things we have a notion of a type we have the notion of a source where does things come from and notions of a subject what is this event about and all of those help you to filter out messages that you're interested in you

▶ 19:29 publish events into space and then you may have subscribers who are interested in particular events we're enabling subscribing to those and some of the principles our publishers don't currently know who subscribed you might publish an event but there's no subscriber for it and that's okay but there may also be a hundred subscribers to it and that's also okay and middleware typically helps you dealing with that and this event flow is generally unidirectional which means you sent messages you sent those events out and there is no response path because you don't expect anybody to do based on that event but they enable really interesting scenarios they enable

▶ 20:12 extension of a of an existing application with new functionality without you telling that new functionality to do something I'll give you a brief example that are always used for a venting which is something that works on for instance the blobstore that we have in in Azure the azure blobstore can raise an event that's called blob created and you can hook on to the blob created event with cloud events 1.0.1 today via event grid you can hook a handler so now you can go and have an application go and send raw pictures from [Music] let's say even RAW format pictures from a camera or JPEGs into that blob account

▶ 20:57 you can now turn that into a photo store because you can hook up this block to the block created event the application goes to looks at the image and first if it's a raw picture converts it with default settings into a JPEG now you have a JPEG now you can have another event which looks at the subject sees that there's a JPEG in there and now goes and creates five thumbnails for different sizes and stores them into a different account you can have another hook that's also reacting on that event which goes and takes that image and runs it through some AI classification mechanism which goes and creates tags and you store those tags into a index database that index database now may go go and raise an event as well you go and

▶ 21:40 hook that hook hook up to that and then you can go with that reactor that event and import that into your news newsroom image database which you can go select from current pictures of the sports game that's currently happening and you already have that pre tagged and you already have the source to the original picture and to all the thumbnails and you have done nothing other than reacting to events you haven't told anybody to do something but everybody is just reacting building functionality on top of it as an extension it's extension programming it's extension driven programming effectively you drive events drives the functionality you don't tell anybody to do it but applications react

▶ 22:21 on things that are happening next okay so so they're deceptively similar messages that tell you to do something I will assign you a job versus events that tell you something that happens messages carry events events carry facts there are consequences to that there are architectural consequences to that and there's also infrastructural consequences to this for instance messages typically live in queues or often live in queues because you want one party to pick up that one message

▶ 23:05 because you want that job to be what done once an event typically runs on the pop sub system because you want to tell something about that happened and then they can go and decide whether they want to do something and very often you will find that messages and events are happening insisted in a combined fashion and you will also find that you will use multiple infrastructures for those if you have streams of events you might use Kafka because you want to go and pull those events to a place to go and do and analyze them and because there are facts and things that are happening there won't be changing anymore and the context won't be changing anymore you can also scroll back and forwards in a history then you have event routers like we have one in grid that goes and take

▶ 23:50 an event and push it out because there's no context required between those events and sent them to a handler like lambda or are a national function and then you have proper message queues which give you the competing consumer model that you need for assigning work so those things are different and what we did in cloud events we picked the event piece and how can we go and deal with this with these unidirectional messages and how we can can we make those look the same we explicitly scoped out the messaging aspects and we might in a different work streams they'll get to that problem of where they want you and unify that but cloud event is not that

▶ 24:33 next okay so there's stuff that is that you will not not find specifically because of that in cloud events there's no to attribute we don't tell you where that cloud where that message should be sent to because we assume pops up we tell you when that message comes from there's also no reply to because we're directing this into space and we don't even know there anybody would reply we have not because we have no expectations of replied this is not a job we're expecting any replies to we're basically just telling any subscriber who likes that something happened so there's no reply - which means that cloud events is

▶ 25:17 also not there to go and model your RPC exchanges at least not in the standard way the envelope is suitable for it no doubt but that's not a design point of cloud events because and and and when I say envelope there's a slippery slope here because there's prior art that we also didn't and I'm not going to mention it but we we didn't want to go on the slippery slope to write the universal Jason everybody's envelope format but wanted to build something that's specific for events and then topic you also the place where you go and pull and put that event it's also not in the

▶ 26:00 message or in the event so you don't have a notion of the infrastructure piece that's accepting that event and the reason why we don't have that is because we assume that events will be forwarded that events will travel over a long route of forwarders that they might go come from a might come over HTTP my then land in a queue might then be stored in a database but then come out of the database to be sent to someone else and you know the whoever sent that publish that event initially to what artifact doesn't matter because we assume that those events those captures of things that happened will travel and will probably be archived for a very

▶ 26:43 long time next and and then lastly and then we already get to the Q&A section there's a few things that we have avoided so far and they may raise some some questions and that is around security specifically because that's something that was raised as a thing and we said let's not do that there have been many many different many many efforts for Standardization where before things were done people were trying to tackle specifically these problems signatures

▶ 27:26 into an encryption for a messaging centered and they ended up sinking the ship just because they of the inherent complexity so we I have insisted and and a lot of people have then ended up agreeing luckily to say let's get 1.0 done first and then let's go and think about how what security and how we go and apply security we know that eventually we have to go and get there but I think that's a bigger discussion and one that also overlaps with things that we have in other areas like in AMQP where we do you know full-on messaging what we need to have mechanisms that are similar so for instance when you start thinking about how to do signatures in

▶ 28:09 an event which you want you want to have ultimately we all want to have the ability to go and sign an event and make sure that the event we get over here it's the same that was published over there and you should be able to sign that the question is how do we get the signing key who signs the signing key how do you get the signing key if you have a puff subsystem you publish to a publishing point and now all the subscribers you can go and subscribe to it how does the subscriber get that signing key and when that signing key roles which it will how does that subscriber get that new signing key how can you tell when you put an event into

▶ 28:51 Kafka and you made a make it forever log and now we can scroll four months into the past how do you know the signing key for the event that it was for toasted four months ago if you put that into a database and you see an event that was posted six years in the past and there have been five hundred key roles how do you get that key to validate the signature of that thing that was there five 500 roles ago six years ago that's not trivial that actually requires quite a bit infrastructure quite a bit of protocol and you could think about all those things and you can think about approaches to this but this is not the

▶ 29:32 time to go and deal with it and and I believe that we need to have those kinds of key vaults and we need to have those kinds of protocol so go and deal with those but it's probably not only cloud events problem to go and solve then you know once you go once you think about you know key distribution it's also who gets access to those keys and why and what is the identity space for this so that gets you further down the line and ultimately you get to the point of identity federation first then you need to go figure out how to have authorization for or whatever the key vault is then you need to have a notion of how that key

▶ 30:14 vault is managing those things and then have a way to go and correlate that key vault to event streams which means that key vault really needs to be able to store very many keys for you know periods of timing you need to be able to go and say I have an event from five years ago and now I need to have the key for it so that's all doable but it's not something that we can go just solve right now so that's future work and to add encryption is exactly the same problem because you also need to have need to have key material it's just harder because encryption keys degrade faster in this world because we have

▶ 30:57 sins who are doing data grams and we have not the ability to go in and negotiate session keys because there's no other party we basically have to go in broadcast and we have to we have to encrypt data grams for broadcast which means the encryption keys degrade very very fast through use we have to roll them more often which means we now need to have a key store which lets you let us also again same same problem get at historical keys fast and then the question how does that work again something that's doable I think that's a job that we all need to go and figure out how to go and do but it's nothing for cloud events only to solve because

▶ 31:38 nqp also has that problem and that's and then you know without even looking into algorithms and algorithm rolling and and all of those other other concerns that have to do with configuration right now I sign with a sha-3 and sha-3 gets goes out of fashion because eventually it will and then you know how do I replace that how do I indicate this I need to have that in the message somehow some indicators of what algorithms have been using to go and create the signature same goes for encryption so that's all stuff that we will eventually get to and there are some other things on the agenda which are similar like we Doug and I just

▶ 32:22 discussed earlier schemas which is a similar problem like what is our schema store look like because there will be schemas and you will be able will want to look them up apache atlas is a is a project which might be helpful in that in that regard but again there we need to have a common API definition and we don't have that yet okay so that's that we have four minutes left for questions if you have any jwe

▶ 33:07 yes so the J yet so the question is the JWT to Jayson web token solve some of those concerns yes it does it so there's a hole Josie Jayson object signature and encryption set of specs that are RFC's now with where my colleague Mike Jones also contributed to and they hail from effectively equivalent standards from the xml world and yes so there are some components that are solved there but it's not an entire suite it doesn't address the store or the vault or any of those things so yeah it defines some

▶ 33:51 metadata elements that you can use for this but it's not a it's not a complete solution next question the question is whether project Orleans has an integration with this I have haven't I have not had any specific discussion about this with those guys but we're early in the sense in that we haven't locked on a version that anybody can really ship so we tried to get there very quickly and then we certainly at Microsoft has the intent to go in standardizing cloud events which means we currently have a native format for event grade all the other services are publishing through event grade and when I go and switch that format over to

▶ 34:34 cloud events because it now has natively everything we need cloud events is winning well cloud events is certainly something that we want to use and there's other providers who are also want to use it so I don't know what winning means but the goal is that we we all create a common basis to go and do events and if that becomes popular we're all happy you're welcome so the question is since signature and encryption all those things are not there do you think it's ready to take on

▶ 35:19 critical business data gtp our data it's etc that's the discussion we've had and so my answer to that and we've had that in the context of the discussion about the claim check pattern you should since you should always assume that the data that you put into a column and against broadcast so when you make an announcement out of a CRM app that a shipping address has changed you should never no matter whether the signs are encrypted you should never include that an address because you should go in and have a reference that's good enough to understand the context of what you're subscribing to and then you should have a URL and that URL should point to the place where you can go and

▶ 36:02 obtain the data because then you you walk up to that point to that place and you can go and obtain with your rights as the consumer you can go and obtain that data so think of cloud events as an alerting mechanism that something has changed but it doesn't necessarily tell you exactly everything about what has changed and if you have privacy related data the privacy the related data should stay where it is and you should just basically just point to it rather than then include it well thank you very much have a good show