A Practical Guide to Prometheus for App Developers [B] - Ilya Dmitrichenko, Weaveworks¶

Playlists: KubeCon + CloudNativeCon North America 2017 (Austin)

Practical guide for app developers on using Prometheus, with PromQL examples and remote read/write.

Topics: prometheus, promql, remote write, remote read, monitoring

Transcript¶

▶ 0:00 so my name is Ilya I work in London company called vivre works she started by RabbitMQ co-founders perhaps perhaps some of you are familiar with rabbitmq and we've been living around since 2014 and that's around the time I started working on cloud native tools had feed worked right the first release day a network for docker that was really the first network that actually worked out of the box for like you know containers running and multiple hosts without having to run containers in the host name space etc so that's that's one of our flagship products that's called the net it's 2.0

▶ 0:44 now 2.1 actually and more recently been contributing to to Prometheus project and Copernicus Aero tools such as cube ATM have a few projects of my own and the contribution that we made towards the Prometheus project was was was done by Julius waltz so maybe here maybe not today and this is the the remote write feature and also remote read that allows us to to store data prometheus data in in the cloud service right we run a cloud service that provides a remote read and remote bright endpoints for your preview

▶ 1:28 server it's pretty easy to unable in existing Prometheus file but you could just use our out-of-the-box configuration so this is this is just a bit of an intro about myself and the company I work for I'll show you a little bit of the product just just by just as means of you know visualizing the data I have in Prometheus and I'm doing from ql examples and and stuff yes for those who just came in this is going to be a very introductory talk to Prometheus I hope you'll find this useful and if you expected something more advanced well feel free to stay it's an app developer Orion to talk as

▶ 2:16 the title suggests and well I'll just go ahead right so I'm an app developer I build apps right that's right yeah I do micro services as everyone right and as I do those things I run them on kubernetes I let Kaveri this random for me so I don't have to worry about silly things anymore I let kubernetes run my market services for me and that's that's that's what most of you do right how many of you don't do kubernetes brilliant great we had to keep going

▶ 3:00 after all so I let Cabrini this run my micro services for me and I'll have this slide blank intentionally because that this is what this is what its gonna look like if you let kibrit just run your micro services but don't put any observability yeah you can guess so observability is pretty essential and as a developer in the age of micro services

▶ 3:43 you should embrace observability and and do your best job towards that and instrument your rod with Prometheus metrics we're no longer have to try and you know foreign the tool that will like look at the packet of system calls and tell you what the app is doing right why should you do that you can actually instrument the app you can actually tell your monitoring system about the business metrics above any protocol level metrics any app specific metrics anything like that even my dog likes it Wow yeah so dog

▶ 4:31 it skeptical cart he's thinking oh why should I just remember you have to do that well perhaps a good question to us great why well I think this is this is mostly to do with marker services being a culture and not to be mistaken with architecture right culture not an architecture you build your app your rodded team takes responsibility to run the app that they ship ops team run infrastructure developers run their apps so they can

▶ 5:15 actually put the work in to make it more observable they can put those Prometheus metrics in just just to make it easier for themselves to run the app to have decent monitoring in place each team has taken on cold responsibility I better do that right because because otherwise what they're gonna second-guess what's going on is you know it's them we're running the up now I think that's kind of one of the the big culture changes with brim eaters micro services so it's a big culture change we're with micro services and prometheus is a unobservable team tool

▶ 5:58 that is built for micro services in a day and age of the container orchestration it has native support for kubernetes and other orchestrators they tall so has support for legacy type systems where you can just write static config file or anything like that and you can also instrument legacy systems in their entirety through an exporter right say that like a set of databases over their own mainframe system you can write an exporter for that mainframe system for that into Prometheus but it works pretty well and

▶ 6:44 doesn't require any fancy exporters if you doing Mike resources and you use to bring this and service discovery and such things and the important aspect of Prometheus is that it's Apple based system so you don't have to configure your app until it like hey you're monitoring thing is over there and then oh I got to change that because there is a new monitoring thing or what I'm gonna do oh my client librarian sports one at a time so previous doesn't have that problem at all and you can run multiple versions do all sorts of things you can have like separate teams running their own instances are from Ethier's and like an ops team running federated or whatnot

▶ 7:29 instance of Prometheus you can have multiples of those and it has pretty efficient memory usage and and disk storage and remote storage which which were you going to showcase today and and all sorts of things like that that allow you that leave you so much flexibility to how you can run Prometheus so these are some some of the key aspects why you might want to care about prometheus and one other thing I forgot to mention it's one of these essential features of Prometheus says the the analytics facility through through the from klq

▶ 8:11 real language which allows you to continuously analyze how well your apps are doing and thereby continuously improve [Music] so how do I instrument my up for Prometheus there's actually very simple you just need to implement this metrics endpoints which uses content type text plane so nothing really complicated oops metrics endpoint plain text easy right

▶ 8:58 can do that in C perhaps and you need to write a string value for every like key value pair for a line name of a metric and floating point value and here's one example metric new users count value 250 you could add labels in curly braces like that and here's an example of this new users count metric broken down to more specific values where we use locale

▶ 9:47 as as a label so here's an example of sort of different metrics that that a metric standpoint of my app may output what's its uptime seconds there's a that's that's a platform specific metric HTTP request duration seconds count that's a that's a middleware type metric your HTTP framework web app framework whatever you call that thing the new users count that's the sort of business metric so you can put all of those in

▶ 10:31 this is a very simplified version you'd probably have labels a lot of place and you know like we will see an example later on and here's a more specific case of HTTP middle via metric where we have get type request and put type requests and error codes associated with those and yeah this particular app has only seen one of each of those you can find out more about few more advanced metric types such as histograms summaries and such things on the Prometheus website in the documentation these things are called metric types and

▶ 11:15 we're kind of like there are there are quite a few more advanced topics there so haja used the funny fuse to improve my app as I mentioned there's this prom cue language which allows you to Kure the the metric data you stored over time and you could do some pretty advanced theories I'm still learning myself so bear with me here so here is a very simple one you just say a metric name and you get all the all the things there are for process uptime seconds easy okay well that maybe not quite useful you actually want to look at a specific job and

▶ 11:59 here Pierre would be looking at my app and this will give me the EPA uptime of all instances of my app for the HTTP stuff you'd look at the duration seconds count and you can do something more specific where you'd look at my apt method get' puff your eye flash right and you can sum those up sweetu sort of aggregate over all of the instances of that my app thing and you could do other things such as rates and we'll look at some of this momentarily in the tutorial part and

▶ 12:44 there's another nice metric to be aware of as the app metric this is the metric that for me to assess internally when it finds a given job so it looks in Copernicus and it finds a service and it goes it looks at that so at all the pods behind that service and if it manages to scrape from from all of those it stores one in the app metric if it fails to scrape for from any of those perhaps they're not instrumented or something like that it will store a zero so you could look at whether we're the home of you so that essentially indicates the health of your

▶ 13:28 up if the app is instrumented so if you if you look at my app like this you tell whether whether all of instances of your app are healthy and able to report prometheus metrics at the very least and you can do a sum over those this will give you the number of instances actually so we'll look at some of this in more detailed tutorial now how we doing this time plaintiff time brilliant ok so I have a tutorial where i'm going to use a nodejs up

▶ 14:10 in the run it on key branches on G key I'm going to use vivre Cloud for Prometheus remote storage and to use replied notebook for curing the data that I have there and I'm going to use tool called draft for deploying the code that as I make changes to it so here's a okay so I'm gonna check out a branch to prepare in a couple of branches here not very good at live coding oh yes thank you v1 start yes you can see this

▶ 14:59 mirror display okay today okay cool I probably should zoom in [Music] we that's easier a start okay so say this would be you will see in the readme if you go to that Reaper I'll show the link once again afterwards if anybody didn't capture that so I've done all these steps earlier so we are prepared I installed draft I installed the beef cloud add-ons if I go

▶ 15:42 back I go to to beef cloud now I can see that for example if I if I go to explore view I can see that here's my cluster it's it's got a bunch of things running including mine OGS them over here and the leaf cortex agent is a Prometheus server that is scraping for from here so I mean I could show you more of this if you want so it's the end if you have time left but we can see that that beef cloud agents are connected to this cluster so we're good to go and let's check out the first version of the app well let me tell you about this up first right so we're launching a pretty awesome

▶ 16:27 API I'm sure how can use readers will be all over it as soon as we launch it it's called string Lee it's pretty amazing it does one thing and does it well to reverse the strings very useful isn't it so okay let's look at the code first so so this this is this is our plain text from front page and here's our million dollar methods and yeah it's pretty simple lets you use a

▶ 17:10 standard library keeping it super simple right so if I run draft up I should see this running on my converters cluster now it takes the later while I can show you a little brief cloud explore perhaps in the meantime so um what can I do here oh this is a previous version of the thing running right now so that's kind of what's so interesting I guess I'm gonna narrow down into a default namespace there's only

▶ 17:52 one thing that I expect to leave there to see it is expected to be there right now then og is up okay well mmm conference Wi-Fi that's it's always so delightful okay so anyway the app is ready now to double check with it have external IP here ooh conference Wi-Fi didn't free the demo gods no am I still connected

▶ 18:41 hmm dr. ping like oh look a birdie this Prometheus IO today okay um interesting mm-hmm this is great I don't even have to download anything in this demo surely this should work right yeah yep let's do that

▶ 19:25 let's do that all right now we shall pray the demigods I drop the mic yeah yeah yeah oh yeah maybe just a few pink pockets it's working okay it's Prometheus I don't work that's good okay great so I'm we were gonna say um you know q-quit we'll get serviced no cheers demo get its external IP yeah that's been assigned that's good we can we can save the external IP as a

▶ 20:08 variable so we can use that later so yeah let's let's check out string Glee app right can everybody see the console yeah it's pretty good okay I couldn't oh oh no that's that's pretty big but maybe one bump okay that'd be okay I guess so yeah string Glee app is like this tells me what I can do is this and yes you traverse a string that does one thing it does it well that's great we'll have the UNIX philosophy right so yeah I've done that already so I'm gonna run Apache bench as you do really

▶ 20:52 good a like you find it on hacking you so you're gonna have fun whether these guys can actually stand this up still stands after after your like Apache bench for a little while run it again there it's still running well but like back back back back back at the start up at the the coffee shop we thinking well we're gonna have to you know make sure that we can withstand the the hockey stick growth and we're gonna we're gonna go and do some more benchmarking and we we figured as I like to keep things

▶ 21:34 very simple and minimal we figured we're gonna get a right little load test think well we're gonna use Apache bench and we're gonna we're gonna add this hits endpoint which is nothing really clever at all so we essentially just gonna have a little counter here for anybody who accesses the API and we're gonna render that /kids right which is gonna render that number so if I go run draft up again it's gonna remain in foreground I need

▶ 22:18 to switch to another shelf and if I if I do cube coral you have pods I should see is it the new version okay draft is still building and making their release okay great so now no we should see that have a new pod that is coming up brilliant okay so now if we if we just curl slash hits we can see that nobody tried to use the app yeah see if I if I do oh yeah I have a little script here

▶ 23:07 actually that that you know reads that slash hits and stores out in the CSV file that I could potentially plot later that's this that's pretty awesome isn't it very very simple load testing framework as I'd call it and yeah framework without the painting it's just a frame it's yeah we're gonna we're gonna run that but first we're gonna run the Apache avenge [Music] but I have to actually modify this a bit

▶ 23:51 I need to actually pass some payload to it right right reverse question mark string equals hello cube Khan and well let me just double check with it this is this even board so that I forgot yeah okay Brennan mmm that's an interesting word yeah okay and I'm just gonna double check whether he'd send point is working oh yeah cool right I saw two hits on the

▶ 24:31 API so we're gonna we gonna run a be 300 C hundred and we we run that and we opened another shell collect samples here and in the very first one I'm gonna run Apache bench bunch times and yeah well yeah I think this this is a pretty cool thing I've got this some pulse file now I've got like all these numbers here and

▶ 25:13 as I'd be looking at it I'll think oh well hold on a moment and is this looking right yeah it does look okay okay how about I scale up my a friend I go and scale this up and you could all scale deployment what's called no G as demo replicas - okay couple replicas so keep Colonel so run the Apache bench again and I'm gonna go get my results

▶ 25:57 some polls so now I'm looking here and okay that's oh this is strange why is it why is it zero here yeah so like I have no idea right the my my my CSV father's and doesn't let me do much it doesn't scale is it okay so I go on Google and I go to a conference and find out like or meet up a meet up you know I'm I got to launch it tomorrow I get to local meetup and I hear about good prometheus it's a thing so I go I get home and I I figure well I mean I'm a

▶ 26:41 minimalist that person I'm gonna I'm gonna try and read the docs about prometheus and I I write this this version was really basic Prometheus metrics and and what do I do here well actually this this is all I kinda need to do Prometheus metrics right so I changed from just hits to be request total and yeah that's it cool okay let's try that so I still use I still have draft running here in another shell so it should deploy this new version that's

▶ 27:23 cool and keep could all get pods yeah so I got new new pods running so if I if I call on I'm just gonna copy this from here obviously take any of this save the AP address and curl metrics code all right yeah that works um so now I could run Apache bench a few more times I think I still have my awesome

▶ 28:06 collect sample scripts writing I don't need that then you were rubbish doesn't scale um yeah so draft is good and my shell here cool so now I go back to slash metrics and I can see this and that's great but I also have configured bringing fuse to to store data in beef clouds so I should be able to use a pretty UI to to see what's going on so I can see like some of the systems metric and that's great but perhaps I don't care so much about CPU usage right now

▶ 28:50 it's a very simple up I shouldn't shouldn't have to care about that oh right okay so [Music] yeah so if I look at it as it oh that's that's that's not just my app I got a look at you know GS demo and so these are two instances of my app I can see that each of them has seen quite a few requests already I'd rather look at it as a graph look here's here's the hockey stick growth right brilliant and yeah well this is kind of cool but you know I also want to make sure that all

▶ 29:34 the users get pretty decent user experience so whenever they want to reverse the stray ad string they when they will get you know fairly quick response well well we'll reverse their strings and we'll do that very fast that's you know that's what what the guy who wrote the million dollar method promised so so yeah okay well I thought about it for a while and I thought well actually you know there are a few reasons I could actually improve this up a little bit more and and I figured I could use a Prometheus client that integrates where it was one

▶ 30:18 of no GS web framework like app frameworks for REST API s and my favorite one is yet the mineral list one the Resta Phi and I'm using this this Prometheus client here and and instead of doing all of that stuff by by hand like I did before I kind of have nice framework situation going on here right so I got like my routes defined pretty clearly here's the API route for Emperor's I already have a to do line here but that's okay I keep my million-dollar algorithm going on here where is that um it's here still same I think I never understood what that line does but they said it's it's good so

▶ 31:01 yeah and now I don't have to do the metrics manually anymore and if draft is still running in conference Wi-Fi that looks good I should be able to see the the metrics that Prometheus client library has given me and I don't have this IP address I need to run this thing and yeah so I should be able to to see all the the metrics that whoa a lot cool alright so I'm not sure what I'm gonna do with all of these like I don't really care too much right now I

▶ 31:45 kind of care about the user experience and we just get requests right so I'm gonna look at some of these things and I'm going to look for oh yeah that looks cool I can I can actually see what ninety-five percent of people are saying right so I can go and check how well most of my users are served so if I if I look at this notebook that I saved earlier we load that and I can see here that oh hold on I need to deploy a load test so I'll stop abusing conference Wi-Fi with Apache bench and I'm gonna like deploy deploy a little test in the cluster keep

▶ 32:30 could all create F have a load test deployment here she's also instrumented with Prometheus and shortly see some some graph going on over here yep so my load test deployment is up it's also using Apache bench but it's bronze lit in the cluster I don't care and yeah cool well that's let's look at the big graph I'll keep this as a table over here it's kind of useful and then look at it as a graph over here so I'm gonna do an actual sum

▶ 33:12 for like yeah I'm looking at all the get requests and actually I want to look at path reverse so I'm gonna add an extra label here path reverse and yeah right so is that does that work no these monitors doing the right thing so well it says deployment is it heating

▶ 34:01 the reverse URL yeah so it should it should be that okay so that's default knowledge is never stable hmm somewhat odd I'm just gonna remove all of these labels and look at the Oh um this is a little unexpected okay I have to look at other metrics so one of the things we have in beef clod is that you can actually use all the completion as we don't know

▶ 34:46 what metrics are we gonna look at so the other metrics here that's not the one okay well I'm not too sure so maybe I have a bug in this code so let me have a look at the metrics endpoint once again and a good thing it's like you know it doesn't require me to to know any particular format it's kind of human readable and oh it's not tracking any of the requests and pass reverse that's all is anything heating that just be our

▶ 35:30 IP reverse string okay good and they follow kind of metrics again am i seeing the reverse ones or not okay Odia have to debug that mmm you know that's not true you know I'm gonna just wanna party bench once again from here because I'm getting a

▶ 36:15 little confused over there so I got another passive edge yeah I'm just gonna run this again yeah that should definitely show up because maybe maybe they have that low test is broken hmm okay demagogues not loving me today I think we are close to to finish so

▶ 36:58 I'll have to go and fix that take a look at the repo sorry about that yeah well that happens it's a demo as we know so a few a few more things so for those who want to take a look at the the demo code take a photo of the URL at the bottom I'm our develop on github and Twitter and you could try the eve cloud that cloud that leave that works and feel free to email me and join our slack we also have a booth

▶ 37:43 come by and ask any questions and do we have time for questions now we have a time we have little time for for a few questions but if you guys want to go and join happy hour or whatever is going on feel free to do so anybody has questions I have a mic over here or just come over and and talk to me a few few hundred thanks so much [Applause]