 My name is Trevor McKay. I'm a principal software engineer at Red Hat, and this is config maps 102 So I've also I've been a little sick this week. So hopefully my voice will hold out But without further ado All right, so You're still here. It's a 430 on Friday a lot of you could be on your way home On a nice airplane, but I know why you're here, right? You were looking at that schedule and you said Wait a minute This guy's gonna talk about config maps. I gotta hear this. I'm staying right. Is that what happened? Yeah, I know it. That's why I'm here, right? Okay, so all kidding aside You've heard a lot of great tech this week. There's been presentations on CRDs on extending kube all kinds of You know Enterprise kubernetes success stories machine learning stuff. There's been a lot of really great stuff, but today we're going to take Sort of a new spin on a golden oldie We'll take a quick look at what config maps are I'm not going to assume that you necessarily know right this might be your first kube con You might have been around kubernetes for a while, but you've just never really run into a config map So no worries. We'll go over exactly what those things are We'll look at some restrictions related to typical usage Restrictions are fine every design system has some limitations We'll see what those are and then we'll look at another way to interact with these things And it's not necessarily the method. It's just a method that we've been using and I hope that it's useful to you And gives you another tool in your kubernetes toolbox and then at the end I will show you a sort of a mini Example of this stuff running and just sort of a contrived example And then I'll show you just a part of a real-world application that we're working on that is using some of these techniques So first of all what is a config map, right? It's simply a key value store that lives as a kubernetes object I get my Pointer turned on here. So here's our key, right? There's kube-con. It's awesome. That's the value. Here's kubernetes. It's awesome Excuse me brilliant That's its value. This is an export of config map object In kubernetes after I've already created one So it's just a key value store if you're familiar with python think dictionary for other languages think Associative arrays or string maps. It's just names and values. That's it and it happens to live in ecd All right, so how do these things make my life better? the the major benefit to a config map is that it lets you modify Application behavior without re spinning an image. Okay, so here we have a couple pods running Go back to pointer mode. So here's our application one and we have a Config map here config map one mounted in the pod as a volume. Okay, and likewise. Here's our second application And we have config map to mount it as a volume in the application pod so if you design your applications so that the malleable parts are Expressible as as key value pairs, right? You can export sort of the change ability outside of your application image And then you can use the same image over and over and over again And you don't have to re spin to change something like You know the number of back-end threads you have on a database or something you just make that part of your config Right, so this has been the power of config map since the beginning. This is why they were invented Alright, so that's how they make my life better How can I create one? There are a bunch of different ways you should read the documentation, but here are This is a pretty broad view here, which gives you a lot of flexibility In the top example We see we're just calling cube cuddle create We're making a config map called my config and we pass in some literal values, right? You can stack these Multiple flags multiple from literal flags. You give it an equals and then a key and a value So in this case, we're creating a key name verbose. The value is true And another key name debug the value is also true a second way to do this is to descend over a directory so in this case if you use from file you can stage a directory with a bunch of config files in them and What kubernetes will do is actually create a config map where every key is a file name and For each each key the value is the entire contents of the file Okay, so that can be quite powerful if you have you know yaml or json descriptors all kinds of stuff the only place you can get into trouble there is that at CD puts a limit of One megabyte on objects and so your config map can't be over a megabyte so if you're You know putting lots and lots of large files in there. Just be careful and of course If you're doing things a little bit more interactively You can just create an empty config map This way this will have no data section in it at all and then you can call Cubecuddle edit and actually add the data section by hand so With those few techniques you should be off to the races creating Config maps. All right. That was how to create one How do I consume one? Again, there were a bunch of different ways. You should check the documentation You can mount these things as environment variables. You can do all kinds of stuff, but One way that I find that's really flexible to consume these is as a volume and The way that you do this is in your pod spec. You simply declare a volume, right? And you give it a name and You tell it I'm waiting for that menu to go away. You tell it That the data source is a config map, I guess it's not going to cooperate with me and you give it in a name so The follow-on to this after you've specified the pod volume is that in your container you add a corresponding definition Which you can see up here you list volume mounts you give it a name again and Reference the name from the pod and then just give it a path So in a nutshell any pod spec you have if you want to mount a config map you add a chunk like this to your container or containers into the pod and What this will result in is when your pod spins up this in this container you will have At the path slash Etsy slash config Everything that was in the config map Will be expressed as a file right the file name for each of them will be a key and the contents will be a corresponding value Okay, and these things can be tiny. They can be a single value So you might have something where the file name is dog and the content is phyto and that's it That's all it that's in there or it may be you know an entire an entire yaml file for some application that you're running All right, so I love config maps. I think these things are great If you haven't used them you should give it a try It's a it's a small piece of tech in the kubernetes world, but it's it's It's really powerful, but there are some limitations right there are always limitations You can't Design for everything. So what are some of these? well the name must be known and What this means is that? The name of a config map that you're consuming in a pod has to be known at pod creation time Okay, you can't get away with an empty string kubernetes won't let you do that. It's an error So it has to be a definite name If you're using some kind of a templating engine that gives you a little bit more flexibility So that you can you know Generate your pod spec from a from a template and you can pass the name in that way, but the fact remains that you must have a Definite name for this thing When the pod gets created Update latency so if you have one of these things mounted into a config map as a volume and you change the config map It will be up to one pod sync interval Latency before the updates get written through and by default that's a minute now for some applications. This may not matter You know it might be fine that the config will change, you know up to a minute after you've actually written to the config map Some applications may care All right, they're not easily composable. So what I mean by this is there is no There is no built-in way to have a config map where a Field value in one config map actually refers to another config map, right? So you can kind of imagine you could take a large complicated configuration And you could break it up into a hierarchy, right where you have your top level config and then you break some You know piece of it out some aspect of your app and put it in a different config map and so on and so on and you Can nest these things theoretically As deep as you want to but there's there's no simple way to do this With config maps out of the box and lastly order matters There's a caveat on this what I'll get to I'll get to in a minute if you reference a config map in a Container spec the config map must exist before the container will be created I've done this to myself It's not always immediately apparent as a developer why your container is stalled and is not being created unless you start You know pouring into the events But if you reference a config map that's not there your container will just I forget which state it ends up in Container creating or pending or something, but it'll just sit there forever, right now the caveat on this is in one six There was this awesome PR Where the optional keyword was added so you can actually call out a config map as optional And in that case if you do that and it doesn't exist Your container will start up anyway And in the in the case of volume mounted config maps if you do add it later that data will get written through however, you still run up against The issue that you have to have a definite name ahead of time You know empty string isn't good enough and you have the latency issue But this was a big help. So if you're running on one six or greater That's good. There's probably some You know enterprise production type people who are still on one five or earlier because right you can't be changing every three months So What is an alternative method here's my path in the woods Like I said, I have another way. I'm not saying like it's the way it's just hopefully something that's useful to you And what is that? All right, so it's a more dynamic application model and in this case What we do is we just embed a tiny Kubernetes client inside our application Which talks to the API server for us and allows you to consume config maps dynamically now This isn't necessarily novel if you've existed in the Kubernetes core or you're already making CRDs or you're making API extensions You know, you're writing deployment pods that kind of stuff You're already used to having access to the API inside your pods, right? But there are a lot of us I think Developers who view our applications Really strictly as running on Kubernetes, right? We don't think of ourselves as part of the ecosystem And so I think for for Those of us on sort of that side of the game This is still a relatively novel approach and what I'm going to show you is that you can do this very easily And safely with just a tiny bit of code and it doesn't have to be a Big involved project Yes. Yeah, and I'm going to show you how all this works Right Okay, so that's the basic model We get some benefits from this what what are those? Well really quickly Going back through referring to some of the limitations did before order doesn't matter, right? If you're controlling this stuff yourself create things in whatever order you want Likewise, you have flexible naming, right? If you want to leave your config map with an empty string go right ahead as long as your application can handle it If you want to change the name midstream, that's fine, too As long as you have some way to communicate that to your app through an event or you know Some trigger that you come up with a url or something and you could change it as many times as you want Update handling you can handle latency any way you see fit If you have a polling interval that is appropriate for your app go ahead and pull if you want to do read on demand Every time you hit a particular piece of processing and you know that that's you know infrequent and it's not going to overdrive the API Server or anything do it that way You know if you receive some event through a Kafka queue or something that tells you hey your config just got updated Why don't you reread it? You could do that the point is? You have choice right you can you're not limited to updates with this variable Duration up to a pod sync interval anymore. You can know you can update definitely inbounded time All right, you also get a composability like I talked about before so Let's pretend we have a web app here We've broken out our theme and our scale as separate configs Maybe I'll grab the pointer here So just assume dot-dot-dot means really huge right which is why you would want to break these things out in the first place But we have a Western theme with some desert and cactus We have some scale, but let's say that we run this thing for a while and we decide okay I really need a large scale right so I go and I define a large scale And then I want to take my running app and scale it up so I can just do this right I can just go into my top-level config change this one entry In my top-level config map to large right and as long as my application Understands that this is my convention that I've set up. This is my design it can go and Read the large config on the fly in an update So you could do this, you know you could nest this to however many levels you like All right Another thing that's great about this is you can run this stuff in a pod or outside of a pod and the only thing that's different Is the authentication mechanism? You'll see that in a few minutes if you're running a client There's a little bit of off-code and if you pull that out and separate it from your your main routine You can reuse the same code. So here we are in a pod Here we are out of a pod running on a laptop You can use the same client snippet Okay, so Some people might say well Maybe this is too much right the Kubernetes API is big You know there might be a security risk here because I'm Exposing more of the API surface area in my pod I might be pulling down lots and lots of dependencies and and this has me worried I'm not sure that I really want to go here But what if you can do this right? What if you take the API and you take that huge stack of books and turn it into Just the little tiny piece that you need right in our case in this example all we need is a single call to get On a particular namespace For a config map right that's a tiny part of the Kubernetes API your client doesn't need to actually include anything other than that so your dependencies are limited and your surface area is limited and I think if you do that then the risks are Negligible and it's worth doing Okay, so let's actually look at this thing and go This is the actual routine. I'm going to show you in a demo later The only difference is that I put a loop around it for the demo. Okay? So what's in here? Well, there's our our client set This is the result of the off-call. We'll see that in a minute This is the part that you break out so that you could run in a pod or outside of a pod We pass in a name a directory in a namespace as Arguments right so those are pretty simple this right here is the core of this whole thing right it's a get on a config map in a particular namespace and what you get from this is a string map okay, it's just a Just a map of strings. I don't know how it's to explain it If you loop over this thing and go right you loop over it and you get key value pairs back And from this point on this is just normal processing right doesn't have anything to do with kubernetes It's what you would do anywhere so I get keys and values. I Create a new path based on the key. I open a file and I write my string boom. That's it, right? So with this little fragment of code you can read config maps all day long and write them to a directory Okay, so Authentication stuff in a pod it looks like this This reads your config from the pod environment. This gives you a client set and you return it Outside of a pod. It's very similar This is using the client go by the way You pass it that's path to your config Build your config based on that Return your client set and you're off to the races. Okay, so I'm going to show you this in action Let's see here Let's start fresh. All right, so I don't know if you can see that. Okay. Let me blow that up a little bit Does that look pretty good? Let's look at what's in our yaml file first. It's very simple. Okay, we have a pod here We're just going to run this pre-staged image that I created You'll notice if you're really observant that this is set up for local access So if you're doing this stuff on mini-cube or on on OpenShift origin Just know that this example is set up to pull something from the local Docker demon And then we pass the name of the config map we want to consume as an environment variable, right? That's it. So If we run this Okay, there is our pod running and now we want to log it oops loop There we go. So it's sitting there saying it hasn't found config map Bob. Well That's not surprising Because we haven't created it yet. So let's get another terminal over here Blow that up a little Okay, and so now I'll create my Bob. All right, so that's empty At the moment and what we should see here is Is it off the bottom on the screen? See I might have forgotten to do one thing. Oh, no, it's reading it. Okay So here you can see the point It's reading Bob. It's not doing anything very interesting because Bob doesn't have any values in it yet So if we go back over here We can edit it and this is how you add a data section and We will give Bob a car He's got a Ford he likes antiques So we'll give him a Model T and Now we should see this update here. There we go. So we're we're pulling every 10 seconds We saw that we changed and now we're rewriting our config directory. You see that we've got a File for make and a file a file for model so we can do this all day. We can change the car He owns we can delete Bob And it will keep updating but I think you get the idea there Let's see so if we go back to Our presentation there's something else. I want to show you we got about 12 minutes left. I Want to show you a real-world example that was Intentionally simple so you could see how it worked. Hopefully that made sense to you I work on a project called rad analytics IO and as the text here says it's a it's About empowering insightful data-driven application development on OpenShift So the piece I want to show you of this today is a tool we have for creating spark clusters Very easily and it uses some of these techniques to let you configure the spark cluster So this is one little part of Of what we're doing But I have links up to our site At the end if you are interested so what is a containerized spark application look like? With using a standalone spark scheduler you end up with a driver, which is where the spark application is running You have a master which is part of the spark cluster and you have a couple workers, right? How do you create one using? Rad analytics IO tools. Well, we have a tool called Oshinko It's very simple. It's a little CLI and you say create my cluster and in this case The default number of workers is one, but we're going to tell it here to create two workers So that's pretty simple But you can also do it this way. So let's say that you You wrote some cluster configurations that you want to save for later So we let you do that and we let you reference them with a flag called stored config Right in this case We're going to use debug and so you just pass this flag to create and Oshinko will go and read the config map And figure out what your cluster should look like This is sort of an idealized view of how it uses them here's our top-level config we include things like how many workers you want We let you turn on optional Services like this oops that turns on spark metrics We won't actually see that today And then here this is where we use the composability Spark takes its configuration from a bunch of Bunch of files just like if you've ever done anything with HDFS, right? Hadoop is configured in the same way. It has a collection of of files in the In the config directory So we put an entry for the spark master config in an entry for the spark worker config But Oshinko expects that these are the names of other config maps, right? So we have another one called debug master. It's got a log4j properties file in there that controls logging You can put metrics properties in there. You can put spark defaults.conf in there. Those should actually be lower cased There's a typo in the slide Anyway Just know that wouldn't want that to throw you off And we've got the same thing for the worker config down here, right? So we break these things out So what does this look like in an actual run? So here's Oshinko running on our laptop? Outside of Kubernetes, we're gonna create a cluster from our command line and we're gonna hook a spark application up to it later So it uses an embedded client like the one we just saw to go and read the debug config It reads that and finds out. Oh, okay This thing has a spark master config in it that points to another config map Name debug M. So I am going to go create the master pod And in this case I since I'm actually generating the pod spec I go ahead and mount it the good old-fashioned way, right? I generate the pod spec and just add the config map as a volume So it's taking a hybrid approach here, right? It's using a dynamic approach where it needs to And it's using the static approach where There's there's really no downside in that particular case and it does the same thing for the workers All right, so I will show you quickly what this actually looks like in practice Let's go back over here We'll stop monitoring this And I should have Oshinko on my path. So we will create a cluster called First cluster now if we go back and look at the OpenShift console This stuff runs on OpenShift from Red Hat, which if you've seen other presentations is really kubernetes package for the enterprise It's kubed with extensions but It's the heart of it is kubernetes. So there's really nothing different here. So Here we created we spun up a master and we spun up a worker. This is our tiny spark cluster I'm going to show the show you the effect of the config. So something really easily observable is I don't know if you can see that is The logging level you can see here that you've got like regular old info level logging in the master pod All right, so let's go back To our command line here. What if we want to create another one, right? Actually, let me get that up on screen a little bit more so you can see it. Well, that's not going to help Is it maybe I can shrink the window? All right That's a little higher up. So if I do Oshinko create my second cluster And I give it a stored config Called cluster config It went and created a second cluster now If we do this Let's just do an OC export here OC is the open-shift client and it has a cool extra command called export Which gives you a nice view of objects Without all the extra values in them you can see that our config map in this case called for three workers and it had an Spark master config Called out in a worker config, right? So if we go back here, we should see a different kind of cluster and we do Here's our our master We got three workers this time because we asked for them and if we go in and we start looking at the logging We can see that we've got debug log entries in here now instead of info So the config took right we got something different if you want to see what the result is of putting config files in a Config config map. Oh, what did I call that? Okay, hold on a second. I must have spelled it wrong Master config There we go. So this is what a config map with files embedded in them look like you can see here's the data section Here's the key name right which ends up being the file name and then you've got kind of the the yaml designation for multi-line stuff You have the license file and everything in here, right? So that's what we passed in But it's nice that you can hide these things away using that little composability trick Okay, so let me go back to where's my presentation there it is. All right, so that was the end of my demos Take a ways so wire config mechs good in general you can change behavior without a re-spin They're powerful with a few limitations which may not affect you, but if they do Reading it dynamically is very easy. It gives you a ton of flexibility And it's safe is if you use the small scoped client that doesn't have superpowers You're not really endangering anything. I mean the only thing our extra client can do in that pod is read a config map And it's scoped to the project, right? So I mean you can't even read anybody else's stuff not very dangerous Here are links to all of our stuff. I'll leave this up here if you want to take a picture of it Rad analytics. I owe is our community landing site if you want to see more about the work that we're doing We have images up on Docker hub our github is listed here and The little looping example that I showed you first. I actually threw up the The pod spec and the code for that on my github account in case you want to take a look at it and play with it It's based on the client go examples So it's pretty easy to consume Yep, go away go away and there is my My email address is there on the bottom, but it's occluded. There we go All right, so the easiest way to get in touch with me if you have questions or comments on this stuff Or you want to know more about what we're doing is just send me an email and with that we've got a Three minutes for questions. I mean the truth is the conference is over. So we have as long for questions actually as you want So Yeah, I'll just repeat the question in case other folks can't hear them. That's a good question. I Mean we are using we are using some config maps in our CICD already, but we're kind of Creating them on the fly as part of the test script ahead of time So, I mean we haven't really tried to change something dynamically in a test unless that's the test itself, right? So typically in our stuff when we're staging some tests in our pipeline Maybe one of the beginning steps is create all the config maps that you're going to use So I don't really have a good answer for you for that use case Thanks. Any other questions He's got an answer Yeah, right. Yeah Yeah, yeah, so I mean it sounds like you can just have your test apparatus itself Modify the configs kind of outside the test and just flow into the next one Any other questions over here? Yeah The that's one of the things that makes composability kind of cool But the limit on the size of a config map is one megabyte across your whole data section So that's a lot. That's a lot of config, but if you ran up against a megabyte you could simply chunk it up You know, it's like packetizing, right? You say, okay fine. I need way more than a megabyte I need five megabytes. So I'm going to break my config up into smaller chunks and chain them together by name So I haven't personally run into the one megabyte limit yet, but I'm sure it's possible So are you asking can multiple applications use the same config map at the same time? Yeah, yeah, that's fine A config map can be mounted in multiple places The downside on that is if you change the config map for one you change it for everybody, right? so in that case you have to decide do do I actually want to be able to change the configuration for Applications individually or is it fine to change them as a group and then you just design accordingly? Okay. Yeah, I mean you could you could give if you have multiple instances of the same application You can set each one to read a different name you could You know you could use the same config map but Set up some different event stream to queue each individual application like okay Read this once and cash it unless I tell you different and then like okay You pay attention to this signal you pay attention to this one you pay attention to this one and then you know key them to To read individually you could do all kinds of stuff once you have this code in your application You can do anything you want right if you can invent it you can do it So it's kind of neat. Is it atomic? so the question is am I aware of any problems with updating config maps and and I'm guessing you know like race condition kind of stuff. I Would guess not I Don't I don't know for sure you're right. I mean by not by reading this yourself You're kind of stepping out of the kubernetes pod sync behavior, but at the same time kubernetes has to know in general that People can do gets on config maps right and people can edit config maps. So I'm assuming there's There's there's guardrails around you know Atomic operations at a lower level. I think it's probably fine, but I can't say for sure Yeah, yeah, so I actually didn't show you that I Didn't know if I had time to fit it in the question is if you're doing this an open shift Do you have to set up a service count and the the answer is yes? In fact, there's a service account in that project, which I didn't show you which I created before you came in Same thing in kubernetes When did our back get included like? 1 6 I think if you're running kubernetes with our back turned on it's the same stuff as an open shift So yes, you need so so out of the box, right the the the role that you need is edit Okay, and it gives you read and write, but that's the one that you get out of the box. However It's scoped to the namespace so you don't get edit for everybody and if you want to restrict it lock it down further than that you can create a Custom role which only has access to get right not even access to list or you know other read Operations and indefinitely not access to write you can define That on your own so that you it's locked down to read But you're correct if you run this without the Without the service account so what I did is I took the default service account for the namespace and just added edit to it So it's system service account. My project default If you run without doing that and you run a version of this with an error handling still in the code it will it will tell you Such and such service account doesn't have privileges to read config maps in this project So it's it's easy to see if you actually have that turned on like oh, yeah, I have to go add this So good question Yeah, no proxy So in that off code from okay, so so let's let's say you're in the open-shift land In open-shift you have to log in with credentials, you know ahead of time and then it modifies your kube config file So it's pointing to the local master. It knows what the API is and it knows what your credentials are And in mini kube It's the same thing except you don't have to do the initial login It's just when you when you bring up mini kube it modifies your local kube config so it knows where the master is So when you run that little those little authentication code snippets that I put on the slide that reads your kube can your kube config file and builds the client set for you and That thing knows where the URL for the kubernetes master is so there's no proxy in the way You're just talking directly to the kubernetes master on Whatever port it is what is it like port 5,000 or something you're talking directly to the API server at that point so Anyone else since we've got the rest of the afternoon No, well, thanks for coming It was great to have this room like mostly full and you guys are still here. I mean, what is it? It's like Five oh seven and you're still here, you know, that's amazing. So thank you very much