 Okay, let's get started. Thank you so much for coming to the JuJuCharm Workshop. We've been running these at every OpenStack Summit for at least the last two years, so I'm really happy to see such a great turnout. So the way this is gonna run, we've got pizza and beer, which is great. Feel free to have some. And we do know that there's also other sessions you might be interested in, so I really want this to be a discussion about your problems and things like that, if you need to nip out, that's cool. If you want to go check out something and then come back, that's cool too. We wanna kind of have a nice little groove here, and then we'll talk about all the problems we're trying to solve. So everything that you're gonna see here today, so how we're gonna do it, I'm gonna give you a quick intro. How many of you have no idea what JuJu is about, right? You're like, what's that funny name thing I keep hearing? Oh, not bad. How many of you are using JuJu? At all. And how many of you are just interested in kind of seeing what George can do for you today, see if I can help you solve a problem? Awesome, great. So I'm gonna give you the quick intro, it's like a little spiel in case you don't know what JuJu is and stuff, and kind of do, it's like a 10 minute intro or so. And then I'm gonna kick it over to Mark Mims, and he is going to, JuJu is best learned when we just show it to you. So we will go ahead and kick it over to live demos. We are doing everything you will see here today in the command line is live and running on HP Cloud. HP Cloud's an open stack cloud. So if you go home today, or when you get back, you can do everything that you see here today on a production cloud when you get home. So I think Drew from HP is here. If you wanna go talk to them, stop by their booth, it's a great combination. So everything you see here will be live. So stuff will break, we'll have a good time, you guys can laugh at us and stuff, and it'll be awesome. So welcome to the JuJu Talk, that's that slide. Okay, one down. All right, so who are you? Why should you listen to me? My name is George Castro. I've been working on Ubuntu since the very beginning. I've been with Canonical for six years. Prior to that I was a system administrator at Oakland University in Michigan, where I was a Linux admin. And back then, pre-cloud and things like that. And I remember some of the problems that I used to run into. And I'm joined here by Dr. Mark Mims. He's our lead ecosystem developer guy. So he's gonna show you these charms and things like that. Yell out whatever service you wanna see, we'll deploy it, we'll open these charms. I'll get to what a charm is in a second. And we'll actually show you how it works. So if you feel like writing your own, we can do that. And we can go as deep in technical detail as you want when we get to that part of the talk. So what JuJu is, so if you think back when I was a system administrator, the subhuman thing over there on the left that was like me, right? And back then we had like bare metal, right? And someone would say, we need to do this, right? And then I would put in a PO and I would get a server and it was cool, here's my new ProLiant. You would open it up, get that smell, it's awesome. And then you would rack it and get your cables all right and stuff and then rip it all out and stuff. And life was good, I didn't know any better. I would just go rack things myself and do all this kind of stuff. And then at some point, hardware got to the point where somebody invented virtualization. It's like, hey, you don't really have to go around buying all new servers all the time. You can consolidate them all and have as much usage out of a server instead of having them idling around and wasting power. So virtualization came around, everyone was happy. I was still kinda doing the same things but now I was doing them with VMs and the first time I saw a live VM migration it was like the coolest thing ever, right? And then at some point cloud computing came along and it kind of allowed and actually looking back now, you know, we're kinda spoiled, it's like, wait a minute, so you're saying a developer needs resources, he just has an API call, magic happens, and then he can get work done, that's crazy. Like, and then people started to like that and they started to like that a lot, so much so that now you have a ton of stuff running in the cloud and how are you gonna manage that? So configuration management came along, right? And you have your Puppeter Chef script that this instance is gonna be a web server, do these things. We believe that things are still moving forward and the next step is gonna be service orchestration. So we sit at a level above configuration management tools and we are all about doing service orchestration and not really caring too much about the individual machines because other tools do that. So I'll get into more detail of what this is. So this is where we feel Juju sits on the stack, right? And just like cloud computing depends on virtualization, depends on bare metal, right? It's, you know, you're not throwing anything away and starting from scratch, we're building on the lessons that we've learned. Then eventually, we'll all get so smart, there'll be robots and we all die, you've seen that movie. So something I want to talk to you today, especially at the OpenSack conference, so this is gonna seem kind of like heresy, right? Is like you go to these talks like Juju Workshop, right? And I was like, this is what you need, you need Maz and you need Juju and you should be running on Ubuntu and then you go to the next booth, some other dude is like, no dude, you need this, this, this and this. And then you come out of here and you're like, my brain is full of acronyms and even OpenSack itself, man, like can you name all the OpenSack services off the top of your head? Of course not. And like we get like really into this stuff where it's like, well I need this and you gotta need that guy, that guy can help me out do this, blah, blah, blah, blah, blah. But let's back up for a second. At the end of the day, nobody except the people at this conference sets up a cloud for like no reason, right? Check it out. I set up an OpenSack cloud. What are you gonna do? Did you just have it sit there? No. We've been deploying OpenSack with Juju for a long time now, at least the last three releases. And the whole deploying and configuring and getting OpenSack up, that's great, you need that, right? You need all those tools that I just mentioned. But we tend to forget we are making these clouds and stuff to solve problems, right? Customer problems. And at the end of the day, it's all about services. So Juju, instead of going bottom up, we take a view that we're going top down. Start with the service. Start with what you need to provide. So like, don't lose your mission. Ah, bummer. So I'll show this live in a minute. Sorry, it didn't scale, right? So what we're building here are cloud Lego blocks based around the services, right? So you say, I want a MySQL database. You get a MySQL database. You say, I want PostgreSQL. You get PostgreSQL. You don't have to manually build up, right? And then Juju will talk to OpenSack and this will go fire off and then you'll get a database. And that's great. And you're thinking, oh, okay, but I've got automated things already that kind of do that. Where Juju fits in, and Mark will show you this by example, are those fields of the lines that connect things together? It's all about relationships, right? And what Juju was written for was to define the relationships between services. So you ever do this before. I'm setting up a blog WordPress. You got one machine. Let's say you have two machines because your blog will be really popular. You install your database and then you install WordPress, right? And then you have to wait for this thing to finish installing so you can put its configuration option into WordPress and then you get to move one more step. And then you have to click the button to make WordPress see the database so you can get, and you're zigzagging back and forth until finally you have a working blog. Wouldn't it be great if that relationship could be expressed and you figured that out? And then when you do that, you get horizontal scaling for free because if I know exactly how WordPress is supposed to talk to my SQL and I manage that at the service level, all of a sudden when my blog gets really popular, I could just add three more units of WordPress and those machines will know exactly what to do because I'm defining the relationship at the service level, not at the machine level. So that is what Juju does. And you write these relationships and install hooks and things like that in a set of scripts that we call Charms and that's what Mark will show you later. Charms are awesome because you can write them in any language. In fact, when you get back to work on Monday or whenever your hangover is done, you will, your custom deployment script thing that you've been carrying around in your back pocket for the, you know, between jobs and stuff. Yeah, I know how to deploy that stuff. You know, you got, you like your little scripts those are Charms. You're already probably most of the way there. What's really cool is we take all of these Charms and we stick them in a Charm store and let anyone be able to deploy them. So right now, today when you get home, you can deploy any one of the 120 plus services that are available there. And we really believe in making Charms reusable and shareable and usable by the community because that's what we use. And we give you a nice little GUI too. So it's really great to say, you know what? I need a load balancer. Click drag and a load balancer just happens. You just build your deployment. And that's what it looks like. And Mark will show you later. And this is a zoomed in view. So if you, when you actually do care about machines, you can go in and see what's going on. We'll show you that later. But I'm a command line junkie. So how many of you are familiar with App Get? Right? So a little bit of heresy here. App Get is a great tool. But a lot of people also say, you know, App Get is so great, you know, it's amazing. But when you think about it, App Get itself is really not that sexy. You know what's sexy? Is that it's connected to the huge Debian archive of all the stuff you need, right? So in a way, Juju itself, yeah, it will deploy this stuff for you. But the real goodness here is in WordPress, MySQL and Memcache Charms. So here is just a quick, simple command line, you know, how you would deploy WordPress. You do Juju deploy WordPress, OpenStack, and HP go get me a VM. And I can tweak that with parameters, of course. And then it will go and install WordPress. Now, I got some experience helping a really busy blog site scale WordPress. And it could be, who's here trying to scale WordPress and you're not a WordPress expert? Not a good time. So as we started learning how to scale WordPress, Marco who wrote this charm, he started putting all that goodness into the WordPress charm. So it comes with that caching and it does all the good stuff. So what's really great is, I'd actually, I don't really care, right? I'm getting WordPress and I'm getting a WordPress that's been peer reviewed and checked out by the community and really tweaked. Next, I deployed the database. The guy who wrote the MySQL charm is a really, really good MySQL expert, right? So you know when you go to your user group and it's like, hey man, can someone come to my job and like help me tweak my database, right? And then the guy gives you like tips. He's like, buy my book or whatever, right? What you really want is to get that guy in your server room, right? To kinda hook you up, right? So it's nice. I know that when I deploy this database, I'm not gonna do anything stupid when I'm installing MySQL. Now installing MySQL is relatively easy now. People can do it. But this is just a simple example. So when you think of things like Hadoop or things that are really hairy, when you get the book and the first half is how to install and configure it, right? I hate that. So this is what we do. We WordPress, MySQL, WordPress with memcache, scale's really nice, so we add that as well. And then those are the relations that we're talking about, right? WordPress, MySQL hooks fire off and he will show you these hooks. And then WordPress talks to memcache and it configures memcache to work with WordPress. Which really great here is when you do things at the service level, you get horizontal scaling for free, right? And that's what's so great about the cloud. When my blog becomes really popular, I can just add as many WordPress as I want, right? And I don't have to go back and talk to the database again or memcache because I've done that at the service level, which means I can just hit up, enter, up, enter and scale. And that's really nice. And I can go back down as well. So other things we ship, you can do like cool little options that are in the charts, right? So Marko thought it was a good idea. Wouldn't it be great if the WordPress content, you could stick that in a repo somewhere, right? So we add an option for it. You can do set debug, yes. And your logs will start to get filled up with stuff. And he also did, this is one of the coolest things we do is he made really opinionated decisions of what a good WordPress deployment would look like. And they're not, it's, the best practice is, it really depends on your needs. So the tuning, by default it's single, that means I'm an individual blogger here. We have optimized tuning that tweaks some of the variables and in the database and in WordPress itself to make it really fast. And then bear where it's like, hey, I'm an expert, I don't want you making any decisions for me, I'll do it on my own, right? So wouldn't it be cool if anything that you do deploy to the cloud, you can do things like this? So I'm gonna, we're just gonna show you because that's cooler. Any questions so far, anyone lost? Anyone having, anyone not having a good time because we still have beer, right? Okay, excellent. Yeah, why don't you show the gooey and I'll step off here. I'll let you talk. Okay, let's come back to that. Okay, so George and I are, as he said, we're an ecosystem team. We're really worried about building services on top of Juju. So we care about charms, we care about what it takes to get charms working and what it takes to get more charms in the store to offer more services that are deployable on Juju and Ubuntu. Before we, and we'll spend most of this talking about charms themselves, let's talk a little bit more about Juju and go through a couple of examples of that first. So, and that's kind of what I'd like to do for the rest of this time. And please, you really, really don't want me to appear just talking for a couple of hours. I'll be happy to do it. I taught college for a while, you know. So please interrupt, please ask questions. It's a little harder in this size audience but ask questions from your seat, we'll repeat them if you wanna just walk up with a mic and catch them on video. Stop me at any time, please. Anyway, so we'll talk about Juju, we'll talk about charms and charm development, what it takes to get charms done and then we'll talk a little bit more about Juju and I'll talk about some of the advanced features that it can offer you as a charmer. So when you're trying to charm up the services that you care about, what would that look like? So, okay, hello, and a TPP. So in this part we'll be completely driven on the command line so we will have connectivity issues at times and TPP issues at times. Okay, well, our agenda. Let's take a look at what sort of the basic point of a lot of this. So George showed a couple of pieces of what we're looking at and let me fix the eye so that we can all see this. Okay, brace yourself. It's ugly, but it's readable. Back of the room, see all that, see that, okay? All right, cool. So it's an example of deployment for something like Hadoop, which actually a decent-sized Hadoop stack is not trivial to deploy. This is an end goal for Juju. Juju deploy Hadoop master, Juju deploy Hadoop slave. Relate the two in a role that you'd like them to be and you want the master to read the name node and in this case we're gonna go ahead and deploy it so that the master is also the job tracker and stick the task trackers and the data nodes on the slave nodes. And I did this, I added a multiplicity into my slave deployment. That's the little minus N12. So this actually brings me up a 12 node Hadoop cluster. That's a pretty simplistic example because it's just Hadoop. In real life we've got more things going on than just the primary service we're dealing with in production. So we'll come back to the fact that there's a bit, there are more pieces involved here. Here's a simple stack. So deploy my SQL, deploy media wiki. I'm gonna deploy it with an alias here. The end of this line here, my wiki is actually just an alias for the service that I'm deploying. Excuse me. So we have two services at this point, my SQL, my wiki, we relate the two. We relate the two and I'm sort of telling, there's a little bit of ambiguity between how you might relate to media wiki and my SQL and we'll see that in the charms in a second. And we're just disambiguating that on this particular relationship here. And then I expose it. So this is the provider-definite behavior. Juju handles a series of different providers. It handles OpenStack, as we've discussed. It handles DC2. It handles a local provider, an LXC-based provider. So you can build little micro-clouds on your laptop. Sorry, buzzword micro-cloud, but forgive me, please. The little, but it basically uses LXC containers, lightweight Linux containers on your laptop that works well for an Ubuntu solution. But the, sure, absolutely. Let's, if you, I need to keep going, let me know. Actually, well, let's keep it, I'd like to show a bit more something on screen. Anyway, so there's also, there's OpenStack provider, EC2 provider, local provider. There's also a mass provider, Metal as a Service. It allows you to use Juju to deploy on bare metal and treat metal more ephemerally, like cloud instances. So what that relies on rapid reprovisioning. We're good at those things as an operating system. So, you've seen the mass story probably in the past where we talk about, we use Juju to take mass, the use of mass, Juju on a mass provider to deploy OpenStack itself, and then use Juju on that OpenStack instance to deploy workloads in that. And actually, we didn't mention that here, but we, could get some water when you have a chance. We, in this conference, we could very much just be talking the whole time about taking the mass provider, using Juju to deploy OpenStack itself. George and I are more interested in providing, in talking about services, the types of applications that you are going to be providing, or that you're going to be providing to your customers. The workloads that will run on top of the cloud you're dealing with. We can get into detail of the actual OpenStack charms, and we can talk more about doing that on the mass provider, but most of what we're going to talk about upfront here, at least, is going to be the workloads that you're deploying in the clouds that you've built. Okay, my point in all this, coming back to Expose. So the individual services requests, requests the ports that they're listening on, and they make that request to Juju. However, nothing is open until you pull the crank to Expose it, so Exposed is what adds rules to the security groups to allow inbound access on the ports that were requested by the individual services. That's important, there are tons of situations in initial configuration and startup where compromises are possible in a partially installed system, and this is a really nice way to sort of gate that, so that's the idea behind this. So, before we go into a bunch of other concepts, I'd like to show you something that's a little bit, it's hard to fit on one screen, but the point here is, and we'll go through a whole bunch of the details on here, the point in looking at the stack that I'm describing with this particular set of scripts, or this particular script, is that Juju, this is not just for toys, you'll see a lot of toy deployments because we can fit it on one screen. That's not all we're dealing with here. We have the notion of subordinate services that can manage infrastructure aspects, like log aggregation and backups and things like this. We normally don't show those, those aren't in the brochures and things like this, but don't think that it's just a toy because we're not showing those in the things that fit on one screen. So this is an example of, and I'm just gonna hop down to the bottom here, the idea behind what I'm gonna do is I'm gonna deploy two wikis. And the idea, the goal of this is, let's say I'm a Wikimedia Foundation contributor to Wikipedia, and I do, right, they have Wikimedia Labs, they have an open stack install there. If I'm a developer in that group, I'd like to be able to test things the way they're gonna run close to in production before they get to production. So what you wanna be able to do is take, is take media wiki instances or wiki stacks and deploy them in production-like environments that's more than just like on your laptop or something. So that's an example of where you would use such a, this kind of a stack. So deploy wiki, pulls in. So that's what I'm doing here. Let me, let me go over the setup. Deploy common services. We'll pull a MySQL master. I'll add in two read-only replicas. I will relate them as a master and slave. That's what that's doing there. I'm going to also, am I doing something wrong with the buzzy? Okay, so I'm going to give me an upload store. When you spread, you know, we're gonna put it behind an HA proxy instance or varnish, I don't remember which one. But we put it behind a head. When you spread your wiki instances, wikis have uploads. So those uploads, you'd be out of sync if those didn't share an upload store. That's what the NFS server's there for. Memcache server, deploy ganglia to watch it all come up and then expose ganglia in the step as well. That's pretty safe to do right away. Anyway, so the script starts out by deploying all of those common services. We just talked about. And then for each of those wikis, I'm going to walk through and I forgive the shell, it's the bane of my existence. The point of showing shell and using shell is that it's easy to read and use in common. Right, so Juju has a notion of an environment. Oh, sorry. So the question was, I'm talking about deploy-deploy-deploy and I talked about an example where a developer might wanna deploy something, but I didn't show you how to set up an environment in Juju or even talk about what environments are for Juju. Great question. So Juju supports multiple environments, even multiple environments on the same provider on the same account. I'll show you some examples of environment configuration in a second here. But you would basically have an environment configured, the developer would have a development environment configured or something like this on whatever provider they're working with, be it local on their laptop, if their laptop's beefy enough to handle LXE instances that can do a whole bunch of stuff, or you just test accounts on a cloud or a private cloud, perfect example. And like I said, Wikimedia Labs account is a perfect example of that kind of development situation. You'd have a different environment for production and the developer probably might not even have access to that, might nowadays the continuous deployment. Okay, so basically I'm just, give me four Wiki instances, give me a head on it, relate it to Ganglia, because I wanna watch it come up. And this actually, this part of the script's a little old, the way I would relate to Ganglia is to put a Ganglia emitter as a subordinate service on the MediaWiki instances, but this works too. So I'm relating Ganglia to the Wiki, I'm relating the Wiki to the database, and here's an example of what I was mentioning before about there's a relation, but there's also a notion of an interface, so Juju has the notion of an interface for relations, so relations implement interfaces, and this is an example of a disambiguation of that. So the relationship between a database and the Wiki is scoped in this case as DB, and for the slave, it's explicitly telling the Wiki that it's a read-only slave. MediaWiki, the charm, supports the notion of being related to a read-only slave or replica. Not all charms do that, and it's a good chance to point out, we have, when you look at the set of charms we have, there's a variety of sophistication in the charm, some are just rock solid, and offer all these configuration options, and we'll go through some of them, and some of them are really new and placeholders, and an example of that, and there's some here that you'll see that I use just to show people what's going on, that I wouldn't want to run in production because I'd want to make that more maintainable, I'll show you in a second what I mean, it'll be clear, they're shell script. And then there are also some that are placeholders, a perfect example of that is PostgreSQL, the charm. I wrote the first version of that, not really good with Postgres, I wrote it as a placeholder, and over time we had someone who really knows Postgres really well own the maintainership for that charm, and he's taken it and done Postgres 9 replication and all sorts of great stuff with that charm. But the point here is that MediaWiki has the sophistication to handle a read-only replica, and that's what we're doing there by disambiguating that particular interface, spelling out the interface explicitly. And what happens is when you do this ad relation, Gigi will gripe at you, if there's ambiguity that you need to resolve, it'll ask you to resolve it and present you with the options, we'll see an example of that. Anyway, add it to the NSF server, add it to the memcache server, add it to the head, was there a question? I can't see very well by the way. And then expose that particular head, and do everything we just did for each of those for two different wikis. Maybe I'd want to get fancy and pass a parameter in that size and something like that. The point here, this is a relatively mature, sophisticated stack of services that has closer to what you're going to be seeing in production, not just a toy from that perspective. One thing that could be misconstrued from looking at this is that there's a cinch in which these scripts are cool. It's really neat to kind of see that all built out at once and have it happen, and the time it takes to do this is really kind of ridiculously. And it's funny, the more capabilities we get, the more we want. So I spend my whole life waiting on clouds to provide, to respond to me. And damn it, it's just taking too long still. I wanted to go faster. But when you think about what it's actually doing with that script when that runs, it's ridiculous the power that that gives you, the leverage that you have as an individual doing that. But the thing that can be misconstrued here is that this tool is only for starting these things up. And that's not the case. I could deploy it without the, and here's an example of, like I might start with a wiki, that's just basic, wiki database, connect them, go to town, expose the wiki to the outside world. Three months later, three years later, at some point there's nothing that's saying all that stuff has to happen at the beginning. If you didn't do it at the beginning, you're screwed. What happens, at some point later I can spread that out a bit. I can do something like, so this is what we were looking at before, deploy my SQL, deploy the wiki, relate the two, expose the wiki. Maybe later though, I might wanna do something like, I wanna add more units to it. In order to add more units to it without going down, I need to add a head. And this is where it would have been nice to think about ahead of time of, oh, we should have deployed it first with a head and a wiki and a database and attached our IP address or public IP address to the head instead of the wiki itself, but whatever. So I deploy the head, I'm going to add a relation between the wiki and the head and then I'm gonna do something like, unexpose the wiki, expose the head. You might wanna do this. I mean, you could actually get this to fail over. You could do your DNS so that this could happen without any actual downtime. You would have to probably reverse these order, right? You'd expose the head, then you'd unexpose the wiki, but whatever. And then you'd basically spread it out. Spreading out in downtime depends upon the charm itself. Yeah, and I don't know many wiki well enough. We can dig into what it looks like under the hood to answer that question, what it would look like and whether there would be downtime when we're adding units. It shouldn't because the units would, actually it shouldn't because the wiki would come up. The way these particular services work is Varnish and HAProxy, well HAProxy in this case. Once the HAProxy will be up before any traffic is routed to the back end, the back ends will be up before any traffic is routed to them because what happens is the, and this is the hook order and that gets called. So install hook gets called and a relation hook gets called. I'll cover more of that later. Okay, so Juju, the infrastructure you're dealing with can be manipulated throughout the life cycle. It's a life cycle management tool. It's not just a deployment tool. And it's not just a toy because I can do more sophisticated stacks of services like this. That particular stack we can see here has, and this is a GUI representation. How legible is that from the back? Is that pretty clear? You know, I don't know. I can and then I think I can. But you know what? When I zoom back to fit the whole thing I just undid what I did. Let me see if I can squish it up a little bit. So the GUI is under development, active development still right now. There's a couple of aspects about it. Infrastructure-wise they're pretty interesting and I'll talk about that. That's kind of squished open. So there we can kind of see some of the stuff we were just looking at. What we're seeing in the relation, so the relations of the lines, what we're seeing in the text for the line is actually the relation name. So that's the, yeah, it's relation because it would be HTTP otherwise. So anyway, so we'll make sense of what those relation names map to as far as hooks in a second. Yes, sir. So what? Oh, yes, thank you. I've not told you how do we find out what sort of relations that a particular charm supports? There's a bunch of different ways. We're gonna look at the source but there's also other ways here. So I wanna say, there's one that's gonna have multiple ones. This guy I think I can, nope, not that. There's a way to do it here in the GUI. There we go. Build a relation. And it basically highlights out everything that can't be related to it. That's not everything, yeah. But that's one way to do this is you can actually map the relations between the things. We can probably do, nope, so click. Yeah, okay. There's a, so that's one way is through the GUI and we're working again under active development. That one's still being worked on. In other ways, we could actually kind of see what, we search for the charm here and rather than deploy it, I wanna look at, we can get the description, we can get the interfaces that are there from within the GUI. It tends to actually be cleaner at the moment. There's a jujucharms.com and I can go to SQL. Get the, so what we're looking at here is a set of interfaces. The interfaces that the charm provides and then the interfaces that the charm requires and what it's doing is it's actually listing the relation. The relation's called Munich. The Munich node is the interface name. Here's an example. The DB is the relation name. MySQL is the interface name and when you click on those, it shows you all charms that implement that particular interface. So the AWS relational RDS charm implements that interface. Galera does. There's a bunch of MySQL ones and then there are people who, oh here's charms that consume it. So outflower uses MySQL, cacti, chive, Drupal, et cetera. Etherpad, GitLab, Hive, Juma. There's a bunch. OpenStack, the dashboard. OwnCloud, Rails, so you can kind of start, there's a little web of dependencies there. I'll show you in a second where that's specified in code for each charm but you can certainly get pretty much any information you need about a charm. You can look at the set of hooks it implements. You can look at the set of relations that it provides and requires. And then there's usually a good readme that shows examples of how to use it as well as the basics of how to use it and typically good readme's, let's find the Hadoop readme is particularly a nice one because it shows you how to do the basics when you're doing it like we just saw in the example before where your name note and your data note are in the same. Or your name note and your job tracker are on the same master. That's not necessary. You can split stuff up as much as possible. He talks about rolling upgrades with HDFS and things like that in here. So we try as reviewers for the charms that get blessed in the store to make sure that they have a really good readme that kind of show you what's going on there. But the future of that does absolutely involve being able to do things like, I keep moving it while I'm clicking the hold it is the problem. I don't know to be honest with it. So the question was, what's the relationship to heat? Where do we see that relationship going forward? My understanding is heat in triple O is specific or specific to continuous integration. They're not sort of planning on dealing with packaged things and stuff like this. And that's my understanding. I might be there. But there's, so juju can be, we'll talk about integration with other tools in a bit because there's a lot involved with that. Short story is juju can be top dog in some of those situations but also can be driven by other things. So it's a juju API. And actually one of the, the Niskuli is a backbone JS app that drives a juju API over a web socket. So that's a, the idea going forward is that there's a lot of flexibility there. Okay, so we can get a lot, we can get a lot of information about what's going on with Charms. We can sort of, there's a couple of tools that let us manipulate things and work around that. One thing I wanted to do before we go too much further. So we saw some examples of how to use juju. Yeah, I'll get to a charm. Oh, so architecturally, let's talk for a second about juju itself. So the idea, you know, the probably the most weight in juju honestly is the providers. Juju at its heart was created to scratch the itch of distributed coordination or coordination in distributed configuration management like George described. I always have these problems of where I need the information at this point of the configuration that I need from this one. I gotta wait for this sort of come up if it ever comes up and get the information out of that and stick it over here. Juju was created to scratch that itch and that itch pretty much alone. So as a solution to that problem, what it is is it's really a declarative specification of your infrastructure. Currently, this is done in ZooKeeper. Put in what you want your infrastructure to look like. At the service level, so you specify basically enough information to see this in ZooKeeper. And then juju is on top of that, a handful of agents that subscribe to topology changes in ZooKeeper. So you've got a declarative specification of your infrastructure that you manage over time and agents that pick it up. Those agents can be considered part of the operating system where an operating system is easier at packages for it. They're actually, I think, in universe. Is it juju and main or universe? Yeah, it's in universe. So it acts like it's part of the operating system. There's no real difference at that point. Anyway, so that's a, and the agents are busy and they make that happen. And then that's all communicating with you when you're looking at things. Most of the state information is stored in ZooKeeper itself so in that state. It's really a very thin, what is that? That ends up being a very thin eventing mechanism, a very thin coordination layer. How, the key is that it's calling the right, it's calling hooks within the charms at the right time. When you do an install, it calls the install hook for that service. When you relate that service to another service, it calls the relation hooks for that service. And there's a little bit of a relation life cycle in where there are several hooks for every relation. There's joins and then there's change. And essentially the relations end up being a communication channel between two different services or rather all of the units on two different services. And that communication channel is open as long as either side continues, and change hooks will continue to be fired. And we'll talk about that in a second. So, and as we saw here, it's really this set of hooks is what the charm is. And that's all a charm is. If I look at the structure for, there's a charm. My SQL's a longer list because it had more hooks in there. But at its base, this is a really pretty basic charm. At its base, it's got configuration. So a charm tells you how to externalize the configuration parameters that you might need to externalize for a particular service. We'll look at that in more detail. Copyright, so let everybody know they can use this. Hooks and then some metadata in a readme file and a revision and let the charm store kind of manage what's going on there. The metadata is the first thing to look at. And so let's look at, it's really pretty straightforward. There's a maintainer. There's a summary description and then there's, I have a MongoDB relation. So this is actually a charm that deploys applications that are based on node.js. It's really trivial charm. It's not meant to be a particularly sophisticated one. It's more meant to show you examples of a really simple charm. And what does this one do is the node apps that this one would take would be a website. On the other side, MongoDB, close that full back. I've got a, it provides Mongo, it provides a database, provides config server interface. It requires other Mongo config instances. It requires a Mongo sharding services. And it does a replica set for peers. So there are potentially interesting deployments I can do with MongoDB, the charm. It offers quite a bit and it offers quite a bit of configuration as well. But what's important there is notice that this guy's requires matches up somehow with this guy's provides. They requires what's the Mongo with an interface MongoDB. And as long as that's the case, I can glue them together like Legos, okay? And so let's look into what actually happens in that particular relationship. Or we'll see that when we're digging in the hooks. Notice that for, sorry? Yes, sir. Are the relations always? You can have a generic interface that can have lots of providers. There is no notion of interface hierarchy at the moment. So it would make sense that MySQL and PostgresQL should be children of a SQL interface, say. There's no such notion right now. What we're doing right now and the way we have it specified is actually we should go back and fix that in the store at some point is MySQL and PostgresQL, all you'd really need to do is take each of those interfaces, turn them in the SQL and then extend that common interface with a type field that whatever you're connecting, whatever is consuming can then load the appropriate driver based on the type or something like that. And that's where you were seeing or when we click here of what's, so let's go look at that. Node app requires MongoDB. That's that interface there. That's where we get all the, that interface is provided by MongoDB charms. And you're looking at different, you're looking at different owner. This is actually not just looking at the official charms in the store, that would be the one owned by charmers. Here you see charmers on Eric MongoDB and it really irks me that we, so internal to Ubuntu really, we refer to releases of Ubuntu, not by their numbers but by their names and outside before I joined Canonical, I was like names, what are names? I don't care. I just want 1204 or et cetera. Anyway, so what you're seeing here two store owned charms, this one and this one, one for Precise and one for on Eric 1204 and 1110. The, correct. Sorry, the question was, we're taking responsibility for name conflicts in the official versions in absolute, yes. And series as well. Notice that there's not a quantal or a raring one. With Juju, it's really, what I do typically is I will have heterogeneous environments. What I recommend you do is use LTS only. There are times when that doesn't work, when what you need has not been back ported to LTS and it is perfectly reasonable and it's done all the time to have heterogeneous environments. You don't care what series is running for that particular service. You care that the that MySQL is running in the best way that it can possibly run. So if that's a quantal instance and other things are running on Precise, that's perfectly okay. There are ways to kind of specify, okay, no, I can only because we're whatever working for the government or something, we can only run on Precise instances that have been approved, blah, blah, blah. So that's an option and we work really hard to maintain the LTS versions in the store and then there are situations where you really need the one that like Ceph is a perfect example, Ceph really needs the later versions of the charm or the versions of the charm that work on later series. Although James may have had the back ported to Precise at this point, I don't know. Anyway. And so here we see, here's the, you know again the charms that provide that MongoDB interface, the charms that consume that MongoDB interface and you see in the GUI, we're trying to make that a little more automated so it's easier for discovery and search and things like this. The website is the best and authoritative place to do it right now. As you saw in Mark's keynote landscape, we'll have it integrated but through the Jujugui that we're looking at here. So that's all single, single source. Okay, so we see some hooks here, we saw the metadata file, we saw that the metadata as long as the provides and the requires match, then we can glue these together like Legos. We've got a dependency graph, if you will, between charms and services. Interesting point, we're actually doing some charm testing. Some of the ways we do charm testing is to walk that graph as widely as we practically can and spin up little stacks of services in an automated manner that spin up the service and then anything that consumes it and anything that it consumes, one step out, sometimes we go to two if we're feeling naughty. For every one of those combinations, it has to pass, everything has to be up and then related in order for the charm to pass that test. That's our automated tests. We've worth mentioning here. It's landed since this has been written. Every charm at some point soon will require tests within the charm itself before it gets accepted into the official store. Also, we were looking here at, and ggcharms.com, if you go the GitHub way, there's a charms repo here, George's big smile and mug. github.com slash charms and what we can see here is just, we're sort of the official store versions of the charms only. What we're looking at here is both the store versions of the charm as well as the, so where, what do you call it? Private charms or not private charms, but charms owned by people, particular branches. Juan Negron has a MongoDB charm here. So, we say, go ahead, yeah, absolutely. So, his question was, I see the charms run a lot of app-get commands. Does it ever make sense to just do puppet apply? Absolutely, we have charms that do that. Okay, so one of the things he didn't mention is the charm store, we don't freeze it, which means some people, for their charms, they need a new version of something. Actually, some charms have deploy options where you can say, we default to distro and then that hook is app-get install, whatever, right? An option might be second fig, upstream source equals github and then it'll pull it, build it on the spot and deploy that, or another option might be use a certain PPA. So, what we want to do in charms is basically give people options, right? Because it is pretty cool, especially as LTSs get older, right? You start aching, you're like, okay, so this version of this thing was good for me then, but now I'm kinda, you know. So, charms themselves, a charm is a script that runs at root as root on the instance when it launches, right? It just executes that stuff. Whatever, however you wanna get that software installed is totally up to you. Some charms have an option to build from source on the spot, some deploy a custom configuration. The whole thing could be a binary block. I don't really care. What we do care about is giving people the option to deploy what they need in the cloud and sometimes it's really nice to be able to just pull your entire stack from github, right? So that's cool. But generally speaking, in this store, by default when you're doing a deploy, if there's a package available, it's actually a policy, you default to the safe thing that was shipped in 1204. Over time, we want every charm to give people deployment options just like that. And we talked about store charms as the sort of official blessed ones and the real workflow there that's evolving is that you don't typically care about MySQL. You don't typically care about HA proxy. I don't, I just want them to work. I want enough config options externalized so that I can tune it the way I want if I want to. But I also want same defaults to be chosen for me for those services. I just don't care. I just want them to work. However, my application itself, the charm that I've written for my Node app or my Rails app or my Django app, I really care about. So I might not use a generic version of those. I might fork and maintain my own Rails charm that then is tweaked with all the special sauce that I want to do for that. So it's perfectly fine to have environments that have mixed charms that deploy either directly from the store or directly from your frozen branches that you keep in a repository that you're then deploying from those depending on how you roll for production. That's, you know, freezing charms is probably just like freezing code. Yes, sir? Depends on your particular service set. So his question was we're doing development on Quantum. We're gonna run into problems interfacing with charms that are running on precise. Usually not at all. It's your particular, what you would do in that case is take your charm that you care about and make that based on Quantum. It's really easy to deploy a Quantum-based charm. Did you deploy a charm store, or an ordinary even, you know, so I could specify a particular series. I can even make it, I specify the owner from a branch and launch pad or something like that. You can't quite do Juju deploy GitHub URL yet. I'm working really hard to push that through. But, and we'll get back to the question of how do you integrate with Chef and Puppet on that because there's good answers to that and I'll show you examples of it. The hooks we're looking at can be written in anything and the things that you care about are probably a limited subset of the services that you deploy. You probably deploy a whole lot of services where you want whatever the experts, the community, the upstreams decide or the defaults or the best practices for, you wanna be able to take advantage of the fact that that charm encapsulates those best practices and deploy it that way. HAProxy, my SQL, yeah, I care about my app. Yes, sir. Yes, there's a couple of tools that we've, yeah, sorry. Thank you. He's asking, he sees a bunch of shell scripts, shell scripts are lame. Is there any sort of format I can use to describe what's going on here and then slurp that in? And absolutely, we have a couple of hack together versions that work with a previous version of Juju or a stale version of Juju that is being changed dramatically right now. We have the notion of Jitsu export and Jitsu import that'll allow you to pull out a one cloud and move to another cloud and we demoed that at OSCON and stuff. What I do in this case to answer what you're looking for is Juju API. So the Juju API can be programmatically driven. You can easily put together a JSON description of what sort of infrastructure you want and then call the Juju API according. That's the idea going forward with that and that is working right now. It's not quite as clean as I would like it to be for production as it will be for production soon. And what I mean by that is just that in order to get that server, you actually have to deploy that server within your infrastructure. It doesn't come as part of when you bootstrap an environment. So I want the GUI, the API server, all of that will be part of when you bootstrap an environment. It'll be there right now. You gotta add it extra. If you look at the... Yes, sir. Excellent question. So he's asking the shell scripts are sequential. Does that imply that the deployments are? No, not at all. They're just done that way because we look at them that way. You obviously have to deploy things before you relate them to make the infrastructure aware of it. If there is any sort of ordering in the relations or there's gonna be no ordering in the deploy services install. So you can deploy in service before another service or vice versa. That doesn't matter. They're deployed on different nodes. The install hooks are isolated. It's the relations that could be ordered dependent. And we are really careful in reviewing charms to ensure that they should not have any order dependence. And in practice, whenever we've found order dependence, we've been able to resolve that with that and not do order, I think. For everything I'm aware of in the store at the moment. And I'm one of the main reviewers. Yes, sir. So they're... Yes. Can I get any information about where my services get deployed in my infrastructure? This is very relevant to a lot of things. And the answer is kind of. Right now, so there are, and I think this, no, this landed in core. We were writing, we were doing this. We've been doing this for a while with this Jetsu external set of tools. This landed in core very recently. I don't know that it's been pushed up to the archive yet. The notion of deploy to a particular machine. So you can control things that way. The way we're addressing that mostly is to be able to do things like specifying constraints. You don't wanna put a database on a thin instance. You want your database to be fat. So I wanna specify constraints that this particular service, when I'm deploying it, do you deploy minus, minus constraints? Is what, I'll just actually, sorry, self promotion, but it's not really, I'm just going to the config. So you'd see deployment lines that look like, the middle there, Gigi deploy from a repository with constraints specified here. In this case, I'm doing an instance type, but I could certainly specify ram greater than four gig comma CPUs greater than five or two or whatever it is for four gig. And Gigi will do its best based upon the underlying provider to match that. However, if you know your provider has the ability to specify it through an instance type that's supported, but we're trying to get away from those, trying to keep it as provider independent as possible. That's a provider dependent constraint. Bad. But that's possible. Yes, sir. Correct. The by default, so this question is by default, everything is deployed to a server of its own. Everything, yes, everything gets a dedicated instance for primary services. There's also the notion of a subordinate service like a monitoring, a log aggregator or a backup service or an NFS client service or something like that that gets tacked onto the same container that your primary services is on. But every primary service gets its own container by default and any variations from that are doable. We do it all the time, they're just forced. And that's the deployed to is the way that we've done that. Situations, so there's a bootstrap node. When you do a juju bootstrap, it takes up an instance. You're paying for that. If you really care what you're paying on for that, you can actually go ahead and deploy your database and your WordPress server onto there and then spread it out later to different instances. And that's where we've kind of hacked around to do that initial deploy. Most of the default behavior is that it's gonna take up that bootstrap, it's gonna take up an instance just for the management itself. And more than an instance if you're doing HA, right? So you would do either a zookeeper quorum or the new, the rewrite of juju involves HA, that's by design. The older version is a hack on that. We've gotten working. So we can deploy with constraints. We can deploy with configuration. Let's see, what else do we need to talk about? Let's go in and look at what actual hooks look like. So, and this is, here's a great example of, so notice that I'm deploying here a, can y'all see this at all? I'll try to get that a little big. This is a juju deploy line and what I'm deploying is Hadoop. And this is where we start to try to get big to do some scale testing. But notice this constraint line that I have sort of configured there, or highlighted there. There's also the notion of config. So when I'm deploying a service, we see that these services offer a, in this case, MongoDB, and this is not the, let's go to, MongoDB has a whole slew of config options down here that you can do. And things like the number of backup copies kept, the replica set name, verbosity, MMS interval, bind IP, OP log size, there's a lot of stuff here. The good news for me when I'm working with MongoDB is I really want someone else to make those decisions for me and Mongo does, the charm does. So, but they're all there as config. How do I specify that config? I pass in, when I'm deploying, is one way. I can pass the config as a group with a YAML file. And that looks like, here's the config for Hadoop master, for instance, you wanna set the heap size, the DFS block size, the fact that I'm going to mount on in this case, whatever the local store was for that particular provider, I don't want it to go off of the root file system. Slave is a little smaller heap size, there were mediums. I do set the block size. I set the number of parallel copies. There's a lot of configuration parameters I can set. I can set that at deploy time via the deploy with config here, but I can also set that along the way with juju set, so I can do something like, and this depends very much on the service you're talking about. I can do a, literally the command is juju set, so let's see what we have deployed. Do you know a good setting to play with on? Do we have a WordPress deployed? I don't think we do. I'm sorry. I'm sorry, can you restate that? Change IPs, I have no idea. That's provider related. That wouldn't be a specific thing in the charm. That wouldn't be a configuration for the service itself. That would be a configuration for the provider, right? Am I misunderstanding something? I don't know what you mean by swap IP. Yeah, what happened? I'm sorry. So, well let's take a look at that. I mean I do see, when you do a NovaList, so I mean that's worth investigating. If I do a NovaList for what I have up right now. 10 minute warning for what? Oh, I think I have my region setting correctly in my NovaGloin. It's different from what I'm deploying right now. Yeah, it is. And I can't do that on the screen with everybody watching. Yeah, so we're not gonna do that through Nova. But we can do a juju status as a way, when I'm wanting to know what's deployed. And really what I'd like to do, well we'll just scroll back through it. There's a way to pass filters when you're interested in only a status of a particular service. You don't have to look at all this craft. But let's see what we're doing up here. So we're getting what looked like public IPs up here in DNS name. Those are machine states. We have some that are still pinned. Oh, you know what? I'm over my account one, I bet. We're doing, yeah. Anyway, so there's some of these, like be able to classroom that I just spun up that didn't come up, because I'm already over my account on my instance limit. The, you see here the name of the service. I'm trying to do VI, I'm writing. You see the name of the service here. You see the name of the unit. So a service can have multiple units and I have my little unit ID at the end of the service name, slash zero, slash one, slash two. And then I see that it's an age, say pending and I don't have an address because it didn't assign one. Ganglia, however, I spun up from the very beginning. We can see that it's in a started state. It's got a machine associated with it. The machine was the list we were looking at above. And then the public address is associated here. The GUI, yes. So depends. If the, yes. He's asking, when I see pending, how do I tell what's going on? How do I debug this? How do I figure out what's going on? There's lots of, I'll show how to debug charms and charm hooks. We actually have a debug hooks that is really, really cool and really, really useful and essentially breaks you. Gives you a teamwork session onto the instance you're trying, onto the unit you're trying to debug. And then it'll pop open a new window and break you at when the particular hook is called. And so you then have a choice to manually run through the hook and then exit. And so it's a nice little debug session based on Tmux. That's when it's up. In this particular case, the problem is, so there's lots of modes of failure. One is that the provider is saying, no, sorry. And we see that because the machine IDs were still pending up top. There's, those could be up and running. And then the services in a pending state, I doubt we have any of those because I haven't deployed anything for a while. The latest thing I deployed was WLB Classroom. But it's, but when the machines come up, they'll be in a started state, but Juju's still running this install hook. And that's, the agent state is pending. Agent state here that started here is pending in that situation. If you wanna watch what's going on there, the best way you can, of course, secure shell data, these machines, it's easy to do a Juju SSH ganglia zero. It takes me straight to that machine. I can do it by name. I don't really care what its IP address is. It injects the keys from the environment that I'm running Juju commands from. There's an environment option to add additional keys that you wanna inject to the instances. There's some production best practices around key management stuff we have written up. You can SSH to the machine, but there's also the best way to do it is a debug log that essentially will, it's a global log of what's going on in the interface. There's a way to, there might be a way to scope it. Like status, Juju debug status only for ganglia. I don't care about the rest. I just wanna see the status for ganglia. There might be a way to do that for the debug log where I'm only going to get, so I can specify warning levels. There's an exclude, but I'm not sure what the include and exclude are. Yeah, thank you. Oh, there you go. Yeah, for individual agents. So yes, we can call that. We can do Juju log, or debug log that's not just the global mess that this is. It's great actually when you're watching things come up though because you get what's going on. The newer version of Juju actually just uses our syslog for this. So that, yeah, and there's work underway to ensure that that's scaling to the degree that we need to go and figure out what are the alternatives as we get to 15,000, 20,000 nodes. And what is this five minutes until what? Till we're done done? Yeah. Dude, I don't know we're near done. Yeah, all right. Let me cruise through a couple things here so this is an example of the node apps install hook. You can see this in jujucharms.com yourself. You can walk through this. But the main points I wanted to get to is you can pick up the config that specified. Some charms are written so that you can specify the config via juju set on the command line at any time throughout the life of the charm or the service. Some charms are written so that you specify it as a mutable initial config. This particular charm, and that is less optimal. We recommend that people, when we're reviewing charms for acceptance in the store, we don't want a mutable config like this. We want config that can be changed throughout the life side of the service. So, but that behavior is very dependent upon the particular service itself. You get a lot of differences between services. Yes, sir. Right, right. Let me show you that real quick. So, his question was, when I have two different services and I'm dealing with relations, how do they share configuration information? And that is really the point I'm trying to get to the whole talk. Here's how I get config from the YAML files that I passed when I'm deploying, is I just kind of pull it using a config get command. And in this case, I'm gonna do things like, I can install for PPAs. And frankly, so as in Ruby development, for instance, I'm sick of hearing, oh, don't use Ubuntu, or don't use the Ubuntu packages for Ruby development because they're so old. You install Ubuntu fine, everybody does on the cloud, but you need to install Ruby from packages or from upstream or something like that or build the enterprise Ruby version or something like this. Juju, in the charms, anything goes. In the store charms, we have some preferences, but in a charm, you can do anything you want. I can do RBM, I can do gem install or bundle install. I can do what I'm doing here. I'm actually pulling from a repo that I've specified in configuration. And then I'm doing an NPM install, NPM install on the package.json that's specified there. Yes, sir. Trusted or not is the signatures, right? So you can't do an add app repository would fail if the keys are bad. So, yeah, right. And so we can tell here that we would know if it didn't come from Chris Lee, but as a charm author, I'm making the decision here to pull it from his VPA. What we try to do in review is make sure that that's only coming from trusted sources, right? Yes, and anyone can write a charm that does anything you want and push it up. That won't necessarily get in the store, though. Who's Chris Lee? Yeah, wait, who's Chris Lee? So in this case, in this case, actually. So this is community police. Yeah, actually Chris Lee is a well-known node guy and people are using his PPA in production and things like that. So I personally trust the people that tell me that Chris Lee is a good guy. However, in the store, by default, like we say, we want, you know, you'll get, by default it would say app get installed, node.js, and you would get an old version. That's why we want charms to have options. Wouldn't it be nice if you could just pass any node PPA from whoever, or the one you built for yourself that you've checked. So, let's move on to relations real quick. Yeah, if you were to submit this, he would yell at you. Yeah, he would say, hey, wait a minute. It would get knacked for, and this one's got all kinds of things that I would change, for instance, HearDocs and Bash. You know, I'd use real templating tools. And this is where a puppet would come in handy with that ERB or even Cheetah or something like that. But repo, it is, but this is better than the curl pipe to sudo. That's pretty standard in the node community, but anyway, the point here is this is the install hook. It's really simple. It's written in shell, it doesn't have to be written in shell. It could be compiled binaries, it could be Ruby, it could be anything with an Ubuntu runtime. You can write this. And in particular, you could just, we have charms that just do puppet apply install manifests or the equivalent for Chef Solo recipes, right? This is a great situation in which to use awesome DSLs for config management. No problem, it's great. The difference is that you'd pull from config and shove in a factor, right, before you're applying that manifest. And that's easy, and that's exactly what the charm I'm talking about does. I can show it, but we're really don't have time to do that. Let me get on. So this is, you know, the install app, it pulls node and installs any dependent node dependencies. It sets up in a NIT script, does some basic configuration for the app. None of this is related to a relation. However, the MongoDB relation is where, and this answers your question, sir, about how do they share config information between the services? And this is where the node service is in this hook, gets called when they're talking to each other, and there's a channel that opens and either side can do a relation set or relation get. And relation set passes information over that channel, and relation get is what allows this guy to pick up that private address. So this is what allows me, as a person that's managing this junk, to never know the IP address of the services I'm dealing with if I don't care about them. I mean, I can get it from status, of course, and I can secure shell there, and even, you know, I showed you an example of I was secure shelling with the instance where the service name, whatever it was on the other there, but I could certainly just put in the IP address, no problem. Ubuntu at 15 dot, whatever. But I don't have to, because I have this channel where they can really talk amongst yourselves. Yes, sir? Absolutely, absolutely. And this is where the sequencing of events, and I really thought we had like two more hours. I'm really sorry. This is where the sequencing of these events happen, and you will notice that the tree for node app hooks and the tree for MongoDB hooks, crap. Let's put that next to each other, maybe. Nevermind, that's not gonna work. The notice that there's MongoDB database relation joined here, and then there are MongoDB relation changed there. So join gets called on both sides first, typically with, and this is dependent upon the particular services themselves of what makes sense. A database typically, when it gets adjoined, it takes the service name, creates a database, creates credentials for that particular service to join, and then that's passed over the channel, and the application that's consuming it will pick that up through relation gets. It gets that in a changed hook. So there's sequencing associated with those two particular relation, the hooks associated with a particular relation. Join, change, change, change, and change will keep firing while, as long as either side is still doing a relation set, you could play tic-tac-toe, no problem. And that's it. So seeing, as a term author, understanding that sequencing of join to change is what the magic is behind there, in addition to the ability to pull something from the relation. And in this case, what you're doing is actually, when it comes time to configure the app, I'm gone, I'm gone. When it comes time to configure the app, in this case, you're just doing really simplistic stuff. Again, this is not something I would use in production because it's just setting a config file. In production, I would use a template, like an ERB template, and apply the changes to that template. But it could be as simple as setting a config file with the changes that you just got from the other side of the relation. Police them, I don't, yes, in review, we very carefully walk through that. At runtime, the thing is, I don't think circular dependencies, yeah, there's both a static review and dynamic reviews, so we do test that. But it hasn't been an issue to date, honestly. So there may need to be, we do have the automated testing though that would fail the test. So it would poke out, hey, we have some circular dependencies here. So that's it. So before you leave, I've got t-shirts for you guys. Thank you for staying for the whole way. Did we float the keg, or did we make a dent back there for our tender guy? I don't drink more, I guess. Yeah, so head back in the corner. There are surveys on your table, and we are very keen to find out what kind of workload you guys are doing. So if you wanna do that, and drop it off here, drop it off at the booth, doc rooms will be at the booth. You know, if you wanna ask questions. The shirts are ordered from smallest to XXL up here. I don't know if I have enough for everyone, but hopefully I will. Little slides of the talk. Thank you very much for coming. I really appreciate it. Thank you. Thanks for coming out. I have no idea.