 What's up? Welcome to infrastructure coding with Chef. We're going to introduce ourselves and then we're going to get into the presentation. Also, if you want to click that link there, you can go and grab all of the slides and code that we're going to walk through right now. It's difficult to click it from your chair, I understand that, but give it your best shot. Is it a Microsoft site or thing? Yes. All right, so this is great. Why don't you introduce yourself while I figure out why my slide is on this page? Exactly. Awesome. I'm Nell Shamarill Harrington. I'm a software development engineer with Chef. If you've seen me speak before, you may know about my undying love of regular expressions. I'm not going to talk about those today, but if you're interested, I've got plenty of work on it and I'm always happy to talk about it. You've got other bullet points you can say. I do. So the main thing I work on in Chef is the supermarket product. It's a Ruby on Rails application. That is the application where we keep all the community cookbooks. So people can upload community cookbooks, download them, install them directly from the supermarket. It's at supermarket.chef.io. You want to check it out. The final thing, I'm an Ayanada practitioner. Ayanada is a Japanese martial art. If you're a student learning more about that, just let me know. Over to you, Nathan. So hi, everyone. My name is Nathan Harvey. I'm a community director at Chef, which means that if I tell you how to run your infrastructure, you should not believe me because I don't manage infrastructure. I manage a community and work with people. But I use to manage infrastructure, so maybe like I used to have credibility, but I wouldn't trust me much anymore. It's cool, though. The other thing I do is I co-host the Food Fight Show, which is a podcast that's all about DevOps and Chef. I'd love it if you could subscribe and listen to that. You might be disappointed, but it will make you feel better if you tell me that you listen to it, so that's cool. I occasionally farm and I love eggs, and if you were to click on either of those links, you would understand that I actually never farm, and you probably understand why I no longer like eggs. You can follow me on Twitter if you like. You can also email me. Yeah, that's me. So let's go ahead and get started. I'd like to know a little bit about you. So if you were a system administrator, could you raise your hand, or if you've ever been one in the past? If you're a leader to that, okay, cool. And how many of you are developers? Awesome, like the rest of you. And even some of those systems administrators raised their hands? That's a crazy thought. How many of you are DevOps? Awesome. That's good. That's good. Good stuff. Oh, I don't want to put my water on my phone. That would have been bad. And business people? How many of you are business people? Bullshit, raise your hand. You're all business people. All of us. If you write software, you're a business person. If you're a system administrator, you're a business person. Because you have a job because you work at a business or an organization that cares about customers or people that use your stuff. So you have a vested interest in that. You are a business person. And how many of you are experienced with infrastructure as code as like a thing or with configuration management, like a software or framework or platform that will do that? Awesome. And how many of those are experienced with Chef? You can lie. I'd rather you didn't lie but it's cool to deal. Okay? Awesome. And so when you're experienced with Chef and you're here, what I always tell folks when we do introductory talks is when we lie, your job is to keep your mouth shut. Like don't let anyone else in on the secret that what we're doing is telling them the hard way. It's a better way to learn. I'm sure. So I want to start with a story, a quick story here. So I grew up in the 80s and a little while ago I was watching this show on National Geographic that was called The 80s, The Decade That Made Us. And this show actually defined DevOps in a really clean way for me because I'll be honest, I've been to a lot of developer conferences and a lot of developers come up to me and they say, I hate DevOps. I don't like to DevOps. I think DevOps is stupid. And the problem when they say that is that you just don't understand what DevOps is. Or rather, we don't have a common understanding of what the definition of that term is. So I want to share how this amazing show on National Geographic explained DevOps for me. So in order to do so, I have to kind of take you back in your mind to the 80s. And some of you, this is a completely foreign time because we're very, very young or not yet with us. But I was definitely there. And here's one of the things that happened back then. There were bands on the radio. You could listen to them. You couldn't listen to them on the internet yet because they hadn't discovered it. You know, I'll go ahead and invent it actually. So we couldn't do that. But the bands, they were kind of defining characteristics of the bands. So sometimes there were rock bands and there were also rap bands. And the two would never collaborate with one another, ever, ever, ever. One of my favorite bands back in the day was Run DMC. And this guy had this crazy idea that what if Run DMC made a rock song? Run DMC, if it's not clear, they were a rap band. So as I said, they would never, never think about making a rock song. But some guy said, hey, you should totally do this. And I want you to listen to this song from this rock band and I'd like you to cover it. And so they went and they listened to it and they said, oh hell no, this is hillbilly gibberish. They did not understand what was happening. And the minute I heard that, myself as an administrator, I related to that. And I said, you know, it's like me, when the developers hand me code, I don't understand this code. I don't know. How am I supposed to run this in production? Well, they talked to the rock band who happened to be Aerosmith who was going to cover, whose song they were going to cover. And they said, yeah, we want to put you together with Run DMC and sing this song. And Stephen Tyler said, what the hell are they doing to my song? Which frankly, kind of sounded like a developer coming up to me and saying, hey, works fine on my machine. What's your problem? So, but this amazing thing happened. This producer, Rick Rubin, put them both into the same studio, put them together and in his words, it was crazy good man. So, these guys came together. The video is a perfect description of Demops also because in the rock video, you start off with the rockers on one side and the rockers on the other side and there was a literal wall between them. And during the video, they tear down that wall and they start working together. Now the thing about Demops is they didn't ask Run DMC to start being rockers and they didn't ask Aerosmith to start being rappers. They said, keep what you're good at. Just work together to make the audience happy. And that's what Demops is. So, I'd like to propose a definition and whenever you hear someone saying Demops, I want you to think about it as a cultural and professional movement that's focused on how to build and operate high-velocity organizations born from the experiences of its practitioners. Now this is how I define Demops. And so when you say to me that you hate Demops, I say we don't have a common understanding of what that is. So, I would propose that this is our common understanding of Demops. And I hope you'll agree, but it doesn't matter because I'm on the stage. So, now you all agree this is our common definition of Demops. Are we good? Alright, let's talk about some infrastructure as code. So, I had to just rant a little bit about Demops, but I hope that was a fun little journey. Okay, so infrastructure as code, we start treating our infrastructure as code for many, many different reasons. Of course what we're doing is we're automating the infrastructure that runs our applications. As it turns out, applications that work on your laptop are not of much value to anyone other than you. We want to take our applications and put them into some real infrastructure, whether that's infrastructure on the cloud, whether that's infrastructure in your data center. And by doing so, we want to, or as we do so, we want to automate that entire process. When we think about, well, when do I need to automate? Like, isn't Git push to Roku? Is that, they might devop it? Like, sure. Of course you can devop it if you're Git pushing to Roku. You're a high-velocity organization, but at some point you may want to put your applications on some other infrastructure. And when that time comes, you'll know it because you will have reached some scale. And that scale doesn't necessarily mean the number of machines you have. It could be the number of people that you have that are building out your infrastructure, or building out your applications. It could be the complexity of those applications, and so forth. So you really need an automation platform that can handle your automation needs, that can handle your infrastructure and really allow you to treat it as code. The thing I'm not going to do is read you all of the bullets on this slide. The thing I have already done is made all of these slides available to you so you can read them for yourself. But automation platform, it's a thing that you want. You can use any automation platform that you like. It should be the automation platform that works well for you and your team. And there is one other criteria that you may disagree with. The automation platform that you use, you should not roll it yourself. You should use a framework that's out there and open source already on the market. So infrastructure is code. What does that mean? It means that you can programmatically provision and configure the components within your infrastructure. Whether it's hardware, whether it's a cloud instance, it doesn't really matter. It means that you treat your infrastructure code like any other code base. Now when I say treat something like any other code base, what do you think I mean? This is the audience participation part. Source control. You put your infrastructure code into a version control system. What version control system do you like? Get. Yeah. You can use any version control system you like. It should start with GIT. What else does it mean to treat your software like code? Or treat your infrastructure like any other code base? Test. Test. We should write some tests. We kind of gave it away in the name of the talk. Like maybe we're going to talk about some testing. So yes, testing. But why don't we put our things in source code control? Why don't we test our infrastructure code? I'm sorry. There were three answers at once. And I think, let me just distill them all down. Because code is a continual experiment. We're never done with it. And your infrastructure code is no different than your application code. Overtime, you're going to refactor it. You're going to, your requirements are going to change. Therefore, your infrastructure is going to change. Therefore, your code is going to change. You four that answered my question. That's basically what it said, right? Yeah, pretty much, right? Okay, good. Again, it's an incredible time. I'll never know. With treating your infrastructure as code, you can reconstruct your entire business with three things. With your code repository, a backup of your data, and with compute resources. I will tell you that I used to be a system administrator. And before, like when I first became a system administrator, I was a terrible system administrator. And that's a longer story that I'm happy to tell you after this talk. But there was one day in my journey as a system administrator where I realized that my next change to my production environment was going to start with a commit to a git repository. That day, I became a free man. No longer was I logging into production and banging on the keyboard and hoping for the best. This is where I want you all to be in the free world that is infrastructure as code. Alright, that's infrastructure as code. With Chef, and with many of these infrastructure as code platforms, what we're working with is a policy based system. So in your code, you're going to capture the policy for your infrastructure. What are the requirements of your infrastructure? What does that look like? And each one of these, and of course we're talking mostly about Chef today, but each one of these will have some sort of application, some sort of program that will make sure that the policy continually apply within your infrastructure. So over time as your needs change, you can change your policy and your infrastructure will be updated. Which is great, except when you change your policy and then all of your infrastructure gets updated. Because maybe you wrote the wrong policy and you just brought down all of your infrastructure, that would be bad. So we test. Let's see. So if we take a very simple infrastructure, we've got some graphite, some rails, a little database. Let's say you have a security firm come in and do an audit on your infrastructure. How many have been through a security audit on your infrastructure before? Alright, so this story will ring very true to you. So security team comes in and they say, you know what, this infrastructure is not secure and here's why SSH is listening on port 22. So the way that we're going to secure this infrastructure is we're going to have SSH listen on a different port. Because that's security, right? You guys are laughing because you've been through the security audits and you know that that's true. That's how we secure things. What we think about, what does that mean for our infrastructure? It's really one line in one file on these machines. Like who could edit six lines of code or one line of code six times by hand? You should all raise your hand again. It's like the business person question. You can all do this. It's probably the thing that you can do. But of course if, let's say we were working with golden images, I have six different images and then I have to replace each one of those instances. And if I'm in the cloud, I come up with new IP addresses which means that now my configurations are all worked and I have to fix them up again and that can be a real challenge and a real headache. So you need a platform that can help solve that for you. And of course Chef is a platform that can help solve that for you. But that's probably what's going there from the beginning. So here we are. Chef, let's talk about Chef. What is Chef? Chef is an open source framework that will help you manage your infrastructure as code through a policy based system. Chef is also a company and it is also a community of professionals. So welcome to the Chef community. If you didn't know you were part of the Chef community, guess what you are? You are now also part of the Chef community. So now when you leave here today, if nothing else you'll be able to be a card carrying member of both the Rails community and the Chef community. That's pretty awesome. Don't you think? Yeah. Okay. So Chef has a bunch of components. We're actually not going to spend a whole lot of time on the components because that starts to feel like I'm trying to sell you something. And what I really want to get to is the hands-on part where we write some tasks. So is everybody good with that? If you want to learn more about all these components, come by our Chef Booth. We'll be here tomorrow and Thursday as well. You can also come by the drone. You can come by there. Just saying. The Chef server itself stores your policy and the state of your infrastructure. Let's just, at a very high level, how does Chef work? So on our node, in the machines within our infrastructure, we have a little application called the Chef client. And it runs on a regular basis. And what it does when it executes, it will ask the Chef server, what is my policy? What policy should I follow? The Chef server itself stores all of your policies. So your policies are things like, here's what it means to be an application server. Here's what it means to be a database server. Here's what it means to manage users on that machine. Each one of your nodes within your infrastructure will only use a subset of those policies. We call that subset of the policies the run list. So the server stores a run list that says, oh, you are a web server. So what you need to do is run NTP because we should all synchronize our clocks. That's an important thing that you do when you dev want. We should manage some users and you should become a web server. Those policies come down to the node and the node will bring itself in line with those policies. Which really means it will inspect those policies and say, am I following the policy? If I'm not following the policy, I'll take the steps that are necessary to come in line with that policy. Now, so that's the policy half of storing policy and state within your Chef server. So when we talk about state, the Chef server itself also has a searchable index of data about every node within your infrastructure. So the Chef server has all of that information for you. And this can be extremely helpful in a scenario where say you're doing some load balancing. So here in this example I have HAProxy as my load balancer. And if you've never used HAProxy it doesn't really matter. It's a load balancer, requests come in and it forwards requests back to application servers that sit behind it. But when you're configuring your HAProxy or any other load balancer, you have to tell your load balancer who can receive traffic requests that you get. So who can you send those requests back to? Well there's a collection of web servers. The problem is your infrastructure is dynamic. It's elastic. Over time you're adding and removing web servers. So when you sit down to write the policy for your load balancer, you don't know exactly which host it's going to have behind it. So with Chef, because it has a view of your entire infrastructure, the HAProxy server can actually ask the Chef server who are the web servers in my environment. The Chef server will consult its search index and it will hold back a list of web servers that sit within this environment. Send that down to the HAProxy server who then can rewrite its own configuration file such that it's pointing to all of the application servers that sit behind it. And then the next time HAProxy runs the Chef client, it will go through the same process. Who are the web servers now? It's like a young kid who keeps asking the same question, are we there yet? Are we there yet? Are we there yet? Who are the web servers now? Who are the web servers now? And it will continually update its configurations. So let's talk about what questions you have before I talk about anything else. So what questions do you have? Is this all pretty clear? How many of you do everything that I said already when you walked in the room? It's cool. You can raise your hand. Don't be shy. Good. So maybe you learned one thing or two or maybe more. That's awesome. Let's talk about building up your policy. So how do we actually take Chef and use it? What does that look like? What does the Chef put a look like? And it's very core, this sort of nugget of Chef is this thing that we call a resource. So resources within Chef are pieces of your system. It's a piece of the system and it's desired state. So let's just look at some code and some examples of what that is. It might be a package that should be installed or a service that should be running and configured to restart on reboot of the machine. It should be a file that should be managed, like this file should live here, it should have this contents, this ownership, and so forth. Sometimes you're in cron jobs on your system. You should manage that. Basically anything on your system that you can change, you should manage as code within the Chef recipes that you're writing. Each one of these different things is a resource. Jam it, it works on Windows. So here you can see a desired state configuration resource or a registry key, not many Linux implementations, distros that I know use a registry and have registry keys like this. So these are resources, a piece of the system and it's desired state. It's those two key bits. So this is a workshop. We're going to do labs. So how many of you want to type some Chef code? You can also lie to me now. I'll be like, yes, what I want to do is type some Chef code. That's awesome. So I'm going to give you a workstation that you can play around with and write some Chef code. And with each one of our labs, we're kind of going to set it up like this. We have a problem in success criteria. So in our first lab, our problem is that our workstation doesn't have a text editor installed on it yet. And you should probably have a text editor if you're going to write some code. So we're going to solve that. And this is where the magic trick comes in. I lied. It's not a trick, but you do need a card if you want to do the labs. So what I'd like you to do if you're participating in the labs, you should go to this URL here and what you will find is it's a gist and it has a bunch of cards. You should look at the back of your card and sort out if your card is a blue card or a yellow card. Those of you that want to argue that it's really orange or it's really amber, it's not really yellow, you're probably right but I call it yellow. And then you have a blue card. If you don't have a card and want to do the labs or just want to take a card home as a souvenir, raise your hand so I can get you a card. It seems like the left half of the room came in late. Didn't get cards. It's cool. Cards. So what I'd like you to do you will need a you'll need something with which you can, well, so first you'll need an internet connection which Grouse Conference has been pretty good about so far. Second what you'll need is something with which you can SSH. If you're on a Mac or a Linux box, you can just use terminal. If you're on a Windows box, hopefully you have putty installed. If not, you should go install putty. I hope everyone has something with which they can SSH with right now. So then what you have to do is you have to find the color of your card and the value and the suit and match it up to an IP address. Once you've done that, you can SSH into the box. The credentials there, your username is going to be chef. The password is not square bracket redacted square bracket. Some of you already sorted that out. The password is chef.io. I didn't want to tell you that in advance because we're putting it on the slides because then you would know what everyone else's password is. Right? Okay, so I'm going to do the same because I'm going to do this lab along with you so that you can see me do it. Let's see. I'm going to go somewhere here. Oh, it's here. I'm going to grab my card. So what I'm giving you is a workstation. What would typically happen is that your machine in front of you, can we see my screen and drag it up or not? That is not the slides. Right. There we go. You can see that now, right? So what I'm going to do, I'm going to SSH to my machine, but I'm not going to do it like that. You should always do what I mean, not necessarily what I type. So I'm going to SSH as chef to that machine. Here's your first DevOps lesson of the day. It's not really a DevOps lesson. It's a system administrator lesson. How do you verify the authenticity of a host? I'm sorry? Yes. You type in the word, the letters Y, E, S and then you get the magic button. This is how you do it. This is how professionals do it, honestly. Remember I told you I'm a community director. So then we asked you for your password. What's your password? No. And see, social engineering is the way that you will crack into any system. I just got you all to tell me your password. Now I'd like you to do something and this honestly is the most important part of the lab that we're going to do for the rest of the afternoon. I'd like you to type in the word touch and then I'd like you to hit the space bar and then I would like you to all as one thing together type in your first name and last name, but not the characters Y-O-U-R-F-I-R-S-T-N-A-M-E-A-N-D-L-A-S-T-N-A-M-E. I want you to type in whatever your first name is. So I, not you, but I am going to say my first name, it's Nathan and my dad spelled it wrong so I'm going to spell it wrong also. And then type in your last name without any spaces and I'm going to do that also. And then the other thing that I didn't tell you before is you should never hit the magic button until either Nell or myself tell you to press the magic button. And of course is the one all the way on the right kind of in the middle. It either says return or enter. You can press the magic button now. So I'd like you to press the magic button. It's okay if you don't care if you do capital. Case sensitivity is like, that's a personal choice. It's totally, totally up to you because down off is also inclusive. So now if you're doing the labs and you have typed this, I'd like you to raise your hand. Okay, now I'd like you to put your hands down. And now if you were doing the labs and you have not yet completed this, I'd like you to raise your hand. Yeah, I do. It's bit.ly-rails-chef-gdd. Did that work for you? bit.ly-ly. I could just show you the whole URL instead of trying to sell it. That would be much nicer of me, wouldn't it? That would feel there. Is that better now? Can we see that? Yeah. While he's sorting this out, what question is going to answer for you? Yes? Were each of these nodes configured with Chef? They were configured with Chef. Indeed. Good question. Configured with a tool we're going to talk about later? Also, yes. I'll even show you how they were configured. At least some of them. I will show you all of my secrets. That would be ridiculous. Studio Works. Yes, Studio Works. So what he really asked there, you couldn't hear his question. Let me repeat the question. What did you do to these boxes? What power do I have here? So let me tell you about these boxes. It's a vanilla, I don't know, it's centos, I think. Is it centos? See how much I know? It's a vanilla centos AMI, which is an Amazon machine image. So it's a machine that's running off on EC2 right now. I've done a couple of things to it. I gave you sudo access, passwordless sudo access, which is different than sudo list password access, as it turns out. So you have passwordless sudo access. I've installed a thing called the Chef DK, or the Chef Development Kit. I've done some other things too, but I won't tell you what those are yet. The username Chef, I allowed that user to log in with SSH, sorry, with username and password, which you would never, ever, ever actually do in real life, right? Why wouldn't you ever log into a machine with a username and password? What would you use instead? Yeah, you could use your SSH keys. And those of you that said you weren't sysadmins before, you were clearly lying. Have you logged in? Have you touched your name? Have you touched yourself? No, that's not the right question. Okay, so now we've all done this. We've all done this. Now comes the really important part. I'd like you to type in the characters L and S all together, and then I'd like you to hit the magic key. And you should see three things here. A file called git.rb, you should see a directory called RailsConf2015-chef, and you should see your name and your name only. If you see someone else's name, someone else said, oh, he just gave you the username and password of all of these machines, and I have su access, I'm going to mess with one of my fellow attendees. So you should find that person and maybe get him to perform a machine. All right, everybody's good. We'll have just our own machine, right? All right, good. So as yet, we've done nothing with Chef. But what we have done is set up a problem for you. So a text editor is not installed on this workstation. We're treating this machine as if it were your developer's workstation, but we typically need a computer that's currently on your lap. What's your favorite text editor from a command line? What's that? Vim. Vim. Vim. What's that? Greater than. Greater than? Okay. That's what I said. That's cool. So Vim sounds like the clear winner. We don't have any Emacs users in the room? Actually, you do. Yeah. Do you know how you'll know if someone's an Emacs user? They'll tell you. Okay, so anyhow, Vim. Vim seems like a good thing that we should do. So what we're going to do here is we're going to install the text editor, Vim. But of course, we're not going to use the standard package manager to do so because that wouldn't be very Chef-y. We're going to use Chef to do that. So if we think about Chef, it's all about resources. A resource is a thing and it's a desired state. So we have this thing, a package. And we want to have this package installed. So we're going to use the Chef tool to do this. And this is the fastest way to introduce you to some Chef's tooling. It's a tool called ChefApply. So you're going to type this along with me. First, we're going to use sudo. We're going to type Chef-apply. Make sure you type everything I mean to type, not necessarily what I actually type. Then we're going to do a dash e, which is difficult for you to see because of screen resolution. So let me see if I can fix this a little bit. That looks even better, right? All right. At least it's all on the screen. But he went for me. All right. So we have sudo ChefApply dash e and then another space. And we're going to do a double quote. And we're going to say package and then a space and then a single quote and then vim. And then close off that single quote and close off the double quote. And if you've typed this exactly as I have, you can then use the magic key. If you prefer emacs, you can replace the vim bit with emacs. If you prefer nano, you can replace the vim bit with nano. If you prefer greater than, maybe the package repository has that. I don't know, put the right package name in there. And then we're going to hit enter. And what happens when you hit enter? No, nothing. Look, nothing is happening. What's going on? It's taking a minute here because in addition to giving you free workstations to play with, they give you really small workstations to play with. So they have no power. So it takes a minute. You can see that Chef has sorted out that this machine is running CentOS. So it's going to use the young package manager and it's going to install a package called vim. If we were to read the docs, like really? Who does that? We would see that the package has a default desired state of install. So really, if we were to say this code to Chef in English, what we would say is our policy is that the package named vim should be installed. And so Chef is going through the system right now and verifying that that's the case. How many of you did a which vim first to make sure that vim wasn't actually here? Yeah, those of you that are like, yeah, I don't believe in anything that you'd say anymore. I've been lying all afternoon. So it goes through and it does a young install in the background. The nice thing is I don't need to know, and I already admitted that I didn't. If this was a CentOS machine or an Ubuntu machine, Chef sorted that out for me. And based on that, it knew which package manager used to install that package. So it used young because this is a CentOS machine. If this had been an Ubuntu machine, what would it have been used? It would have done an act to get under the covers. But I don't need to worry about that. It just distracts all those details, all that nutia away from me. And now it's installed the package for me. So which vim is going to show me that, in fact, there is a vim installed. Now, as a thought exercise, what would happen if you hit the up arrow until it said su-chef-apply-e package vim and then hit the magic key again? What would happen in that case? It would tell you that it's already up to date. That's exactly right. See, you guys, you're all so smart. We have slides. We're walking through all this. I'll tell you some of the slides. Let's get all caught up here. Yeah, so if we were to run it again, and you can run it again, I'm not going to just go back to the slides. What it would tell you is that the package vim is already installed. It's up to date. It doesn't need to do anything with this. In Chef terminology, we use really big mathy words to describe this. And you should forget those really big mathy words. I'm going to tell you how Chef works. It works in what I call a test and repair process. So all of our resources follow test and repair model. So when we see package vim, Chef executes a test. Is vim installed? Is that package installed on the system? If it is, Chef will stop and take no further action with that resource. If it's not installed, then the system is out of policy. It's broken. It needs to be repaired. So Chef will install it, and that's the repair half of our test and repair cycle. If you understand this about Chef, you'll understand everything that you actually need to know about Chef. Resources work in a test and repair fashion. Everything about Chef is abstractions and things to make it easier for you to work with resources. This is the core bit of Chef. Understanding that resources are a thing, you describe the resource, the thing that you want, and its desired state. So in this case, the default desired state for a package is that it is installed. What questions can I answer for you? Yes? We're working with a Chef client. So what you're working on, Chef client is a separate application. So we ran ChefApply. ChefApply is a little tool that comes with the Chef development kit. It's an easy way for you to explore resources. We haven't really applied a full policy. We just applied a singular resource here. So because this is a workstation, really I think about that as my machine, your laptop, where you're going to write the code. It's not necessarily a machine that's eventually going to run Chef client to be part of your infrastructure. Great. Thank you for the question. Yeah, words. There are lots and lots of resources that come with Chef. And you'll find them all on the Doc site. Here are just some examples of them. And also you can create your own custom resources, which we won't get to today, because we do have a limited amount of time, but it's totally something that you can do. So we have this thing in Chef called recipes. A recipe is really just a collection of resources. So we just applied a singular resource, a package, but usually you're going to apply multiple resources in the same policy. When you put multiple resources together in a file, we call that a recipe. As you might guess, we take our recipes and we put them into cookbooks, and those become our distribution unit. You can think of a cookbook as something similar to a gem, because it might help other stuff. And with that, I'm going to switch over and let Nell take over for testing the infrastructure and start writing through some tests. I think we might have a question in back there. Yes, awesome. Yeah, it seems like it's overlapping my mind in a real between nodes and resources. Sure, so let me help, let me try to help resolve that overlap. A node is a machine within your infrastructure. And with my machine, I'll show you the results, because it might be a physical machine, it might be a cloud instance, like running off on EC2 or digital ocean. So it's the full machine. A resource is just a thing on that machine that you can manipulate. Okay, perfect. Okay, great. While we get switched over here, what other questions can I answer for you? Alright, as Nell takes us through some of these labs, it's going to be my job to run around and help you when you need help. So when you need help, make sure that I know. You can raise your hand, you can shout out my name, whatever it takes to get my attention. The other thing I will say is that when you have errors, it's actually better because you can learn more from errors and mistakes than you touched everything exactly the way Nell told you and it worked. We kind of know that's going to happen. So if you have errors, we can learn more together. Now, over to you. Alright, let's talk about tests for testing our infrastructure. So our process so far has been to write a policy, a policy which would install an editor, then apply the policy on our workstation, then verify the policy manually. Now, this is fine for a simple case, but just like Ruby and Rails code, testing our infrastructure code manually quickly becomes untenable. I don't think I need to spend too much time preaching to this crowd about the benefits of automated tests. Our application code, having tests around our infrastructure code will let us speed up those feedback loops and let us push to production with confidence. So the ultimate purpose of tests is to tell us whether or not our code is safe to deploy. With our Chef code, there are four main questions we need to answer before we can deploy to production. Did the Chef client complete successfully? Did the recipe put the node in the desired state? Are the resources properly defined and does the code follow our style guide? So let's jump into answering these questions. We're going to use a very simple scenario which will let us explore the many facets of test driving infrastructure code. And that actually is the alarm, which means it is time for the 10-minute break. Go ahead and take a break for 10 minutes in between part one and part two. Please do come back, and I will take you through an exercise where you will write code, test driving, actual Chef code. So we will see you in 10 minutes. Alright, let's bring it back. We'll definitely save time for questions at the end. So we're going to use a very simple scenario to test drive our infrastructure following that test-first, test-driven model. So our scenario is that we want a simple Rails app deployed on a server. We only have time to build some of it in today's workshop, but you will leave the workshop today before working code, and all of our instructions and slides are on that github repo, and you're welcome to go to that at your leisure and work through it. So make sure you're back on your workstation, and let's go ahead and create a new Chef repo for your Chef project. So go ahead and write Chef repo, Chef repo. And I'm going to leave my code up on the slides just because there's some extensive code. I also have what I call someone is watching me typing syndrome, which means it's really hard for me to run the code, and he's going to let me know whenever the plans are completed. So now CD into our cookbook let's first CD into that Chef repo, and then CD into our cookbooks directory. This is where we're going to do the majority of our work. So CD Chef repo slash cookbook should get you there. And next we're going to generate an actual cookbook. You do this by letting Chef generate cookbook cookbook slash Apache. This cookbook will allow us to install and configure Apache. So if you use Chef generate cookbook, the Chef DK has some magic built into it. It's a lot like using the Rails Generates. And what you don't do is it'll generate some scaffolding and maybe start it and put it together in your cookbook. So let's go ahead and run that. So just one quick thing, if you're in that cookbook directory, should this be Chef generate cookbook Apache? Apologies. So under the cookbook directory do Chef generate cookbook Apache? For every one second for that. The first question we need to answer is did our Chef client complete successfully? Fortunately, this is the easiest thing for us to test. In order to do this, we need to verify that one, we have a place to store the cookbook artifact. Two, that we have a Chef client with access to that cookbook. And three, that we have a target server which is running the same OS as our production server, either is running or will be running. So to do this, we use this tool called Test Kitchen. Test Kitchen is a delightful tool that allows us to spin up a VM or container, configure it with our Chef code, then verify that that VM or container is in the state we expected to be in after our Chef run is complete. It's created and maintained by Fletcher Nicholl, who also works at Chef. So one of the nicest things about Test Kitchen is that we can test our Chef code on multiple operating systems at the same time. I can test whether the same cookbook will work on both Ubuntu 12.04 and CentOS 6.4. I can then define which recipes in the cookbook I want to test and test it on both those platforms. If I have multiple recipes within the same cookbook, let's say I have a default recipe in my Apache cookbook and I also have a recipe for configuring SSL with Apache, I can test them on both platforms using Test Kitchen. I have another operating system and one of those same suites on that operating system. Isn't this delightful? I'm not. So we can figure what operating systems and what testing suites we want to test using our .kitchen.yaml file within your cookbook. So go ahead and open up the .kitchen.yaml file within your Apache cookbook directory that you just created. And then there's that file. Let's go through it step by step about what it does. So the first thing we do is we declare what driver we want to use. This defines where Test Kitchen will tend to spin up that instance to test our cookbooks. Now this can be vagrant. It's a very common one if you want to run everything locally. It can be a cloud provider. I tend to use Digital Ocean just because it's nice and fast. Then next is our provisioner. This is how we declare how they're going to simulate a Chef Client one on that Test Kitchen instance. When we use Test Kitchen, those Test Kitchen instances are not connected to an actual Chef server. So we can use either Chef Zero as we do here to simulate that Chef Client, or we can use Chef Solo, which is a way of provisioning Chef nodes without using a Chef server. So I like to use Chef Zero because it runs in memory on my Test Kitchen instance and it runs nice and fast. So next thing is platforms. Here we define what OSes we want to test our Chef code on. And then finally we come to Sweets. Sweets are ways to group recipes to test. The run list of a sweep defines which recipes will be applied to our Test Kitchen instance when we run that sweep. Now attributes allow us to add in values we would expect to find on the server, but can't know until the server is actually spun up. An attribute I like to use a lot is IP address. When I'm spinning up a server or spinning the Test Kitchen instance on a cloud provider, chances are I'm not going to know what the IP address is going to be until I actually spin up that server. So I provide a mock IP address as an attribute and that way I can test that my Chef code finds that IP address and handles it in the ways that I expected to. So a Test Suite can run multiple recipes. So we can have our default sweep which would run our default recipe. And we can have another sweep which would test a recipe called SSL which would configure SSL for catching. Finally if I want to add another operating system it's as simple as adding it in under platforms. Let's say I want to test my code on Ubuntu 14.04 along with CentOS 6.4 and Ubuntu 12.04. Things behave a little bit differently and certain things do on Ubuntu 14.04 versus Ubuntu 12.04, so it's good to know that my cookbooks work on both. So I'm going to pause here for a second. Is everyone still with me? No, you don't need to type. If you open up, are you in your actually cookbook directory? Okay, did you open up .kitchen.yaml? Yeah. It doesn't have anything in it? It does, but you've got the edit now. Alright. Hold off on typing for right now. We'll be typing it just a little bit. I'm sorry, I wasted some keystrokes for you. Oh, no, no, no, wait. Alright, so make sure you're still in your patsy cookbook directory. You should be if you just look at your .kitchen.yaml file. And now we're going to edit .kitchen.yaml. So we're going to first update our driver. All the cool kids are using Docker these days. So we're going to be cool. We're going to use Docker to run our test kitchen instance. And then we're going to leave the provisioner of Chef zero so it stays nice and fast. The test this on is a bunch of 12.04. So go ahead and take out the sento's line and a bunch of 14.04 if you have it in there. Finally, we're going to define our sweets. So this should already be in there. We're just going to run the default suite at first and we're going to run the patsy recipe within that default suite. So I'll give everyone a couple of seconds to get that typed in. Is it necessary to have that parenting of the attributes? It's auto-generated. I don't think it's necessary to run to the cookbook though. So you can take it out if you want to. All right, so raise your hand if you're still typing. Raise your foot if you're still typing. It's our only way to ensure that you stop typing. Raise both hands if you're still typing. All right, so go ahead and save and quit that .kitchen.yaml file. I'm going to command kitchen list and go ahead and hit that magic enter button. Now when I run kitchen list, it shows all the test kitchens we have already configured. In this case, it's only one. We're only running our cookbook on one operating system and only using one recipe. If I were to add in back in a bunch of 14.04 or CentOS or 6.4 or whatever it was, you would see that in this list. It would show each of the possible combinations you could run, the OS and the testing suite. So in that, we're testing suite in .kitchen.yaml and it looks like we have a question. Yes, sir? I'm getting an error that homechefchefrefo.kitchen.yaml is not exist. Okay, I'll go ahead and send Nathan over to you. Is anyone else getting that? You get it once you're in the directory. All right, you get it once you're in the directory. All right, so we'll define it in our .kitchen.yaml but we still have to actually create the Docker container to run our tests in too. So you do this by running the kitchen.create command. And this is going to take a few minutes so I'm going to go ahead and oh, Nathan's already started running it and let's give it a little time to run. Are there any questions while this is running? Yes, sir? Kitchen.create will be surrounding, for example, if you specify a program provider it will run a program. Exactly. Yep, it'll run bigger boxes. In the meantime, we have one recipe on our run list right now. I'd like to show you the community MySQL cookbook. Four recipes that are in that run list. First kind of goes, and you don't need to follow along with this if you don't want to. There's a supermarket. It's here for MySQL. And let's look for their source. It runs on. Let's bump that up a little bit. But they want to make sure that MySQL cookbook will operate correctly on. And here are all the sweets. And it just keeps going and going and drawing. So your .kitchen file is the master file for your tests, essentially. All right, throw back the slides. Nathan, has your Kitchen.create completed? My Kitchen.create is not yet complete. Oh, right? It's on step nine. Okay. The doctor tells you which steps it's on. Just so that we all know sort of if I'm faster or slower than you. Kitchen is finished. My Kitchen is done. Is that a question there? Yes, sir. Does the doctor tell you how many steps there will be? The question is, does the doctor tell you how many steps there will be? I do not know that answer. Does anyone know that answer here? Yes, it does. When it says Kitchen is finished, you use your scroll bar to scroll back up and you look at the last step. That's the way to do it. All right, now that we've run Kitchen.create. Yes. Let's take a look at a little visual now that our Kitchen is up. So, we have our workstation. That's what you're SSHed into right now. And we're running a Docker container with our kitchen. That's a sample operating system that we're going to test our Chef cookbook on. And that's running within the workstation itself. One of the nice things about Docker is running containers within your own workstation. Yes, sir. If you install Chef Decay, that sets everything up for you. I don't think Docker comes with Chef Decay. You can install that separately. If you want to use Vagrant, it will set up everything so you can use Vagrant. But the kitchen... Test Kitchen comes with... Test Kitchen does come with Chef Decay. All right, so if your Kitchen.create. is complete, one of the nicest things about Test Kitchen is that we can log in to that container we're running and run commands for that container. So go ahead and run Kitchen Login from your workstation. And when you're prompted for a password, use the word Kitchen. Pick one. Oh, you would specify the instance. When there's only one, you only have to do Kitchen Login. When there's more than one, you have to do Kitchen Login. I want to work together for this test suite. So that's where Kitchen List comes in very handy. All right. Yes, sir. Do you pre-install Docker? Docker needs to be installed separately from the Chef Decay. All right, so now we're in that Docker container on our workstation. Raise your hand if you're not seeing this command prompt. All right, looks good. Oh, certainly. So let's demonstrate this visually. Again, we have our workstation with our kitchen container running Kitchen inside our workstation. And what we did was we ran Kitchen Login from our workstation. Kitchen Login SSHed into our kitchen container. And now we're able to run commands from within that kitchen container. So we still need to verify that our Chef Client runs successfully. Remember that we need a target server running the same OSS production and a Chef Client with access to that cookbook. We now have that target server setup. We just need a Chef Client with access to that cookbook. So in the first lab we're going to do a problem with our workstation. So the first lab we're going to do our problem for this lab is that we have not applied our policy to our test environment. So test criteria means the default Apache recipe will be applied to this test kitchen environment. And by environment, I mean the test kitchen container we can figure. So go ahead and run exit to get out of your test kitchen and make sure you're still in your Apache cookbooks directory. Step is we're going to run the command Kitchen Converge. Go ahead and put that command in button. Now Kitchen Converge will install Chef if it's not already installed on that kitchen. Whoops, I went up a slide rather than there. Test kitchen container. It will install Chef. It will also upload all the cookbooks and then apply the run list we can figure in our .kitchen.yaml file. So I'm going to go ahead and let that run. Nathan, let me know when that's complete. Are there any questions while that's running? So our couple of hands trying to go halfway up. Why is this command isn't around when you are working in Kitchen Converge? Why is what not around when you're in Kitchen Converge? Why is this last command isn't around when you offer us Kitchen Converge? It's around, but oh, why doesn't it do it right when you hit Kitchen Converge? Because sometimes people want to just spin it up or create the kitchen first before they install their Chef pressing piece on it. It's a little quicker that way. It only doesn't create command. You could run, create, and then Converge. And later on I'll show you the command where you can run all the Kitchen Commands at the same time. When our Kitchen Converge runs successfully we can answer that first question. The Chef client run did indeed complete. So congratulations. You just completed your first test-driven Chef run without running any tests or Chef code at all. So now let's move on to the next question. We need to verify whether Chef we're in our desired state. So we could do this through manual manually testing our container. So go ahead and run Kitchen Login again for Kitchen. And that'll get us into our Test Kitchen instance. So one manual way to test whether Apache is working is to run Curl. So within your Test Kitchen instance, go ahead and run that command, hit the Manager Venture button, and get back to Successful Response that would be when Apache was running. You can't connect to the host, which tells us that Apache was running. The visual just reminds you where you are. It's sometimes easy when you're working with containers to lose sense of where you are, whether you're on the workstation or container. Remember, we're working that Curl command from inside our Test Kitchen container. So our next lab is we need to manually verify the state of the test node. Well the problem is that manually verifying the state of the test node is tedious and error prone. Our success criteria for this lab is that the end state of the node will be able to be automatically tested. Yes, sir. Is there a way to get that to work correctly? Not yet. We're going to make it work. Okay, it's not that we did it incorrectly. No, it's not that we did it incorrectly. It's setting up a failure first. Automated tests we're going to use a tool called ServerSpec. ServerSpec allows you to write tests to verify the state of your servers. And it's not just limited to Chef. You can use it if you're using Puppet, if you're using Ansible, Salt, just about whatever have you. And just like Chef, it defines resource types that we test whether that resource is in the state we expect it to be in. It works extremely well in the Test Kitchen and if you want to get an idea of all the sorts of resource types you can use, go ahead and go to that URL, serverspec.org. So if you're still in your kitchen instance, go ahead and exit out of it which will take you back to your workstation and make extra sure you're still in your kitchen. And when we ran Chef Generator Cookbook, Chef CK automatically created a server spec test file for our default recipe. So go ahead and open it. It's test slash integration slash default slash server spec slash default underscore spec dot rv. It's a bit of a mouthful. Absolutely. Just making sure that you're in your kitchen cookbook directory. So when you open this file, if you're an RSpec user, it's going to be very familiar. Raise your hand if you don't have this file open yet. Yes, sir. Our server spec is automatically installed in DKS. So let's take a closer look at that syntax. Just like RSpec, we're using a describe lock to define the subject of the test. In our case, it will be the default recipe of our patchy cookbook. It blocks the scribe, we expect that subject to do or be doing and we use that same expect syntax that RSpec uses, later versions of RSpec at least, to define what we expect the results of our code to be. So let's add in a real test. Yes, sir, agree. Does the describe string need to match the cookbook number? You know I don't know on that actually, but I can find out. It doesn't need to match, but when we put in the generator you can totally change it. Alright, so let's add in a real spec. The one that describes the default recipe of our patchy cookbook. Let's add in a test to ensure that our recipe is awesome. When our recipe is awesome, we are awesome. So go ahead and add that in. We're using it is awesome, that's our it block, and we're expecting true to equal true. I have a couple more seconds to get that in. Go ahead and move on and close that file. Then from your workstation, run the command kitchen verify. Actual test on our kitchen container. Now this is going to take a little bit to run at first, but ultimately we should see a success. That our recipe is indeed awesome. So I'm going to give that just a second to work to run through. It always takes a little longer the first time we run the test. Alright. You have an exception? It's because we have a special technology. Can you open that test again? Absolutely. Give me a thumbs up when you're ready. Thanks. One last question. On ensemble. Oh thank you. Any questions? All right. Let's go ahead and move forward. So we read Kitchen Verify and saw our test pass, ensuring that our recipe is indeed awesome, therefore we are awesome. But this isn't a real test. Science Interaction Exercise. What would you test to make sure that a patchy is running on your test kitchen instance? Curl of Arrows, that's one. There's a few others. PS, that'll work. Which patchy? Check the package with your gene package? Yes, sir, the blue. Should I check the process file, the pit file? Excellent. All right, so for now we're going to add a test to make sure a patchy is installed. Open up your service file and add in this test at the bottom, which is checking whether a patchy 2 is installed. We're using an expect syntax, just saying that we expect the package of patchy 2 to be installed. Silver spec takes care of the rest for us. Give everyone a few more seconds to get that in. Here in less keyboard clicking. If you'd like me to wait for going on, go ahead and raise your hand. Was there a hand in the back? So we're going to Kitchen Verify again and this time it'll run much faster and we'll also see a failure. Telling us that a patchy 2 package is not installed. And this is good. It's always good to see a failure when we first run a test. So you probably already know this, but just like our application code, we test our infrastructure code by writing that test, watching it fail, writing just enough code to make it pass, then repeating. So we've written a test and watched it fail. Now let's go ahead and make it pass. And to do this with Chef, it only takes one line of code. Go ahead and open up your default recipe. If you're within your patchy code directory, that's recipe slash default dot rb. And then that one line package, we're defining our Chef resource. Then in quotation marks, a patchy 2. Right after that, we're going to converge the node again. So go ahead and run Kitchen Converge. Then after you Converge, Converge Supply and that Chef Recipes are known. Then you run Kitchen Verify and watch that spec pass. You can do Kitchen List. I don't know if I'll tell you whether you haven't converged yet from your previous converged. But if you haven't converged at all, you'll be able to see that way. Because you're using a full-size keyboard instead of a phone. We're starting to run short on time and we want to make sure we leave plenty of time for Q and A. So I'm going to go ahead and speed through the next slides. Let's go ahead and skip ahead. All of this is available in the GitHub repo we showed. So your Kitchen workflow is doing Kitchen Create, Kitchen Converge, Kitchen Verify to actually run the test. Then Kitchen Destroy which will destroy that Test Kitchen Instance or Container. I'm going to speed through this. I'm going to skip ahead. Test Kitchen is wonderful and Test Kitchen Server Spec are wonderful for integration style tests. But the problem is, you can see you have to converge your changes, rerun those on the node, spin out the node if you haven't spun it off yet, just to get back your feedback. With Docker, this takes about two to three minutes. With a Cloud provider, it will probably take even longer. So there's another testing framework we can use and this one specifically for unit tests. It's called Chef Spec. This allows you to test before you converge. So if I were to write my code and just have a typo in that code, it would fail and let me know about that typo before I did the actual converge. We're not going to run through that exercise, but feel free to check out the GitHub repo and run through that on your own time. And there's one more framework I want to introduce you to. Chef Spec is just RSpec, so you run it just using RSpec Spec. And that is food critic. So sometimes after we verify our integration test work, our unit test work, we also want to verify whether it follows our style guidelines, whether we're using any deprecated features of Chef or Ruby, and whether there's any syntax errors in it. So food critic is what's called a limiting tool. It doesn't actually execute the code, but it does go through the code looking for errors. So a way we would test this is changing our package name to be in a string called package name saying that to Apache 2, then trying to interpolate that string into another string when we actually declare the package resource. Food critic would tell us, hey, the string is for a lotion. It doesn't actually break anything, but it's not required and it makes the code unnecessarily complex. So it's a nice layer on top of your code. Wait a check that front layer. And with that, I'm going to skip ahead again. What we would have done is install Ruby on the system then configure that full Rails application. Please take a look at that GitHub repo and configure it on your own time. And now it's time to wrap up and I'll sort it back to Nathan. All right. Thanks, Nel. What I want to do, I have some wrap-up slides that I'm not going to use. What I'm going to do instead is just jump over to that GitHub repo really quick and give you the two-minute descent tour of that repo. It's not on there. It's not on there. No, it's not on there. Oh, right. But that's not on there. Right. Okay, so this repo has a bunch of branches here. As you can see, you can probably see it better than I can. But it basically walks you through these. Each branch is kind of built on the previous branch. And at the end, you'll have the full working application. So there's a lab called Widget World Ruby that has a cookbook with all of its unit tests and integration tests that will install Ruby 2.2 on that particular machine. From there, you can do a lot of postgres, passenger, get a pack to install, and then finally a working Rails application. All of these work pretty well. With a cloud provider, they won't all work with Docker as your test kitchen, though. When you install postgres in the default Docker containers, it's not going to work so well. So I'm actually using DigitalOcean locally. And when I go through into the labs Widget World application, what you'll see there is that there's one... Oh, this is fun to browse. It's all good. I can deal with it. So there's this Widget World application that will actually pull a Rails app from GitHub, put it into the machine, run the DB Migrate, and at the end, you'll have that application that you can pull up, which is pretty cool. What I'm going to do is look at my screen and type the URL into the R over there while you ask questions before we wrap up. Actually, before you start asking questions, Nell and I are going to host a Birds of my Feather session at 8 o'clock. If you want to talk more about Chef or more about DevOps, please join us for that. It's in a room. Which room, you might ask? I might ask the same question, but if you look at the Birds of my Feather board... It's AB. AB. Nell knows. She's smarter than me. That's awesome. So the Birds of my Feather board is out there, so you should check that. Please do. Just anything. We've got a couple more minutes left. Yes, ma'am? I like... I can answer that personally. I personally like Chef better than Pop It because Chef takes a developer view. I came from a development background. System and did not make any sense to me until I found Chef because I could develop it or I could create my Chef recipes with the same Ruby code that I used and in the same kind of developer mindset. Pop It very much comes from the system and mindset. It's usable, but to me it's not intuitive. Also a nice thing, even Pop It is also written in Ruby. You write Chef recipes in actual Ruby, so something like a block or a form or in each loop that will look like an actual Ruby in each loop. As far as Ansible goes, Ansible works well for small deploys. I have nothing against it. Chef works very well at scale. That's why we have Facebook as a customer because they run tens of thousands of servers and other customers with massive infrastructures. It's a great way to have that Chef server and be able to deploy small changes out to your entire fleet. Testing frameworks. You can use them with the other frameworks. They're really a short of a Chef, which is nice. Yes, sir. I want to talk about in way the impression of a lot of tools that I already knew and didn't really have it in my application. It looks to me like things have settled down. There's sort of like one official line to have to do all these things. Well, you will find any sort of framework of client people who do it a different way. But most commonly, our stuff at Chef, we use server spec and Chef spec, and test kitchen most commonly. Some of them are not open source, but we try to maintain contact with the open source creators and make sure that we're working together on it so we don't break their stuff and break ourselves. Yeah? A server spec is created by a Japanese developer. I don't remember his name off the top of my head, but if you go to the server spec website, you can probably find out. Any other questions? Yes? How do you remove resources? Especially if you remove resources and resources. Right. I will turn to Nathan for that one. Sure. So if you want to remove a resource the way that Chef works is it manages the resources in your policy, and it will make sure that those resources follow the desired state of each one of those resources as laid out in your policy. So for example, if we were to change our Apache recipe and just take the package or Apache tune out of that recipe, Chef will stop managing that resource, which means that if it's on the system it's going to be on the system. Chef isn't going to do anything with it. Chef, remember, resources also have a desired state. So your desired state could be that that resource does not exist on that system or is removed from that system. So while the package has a default desired state of install, it also has a desired state that you can specify. It's called an action, and that would be to remove it. So you could say that the action is to remove it. And then Chef will follow the same test and repair. Is the Apache package installed? If so, remove it. If not, our following policy will continue on. So we are at time. I want to respect everyone's time. However, Nathan, there's nothing in here after us. We're happy to hang around a little bit if you have more questions. We'll also be at the Chef booth tomorrow. Both of us, if you want to talk Chef, talk Rails with Chef, whatever you like. I like bunnies. You can talk about bunnies with me. That's right. We have Chef stickers and little info cards up here which you can also get at the booth tomorrow. And thanks very much. Thank you very much.