 Okay, cool. Let's get started for this last talk of the day Thanks for coming. I know it's the last one. So, you know, it's pretty sparse audience I'll try to keep it entertaining I have very few slides and then I actually want to go into demos I want to show you, you know a few tools and what you can do on a cloud stack cloud So my name is Sebastian Gwassgen. I'm a PMC member for Apache cloud stack So I'm a I'm a committer and I'm part of the the management committee for for Apache cloud stack And we're gonna talk about clients and tools, you know, of course when you look at infrastructure as a service the landscape in open source You got different solution to build that cloud, you know private public So, you know open Nebula from Spain eucalyptus, which was really the first one to try to clone EC2 There was a talk about GANETI in the you know in the next room earlier Open stack, of course, you know the the foundation and then cloud stack which is an Apache software, it's a top-level project originally it was written by a startup called cloud.com and it was acquired by Citrix in 2011 Then in April 2012 they donated the software to the to the Apache software foundation and then we became a top level project in April When you join Apache, you know, if you don't know about the Apache processes first when there is a donation you go through incubation And then the incubation period, you know, you have mentors who are established member of the Apache software foundation And they help you grow a community Make sure that that software and that community are going to be sustainable So we went through the incubation process, you know In addition to building the the community, you also have to make sure that all your licenses is in order and things like this So we got out of the incubator in March of this year. So it's been You know six to seven months that we've been a top-level project So that's your that your choice in open source if you are trying to build a cloud whether it's public or a private cloud But really, you know, I don't want to talk about, you know, what's really inside cloud stack I'm making the assumption that okay, you take cloud stack and you build that cloud Okay, and then the question is really okay, you have a cloud know what okay, you can you know, all of those system You can start virtual machine start stop, you know, suspend you name it That's the basic functionality if we go into all the different features, you know Of course you support different types of storage different types of networking networking services and so on But really, you know, my question is you have a cloud now. What do you do? Okay? So that's the point of going through a few tools and showing you a few clients that you have with cloud stack showing you the the variety of tools that you have and And then show you a few a few demos You know, I do have to to give a quick intro about cloud stack who uses cloud stack, you know out of the six in the room Okay, six of them So, you know, if you don't like the term cloud or infrastructure as a service Term, it's really a data center orchestrator. Okay, so you install the app In class that works as a management server. So you have a single note. That's a management server You can you can set up, you know, a high availability management servers load balancing and then replication of the database But basically it's a single management server Java application running with Tomcat and you have a my sequel database Then you need a form of hypervisors and there we are hypervisor agnostic So you can pick, you know flavors of Zen KVM via more area sex We also do bare metal and then recently we have the the LXC containers In there and I'm sure we'll get Docker, you know, pretty soon So it's really a data center orchestrator You build that infrastructure and when you run the management server you get, you know Services features to manage the virtual machine hypervisor agnostic you get a System to manage storage not only the the image the images But also the data volumes if you want to to attach data volumes to your virtual machine and you get network services Those are really the three main characteristics of any yes, you know Manage the compute the storage and the network the network can be very You know involved and I'll I'll talk a little bit about this. We have two types of networking We have basic networking and advance in advance networking We isolate all the users in VLANs and in basic, you know, all the users share the same Broadcast domain, okay, so isolation between them needs to be done differently and we do it with security groups Just like Amazon Now in networking you can also talk to physical devices load f5 load balancer juniper SRX firewall net scaler you name it When you install cloud stack you also get an image management, of course You get an identity management system with an LDAP integration and so on you get a default dashboard And I'm gonna show you a snapshot You get usage monitoring Usage mirroring sorry for no billing and so on and you get an API of course all the system They expose an API and that's the you know all the tools that I'm going to show you are using this API And that's how you program that crowd We have a native API which is really cloud stack specific, but we also have a mapping with the Amazon EC2 API When you install the management server you get the default dashboard and that's that's the example. That's just in Chinese We worked a lot on translation and You know this is just for fun to show it in in Chinese, but that's the default is just JavaScript application You manage the infrastructure the images the networking the service offering the instance types Everything can be done through the through the UI But basically the UI is making use of that native cloud stack API. It's it's fairly involved There are over 300 methods You know in that in that API It can be as simple as all the you know the pleasure to machine start stop you name it create VPN create load balancers You know manage the infrastructure create users you name it. So it's a it's fairly involved It's not a it's not a restful API in the very strict sense. It's a HTTP query API that can return XML or JSON, okay So basically once you get the management server you can look at you can go through the dashboard Do a few basic calls and then if you run fire bug for example, you'll you'll have a clue of the the way the API Works and you have it here. So basically we pass the method as a command Key and then you pass key value pairs as parameters. Okay, so pretty pretty standard things So now that you have, you know, you've deployed you've created that infrastructure with cloud stack You have that cloud, you know that you have this API. So what do you do? Okay? You have a cloud now what? So of course, you know some of the Clients or users even can still use the default dashboard But the the clients that you can find out there are even more powerful So what I wanted to do in the talk is go through, you know a few of the clients and then go into the demos The basic client is cloud monkey. That's our default Commonline interface that's really for sysadmin who want to manage the cloud Users can also use that that interface, but it's very It's basic, but it's 100% API complete. So there's a full coverage of the API It's as simple to install. It's as simple as pip install cloud monkey And then you set up, you know, your endpoint you set up your API key secret key And then you start talking to your cloud issuing all the all the API commands that you know You wish to do so deploy virtual machine if you're an administrator of the cloud You can also add host change a network You name it This is Python But of course there are different languages depending on what you're looking for and and what's your preference, you know in language So you'll find a ruby client that is now fairly complete They're also, you know, even pearl client if you go to github you'll see, you know, roughly 20 clients out there that you could use in your preferred programming language Apache itself has lots of very interesting projects that have been built or written has Wrappers on top of all the clouds that you can find. Okay, so J clouds is one of them It's it's a written in Java and it can talk to all the cloud providers public cloud providers rack space cc2 Amazon Go grid, you know soft player you name it, but it also talks to you know the basic API of the Software So if you're deploying a private cloud with cloud stack, you could use J cloud and talk directly To your private cloud. Okay, so it J clouds and I'll mention leap cloud But they support public cloud provider, but also they have interaction with the the native API of all those solutions You find other things like there are more involved like stackmate is something that that's coming up for cloud stack It's basically a WS cloud formation Clone okay, so it right it works as a server that you deploy on your cloud And then it can actually process Amazon cloud formation templates and cloud formation is interesting because if you're trying to deploy You know several machines that have relationship Between them are typically a master and then some slaves then you define that in In a cloud formation template and you can feel it to stackmate. So that's that's very interesting But in in general the the main category is that you're gonna have the the native cloud stack API Coming with with cloud stack with we have a bridge to the Amazon services So if you've used Amazon and you have tools right now to use EC2 you could use those same tools and even those the same scripts And then talk directly to a cloud stack cloud and that would that would work There's been a lot of debates about you know, you need a standard for cloud And of course a cloud stack native API is not a standard There are only two standards out there these days, which is you know OCCI from the open grid forum or GF and then That's the OCCI right there and Then the other standard from DMTF is CME CI MI so those are the only two standard We now have a no CCI Interface so if you want to use tools that you know, no CCI if you want to use a no CCI client So you want to make sure that you're not Locked in into a particular vendor and you want to use a standard then you could use a standard like OCCI and talk to cloud stack We're working on a CME Interface so that that should take you know within a month or two. We should also have a CME interface So I think you know, it's very important to to make sure that you know We don't want to lock people into our native API. It doesn't make any sense We want to provide all the interfaces that you guys want to use So if you want to use standards, you should be able to use the standards if you want to use the native API You should use the native API Amazon you can use Amazon and then I work with some some students in Ireland and this is the Google computer engine logo So we have a GCE interface So if you if you're using Google computer engine right now You deploy a private cloud with the cloud stack you run our GCE interface on front of it Which runs as a server and then you can use the same tools the GCU teal from Google You use that client and you talk to cloud stack and it works great So that's that's fairly important The CME is a little bit of a shame that that we don't have it already because in Apache There is a project called Delta Cloud Apache Delta Cloud. It's a top-level project as well. It's all Ruby It was given to Apache by Red Hat, but unfortunately they they decided to you know not contribute that much to to the effort but Delta Cloud was the only software for a long time that provided the CME interface So if you were using Delta Cloud you had that standard You know available will probably will probably you know, even though there is less activity in Delta Cloud We'll still go forward and put a cloud stack back in there So that if you want to see me through through Delta Cloud you'll you'll get it And please you know stop stop me if you have if you have any questions, you know, no problem So demos, you know, that's what I find really interesting I give quite a bit of talk and I'm tired of going through slides and slides and slides So, you know, let's go through through through demos. So what I want to do is is for demos okay of using different clients and Contrugation management system because if you have a cloud you want to deploy nodes and you want to configure them So that you run your apps. Okay, so whether it's, you know It could be as simple as my SQL database as involved as you know a MongoDB cluster with multiple You know shards and replication and things like this a React cluster if you want to build your own S3 interface and or a Hadoop Cluster if you want to do Hadoop on demand for example So when you're trying to to deploy those, you know involved infrastructure onto a cloud You need some tools and you need to automate things So it's very important, you know from our cloud stack standpoint that yeah, okay You build a cloud but then we need to make sure that all those tools work well with cloud stacks so that you can automate those things I'm gonna go through Libcloud a little bit which is It's very similar to J clouds, but see it's a Python It's a Python. It's written in Python It's an API wrapper on top of all the the cloud providers and cloud software that you can find So I'm gonna show you how you interact with a cloud stack based cloud using using Libcloud I'm gonna show you salt stack Because of course, you know if you're doing configuration management, you've heard about puppet and chef There is another alternative out there, which is salt stack It's pipe on base and I'm a Python guy So of course I tend to gravitate towards Python the project But salt stack is very nice and what we're gonna do is that we're going to provision a salt stack master master node That's the equivalent of a puppet master We're going to provision that automatically in a cloud and then we're gonna start additional nodes And we're gonna see how we can configure those nodes Using the the master so all of this deployed from my lab from my laptop in the cloud It's pretty cool. We're gonna do fairly the same thing with chef The reason why I'm doing chef is because there is a very nice Knife cloud stack client Available, so if you're using chef and you know the knife utility who uses chef Yeah, so if you use chef and you know about knife you guys use knife, right? So you can use the cloud stack plug-in and automatically you can provision machines and Define recipes and then provision nodes like this So I'm going to do a little chef demo where we're going to start an elastic search node In a in a cloud stack cloud and then the last one is also with an Apache project Apache where It uses jet clouds underneath, but where was you was developed originally by the clouds the guys at at cloud era who are doing a big data Hadoop distribution, sorry the the cloud era had to distribution CDH and the developed we're so that you had a Commonline tool to be able to configure Hadoop clusters in the cloud Okay, so I'm going to show you how you use weird to to provision a Hadoop cluster Sounds good excited Okay, cool so leap cloud as I said, you know, it's It's Python and it's an abstraction It's an abstraction layer on top of all the the cloud provider that you have and you know I have to agree that you got you got so many, you know, it's It's almost tiring Cloud Sigma soft layer eucalyptus open stack, you know software distribution Terra mark, you know a cloud There's probably HP cloud open Nebula elastic host line node GCE go grid. I mean you name it all of these have A driver in leap cloud and what leap cloud gives you is a common API So it was not trying to create a standard. It was really trying to create a wrapper that had a common API So it's very nice because you can use like list all the images list list list the instance type List the locate the availability zone create the nodes configure the nodes all those are in the common API But that common API is very small. Okay, so that's a little bit of the the issue so every Every back-end in there actually has extension functions to provide additional functionality that are specific to that cloud Okay So let's let's have a look at leap cloud I Don't want to bore you with code, but if you want to look at the code. That's that's a tiny, you know example of You know Python loading the cook The modules you create a driver that she's class tax specific You define your API secret keys just like with Amazon and then here in that example I'm running an interactive shell So you'll you'll see what it what it does Can you see this is it big enough? Yeah, okay, so this is I Python if you don't know Python gives you a nice a Nice shell so you have here. I've done a connection to this cloud Exo scale that CH. Okay, and actually I should go back and show you exo scale I'm using it this as an example of a public cloud that is using cloud stack It's a it's a cloud in Switzerland and I live in Switzerland So I know these guys and they've done a very good job So they are basically a public cloud provider that's using cloud stack. They're running a business, you know selling cloud services they have This one open cloud which is That's the cloud stack cloud. So you can easily get an account They give you a little bit of credits if you want to if you want to try it out and learn the the cloud stack API You know just going through this they've created their own dashboard So it doesn't look like anything I showed you earlier. That's not the default dashboard from cloud stack. They created their own You know dashboard if you want to create an instance here through their UI. It's as simple as you know here demo create There's only one zone one data center you pick an image. They have predefined Images you pick an instance type. Let's do micro if you want to create additional disk blah blah and key pair and security groups There you go What they've done in terms of networking here? It's a little bit interesting that they're using a basic zone So all the virtual machines share the same layer to so isolation between the user is through security groups Are you guys familiar with security groups? Okay, so in Amazon that's that's also how they do so basically you create a security group You give it a name and then in that security group you define rules Which are basically IP table rules where you say okay open 22 port 22 for Everyone and then you can also define specific rules for within machines in that group Okay, so that's the basic dashboard and they've done some clever things to pre-stage the images So the here the instance is already running What happens since it's a basic zone is that every instance gets a public IP here? Okay, so it's specific to their cloud So here that's that's running so here in that example of lip cloud, you know, I start the shell and it you know it Automatically created a connection to that that cloud. So now in lip cloud you do things like this list You get type, you know tab Type completion list location and you get You get things like this so it's an instance of the location it tells you okay, so it's a sweet Zelen The country is wrong, but that's that's no problem. You get list images So those are all the images that are available You get list sizes Those are the instance types Available in your cloud okay, I Actually created additional functions that that look a little bit nicer. So if you do list images you get in return So what do you get on those clouds? Here for them they actually attach specific Dicks disk sizes to those images. So that's why you see different images and this is the UUID that's internal to to cloud stack You get interesting things like you know if you want to access an image everything's gonna use SSH key pairs so if What do you do if that if you have functions that are not part of the lip cloud base API then you have to use the EX You know naming convention, so you see all the functions that are available so list key pairs List key pairs Type completion and here you see that I only have one one key pair. Okay, so let's create another key pair for example So it's going to be EX create key pair and we need to give it a name name demo Great key pair So you get you create a key pair it returns the private key stick that private key You're good to go if we go back to the dashboard and we list the key pair. We see that the demo key pair has been has been created We can do the same thing with security groups, of course because once you start a machine you want to be able to You know access it so let's let's look at all the security groups that we have in there So you see that I really have a few security groups, you know So I could create it, of course, I could create an additional security group create security group and you see how things go, okay Now let's let's create a machine So it's going to be create node So in that particular case you see that I'm not using that EX extension functionality because create node is Common to all the clouds in lip cloud. So whether I'm using Amazon open stack or cloud stack It's always going to be create node. Okay So it's going to be the same the same call. So let's call it demo 01 It takes an image argument. I have preloaded an image Name so you could specify your own image, you know, of course here You specify an instance type So amount of RAM number of CPUs and so on and then we can specify a key pair Key name And I'm going to specify the exoscare key. Okay, so that's how you start a machine. Oh Yeah node equals sorry connection connection So that would be the same call in any cloud open stack you see to you name it It's fairly quick here. I'm not specifying security groups So since I'm not specifying security groups, it's going to use the default Okay, and my default as SSH open. So we find so here it returned. So if I look at my node object you know, I have I have my my node instance here running and There's an extra attribute So when we see here that, you know Kind of go here because of the mic Leap cloud created the node here It returned the node object and then I have some extra attributes which tells me that it's been created like right now 1607 more less It tells me exoscale is the key. It gives me a password That's created at instantiation But that's actually not gonna work Because of the way that exoscale set up their own cloud. Okay, they don't allow Password access security group is default and it gives you the zone. Okay So if we go back to the dashboard, we should see an instance demo zero one created Yes Yes, okay, so demo zero one has been created And of course if I take the IP address Let's go here No, sorry Wrong IP Bang So here I have created an instance in that cloud in switzerland using leap cloud and I have Key pair access to it. So now I can do whatever I want. Okay, so that's the that's the basic functionality Leap cloud also has, you know, lots of other things that you could do is cloud stack like Port forwarding You know adding rules authorizing of course specific rules in the security groups You you name it you could do you could do lots of things If you want I can show you later if you're in If some of you want to see more Okay So leap cloud leap cloud is very nice Works great Now starting a note that it's kind of boring. Okay works great Now how about chef? Okay, so you want to do you want to actually configure a machine You want to start a machine and then you want to apply some recipes to it so that you can configure it So what are we going to do here? I'm going to run knife on my on my laptop. I'm going to start a machine in that cloud the same cloud exhaust scale I'm using hosted chef Which means that this chef server this master here this chef server is actually hosted by ops code. Okay You need to do a little bit of setup with them like Credentials and things like this but once you set up on your laptop I mean it takes like I don't know 20 minutes to register for hosted chef and And get set up So what's going to happen is that you know We're going to start the machine and then the machine is going to contact the chef server Download the recipes and then configure things in our case here. What I want to do is is set up an elastic search node It's a little bit more advanced. I'm also going to set up nginx so that I can access it through a through a proxy So let's go here Which directory this one? Okay, so knife knife is very nice Knife cs cloud stack server list, you know, it's been configured Okay, the line gets wrapped because the screen is too small But I configured my keys in there so that it talks to that exhaust scale cloud the same thing So knife cs server list I see the two instances that I have running I see their ip their service type. They are micro instances. I see their os and things like this I can also do things like cs Template list to list the instances So extremely similar functionality than with leap cloud except that, you know, it's uh, it's with the knife plugin Knife cs I think it's service list So that that that's gonna give me the The instance types that are available Okay, great And I think they have a knife cs. Is it location? No zone list sorry knife cs zone list Okay, so here you just list the availability zones that I have in this cloud So here I only have one and we recognize that Geneva, switzerland zone Okay, so now to to start a machine and actually, you know, give it some information so that it It sets it configures things I'm going to Hopefully back up there you go and use this command Let's go back up so that you can you can see better Okay, so here, you know instead of server lists, I'm going to create Very similar starting the node Uh specifying the the instance type. It's a tiny instance Specifying the template ubuntu I'm I'm saying that I'm going to ssh choose that machine as root Could be something else I'm specifying a key a key pair that I'm using so this is the path to my private key Remember that I've created key pairs on the cloud So this is the path to the key pair that I'm going to use And then run list. This is specific to chef. Okay, those guys those are you using chef You recognize that you need to specify a run list here I've defined a role and it's an elastic search node Actually, let's make that an elastic search head And we're going to look at that that role Head this is all chef stuff. Okay, this is not cloud stack very specific I mean the commands are specific to the knife cloud stack plugin But this is really chef configuration management stuff here. I'm specifying that I want The instance to be started with the exoscale key You know, I could create another type of key pair I can also create a key pair with the knife cloud stack plugin, which is pretty cool Specifying the security group here. I'm not using the default. I've created already previously an elastic search security group And no public IP that means that I don't want knife to do anything To enable static net or port forwarding or anything. Okay, so no public IP It's basically a way of telling this this plugin don't do anything And the reason I'm doing this is because the cloud gives me instances that have public IP addresses So I don't need you know, I don't need anything to happen. I don't need any any net stuff And and then I give it I give it a name an instance name. Okay, so let's say Uniscon 3 Okay So we launched that Knife is written in ruby. So it has you know Basically, it's making api call to that that cloud and it's going to start deploy virtual machine With the proper parameters Let's look at a look at that at that role here. That's that's defined here What is it knife role edit Elastic search head So that's again, you know knife specific This role is actually stored on the chef server at ops code Okay And and what you see in that in that role here is that it's especially this run list has a list of recipes that are going to be applied to the to the node The the apt each of these recipe points to cookbook that chef App is just going to update, you know, all the packages It's going to install git because I for something else. I needed to get Get java and then elastic search elastic search has some nice recipe on github Set up nginx and then configure everything as a reverse proxy So if you're trying to automate configuration of machine in the cloud just prepare your run list for your machines and then You know, you'll be good to go So here we see what's what's happening is that you know, we have the standard out of what's happening on the machine It's shutting up the the chef client It's going to install the the needed packages and configure things out If we go back to our dashboard, we see that linux con 3 has started and the machine is being bootstrapped Okay So we let it do its thing. It's not going to take too long and then we'll we'll get back to it So knife integration in cloud stack works like a charm Really, it's very nice uh I don't have a a puppet example, but I have a salt stack example There's a little backstory with salt stack, but because it's written in python Um I got involved and I wrote the the cloud stack Driver for salt cloud. So salt stack is just another configuration management out there alternative to chef and puppet You know fine Evaluate the one you prefer Pick the one you like Whatever It has the equivalent of a knife plugin kind of okay, it's uh, its functionality is much smaller But salt cloud is aimed at being a client that you can use to deploy those machines in the cloud and And then configure them. Okay. It's it's not as powerful as knife, but that's that's it's the idea So what are we going to do is that we're going to Instead of using in the case of knife. I used the uh a chef server hosted by ops code Here what I'm going to do is that I'm going to start a salt stack master in the cloud directly And then I'm going to start additional nodes that are going to register with that master. Okay A little bit weird, but salt needs to run as root And when you use salt cloud, can you read this? Yeah Okay, so again similar functions. Then you have with leap cloud or knife list location Exo scale Yeah I should have I should have fixed that Don't don't worry about the the dictionary and the warning as as debugging things just Just concentrate on the output Here so you want to list the location you're pointing to the cloud again and it tells you exo scale is a cloud stack cloud It's uh, you know confederation elvetique jeunesse This is wrong, but that's it's a leap cloud issue UID whatever So list location again, of course list images And we're going to see the same error So you list your images You list your sizes Fine tiny instance It tells you one gig of ram. Okay fine same same type of things The way this works is that with uh with salt you define providers So if I look at the exo scale provider uh here Well, that's fine. I'll I'll change my keys It tells you api key secret key because that's what you need to talk to that api again specify the endpoint the host the path Security group that's the security group that I want to use when I'm going to start those machines Uh, the user, you know, because of course it's need to ssh to bootstrap things Where is my private key and the provider type here? I'm specifying cloud stack. This is going to use the cloud stack driver. Okay I could use another another cloud. I define, you know different type of information for a different cloud Here I have three different Cloud stack cloud that I can use I coula coula That's another production cloud in Paris that's using cloud stack I'll change the keys. Okay. I always forget to uh So now now you we're connected to the cloud we can we can list things Let's start a machine. Okay, and Let's call it demo zero three So here I'm calling I'm using salt cloud and I'm going dash p. It means use the profile And first I need to deploy a master So I have the name of a profile which I called ubuntu exoscale master I don't know why it's exoscale er. That's not going to work So what you need to specify also is a set of profiles So here I have a set of profile. The first one is a master And I'm saying which image type which instance type I need to use the interface for ssh the key And here I'm saying this this profile here make this a master. Okay Fine So we start this So it's going to do its thing Yeah, the warning is fine And here we see demo zero three as already is in startup mode. Okay So the instance has started and then salt is going to to bootstrap it Let's go back. Let's go back a minute to chef Yeah to chef so chef has finished Going through all the recipes So technically we should have an elastic search node running, right? Hopefully So I use chrome I use sense as a sense is a nice chrome plugin for Elastic search and So I need to find the IP of that node Let's go back to the dashboard. It's this one. It's the 137 Just 37 So it installed elastic search. It also installed nginx as reverse proxy because I'm just doing this Otherwise anybody could hit the elastic search cluster and start adding data. So here I have some basic HTTP authentication. So here you go This credentials are all set up in a in a chef Data bag I log in I send the command sometimes Sometimes it doesn't work. It configures things, but then so let's ssh to the machine Yeah, sometimes it doesn't start properly, which is not the way it should work there you go So here what we have done, okay, is that we've used knife Defined the recipes for elastic search nginx reverse proxy created credentials and then in like, I don't know five minutes we've configured a node in the cloud with elastic search installed and now you have that that that Elastic search node only one running you could start putting data into it and it would index things and then you could do search and things like this Uh, I also have recipes to add node So I could start additional nodes and they would automatically join Well almost automatically. I'm still debugging a few things They would join the cluster and we would have you know, let's say 10 machines In that elastic search cluster Okay, cool I think it's very cool So it looks like here, uh Salt is still going Salt is still doing his thing There you go So finally it went through so same thing then with chef and any configuration management now It's bootstrapping the machine and it knows that it needs to make this node a master So it's going through all the packages and and installing things So we'll get back to it. Okay shouldn't shouldn't take too long The uh, the last one that I wanted to mention Is uh, we are Apache where it's a top level project also at the Apache software foundation It uses jet clouds and we are was uh, especially created for big data Okay, so if you're trying to go deploy a hadoop cluster, uh with whatever distribution see the the basic hadoop or cladera Or you want to put hbase on it for big table Uh yarn for map reduce You name it You can use where you can also use where to uh to deploy kassandra Even elastic search You know, you could do it. So basically where is orchestrating the deployment of multiple nodes in that cloud? Okay, deploying one machine is easy deploying, you know dozens of them and have them, you know linked Is is harder. Okay, so look at where It can be used uh where it can be used as a Uh as a library Well, actually salt has finished Okay, so salt has finished configuring everything and it returned So what i'm going to do here actually i'm going to launch Additional nodes, okay, so they're going to be in salt terminology. They're called minions So let's start another node with salt that we're going to call minion and let's let it do it do its uh do its thing, okay um So let's go here Go back to uh talking about where So you could use where as a java library Writing java code and actually using uh where as a library and then you know deploying things or you can use it as a Common line client, okay If you use it as a common line client, then you need to define File for the cluster that you want to start. Okay, so here that's an example of uh an adobe cluster That define it's pretty it's pretty basic you have a you know Keys that you define so the name of the cluster adobe Here this is a little trick to configure a head c host if dns Is closing your issue Define a key pair for access, you know to the machines again define, uh, you know Yeah, clients there was for the access And here you see you define your cluster. So you have one adobe name node. That's your head node and then 10 10 data nodes Okay, so you could you could have five 15 you name it you you pick uh Going down This one is interesting it tells you This one here says I want to use cloud error adobe distribution for Uh, and here it's telling you know how to install and and configure Then you define of course the uids of the image uh instance type that you want to use Okay, and here you define the cloud that you that you're going to to use to deploy that cluster So it's a cloud stack cloud. There is very good support for cloud stack in j clouds Define the endpoint the uh the idea of the template that you want to use okay so If I didn't go over my quota of machine in that cloud Uh Can you read that? So you would go where lounge cluster config And you just type the uh configuration file Okay That's that's the that's the uh the basic use of of where So now same thing where is going to talk to that cloud use j clouds to to use the talk to the api start 10 machine 10 or 11 machines and then ssh onto them and bootstrap Cloud error so that you get a adobe cluster Let's go back hip and here We're seeing that I have All my adobe nodes. So I have I should have my salt stack minion demo zero four that I started And then I have all my adobe nodes You see that the name has been picked up by weird automatically and they are they are in startup mode Okay, so we have to leave that a little bit of time To configure, you know Start configure adobe get adobe running Now if we look at uh, if we are back to salt Here we so we have started a master and here I've started uh a minion, okay, so My master what we need to do now is actually you know connect to the master Uh, which I guess is uh This one 138 So let's connect. Let's connect to the master node which lives in the cloud Ah, no, I shouldn't have to do that. Maybe it's the virtual and stuff Yeah, okay So I'm connected to that that master and I should be able to do salt key dash l So what I'm trying to do here is that? I've deployed a master and I've deployed two minions the way salt works is that now. Ah, okay. I made a mistake My bad Okay, so I've deployed the master and I've deployed the minions so salt Normally should the the minions should automatically know where is their master and they should contact the master on a specific port And say hey, I'm available for you if you need to configure me Okay, and they established a secure channel by you know defining keys, but I did a mistake which is I forgot to edit my Profile Now you'll see you'll see why Here I've defined the profile for the master node, but the profile for the minion Here I need to put the that's that's a limitation to salt cloud. It's not automatic is that you need to uh You need to enter the IP of Of the master. Okay, so what has happened? I started the two minions and they are trying to contact a master that doesn't exist okay So 1928 138 So what are we gonna do is? Oh for we're going to delete oh for I'm just gonna Try a new one Okay, my bad. That's the problem with live demos Sometimes you you make mistakes Okay, so we're gonna let let that mean start. Let's go back to work to see what's happening here Okay, so here again, you know, we see we're doing its job Which is configuring It started all the nodes and now it's configuring hadoop and So where is java? But of course there is some bash at one point Because the installation and the configuration is just basically bash scripts that are You know pre-set up in in where so you see the the bash You know being being executed here So, you know copying the configuration file defining variables Okay, so yeah, I guess it may take 10 minutes to start To start a hadoop coaster of 10 nodes Okay, while we while we wait a little bit Wait, I'm done So maybe we can we can wait to to see to see success But basically what I want the message that I wanted to tell you is that Okay, fine You pick you pick the cloud the cloud solution that you want and now, you know, you have a cloud now What so you're going to want to do on-demand provisioning of machines and you're going to want to automate Configuration of those machines not only single machines, but also clusters So you're going to need tools to do this and you're going to need to to work on the the automation Okay, and you know if with cloud stack here Hopefully I've I've showed you that you can use, you know lots of tools You pick your language, you know if you're more comfortable with ruby or python You can you can pick your language if you're comfortable with puppet or chef for salt stack, you know, you can pick We are really trying to to get very good support for cloud stack in in all those tools To allow you to to basically, you know automate things and then build really on-demand infrastructure Um, so thanks a lot you can you can follow me on twitter I post a lot of tutorials and and screencast on how to do all those things um Of course the apache the cloud stack website cloud stack that apache.org or w w cloud stack.org We are on irc the entire community is on irc You can find all the slides on slide share screencast on youtube You name it if you want to participate in cloud stack Then you know you go through the website, of course and then Apache is all about mailing lists. So you join the mailing list and you you start participating put some filters because we have quite a bit of traffic on those lists So you don't want to be inundated of You know of email so filter your cloud stack email and and then participate We have a cloud stack conference in Amsterdam november 20th and 22nd First day is hackathon workshops. We have jenkins workshop chef workshop elastic search workshop And then the the following days the 21st 20 seconds We have talks from users people who have deployed cloud stack in production public cloud private cloud And and also, you know developers, of course talking about new features and and so on So I can take I can take a few questions if you want while we while we wait things haven't finished here Where is still working and Salt is still working on that on that minion here Yep Not done yet So do you have any questions? One question. I need at least one question Last talk of the day And thanks guys for coming. I know, you know sticking around until the end is Always difficult Okay, so that minion has started if I fixed it correctly I should start seeing the that minion register with my master Okay, so let's go back to my ssh session here Yes Thank you So you see that it showed up That means that The master and the minion are in the same security group and it's set up so that the the port is open It's like 4505 or something So they they managed to you know make a handshake and the master is saying hey I got a request from this guy demo zero five. He wants he wants me to accept his keys So you need to you need to say yes capital a Yeah, the following keys are going to be accepted demo zero five. Do you want to accept? Yes, I want to accept Okay, fine now accepted keys demo zero five. So I've registered that minion So that means that now I can use all the power of salt to configure it Okay You know, I'm not going to show you a recipe but the basic test is to Is to say, you know ping Okay, fine. So the master has managed to to ping that minion So salt recipe are kind of nice because they're you don't need to learn ruby or you don't need to learn a dsl specific Which is the case for puppet and and chef the the recipes in salt are all yaml yaml files, so You want to see a salt recipe? Um Let me show you a salt recipe react So that's a salt recipe. So it's all it's all yaml based. So here it says for example a repo Uh, that's the name of the repo Disabled no here's the key that you need to set up and uh require you want You want the react package from that repo and here you define react? It says what it requires it requires the react repo package Okay, so you don't need to learn a Language i'm not an expert in salt, but that to me is a little bit easier than than learning, you know Especially I like chef a lot, but I don't know ruby. So it's a little bit more challenging for me to learn a chef recipe Uh I I don't know I don't know No, do you know? I don't know. Yeah, so Chef is nice with data back so that you know those credentials that I use for For elastic search. They are stored in a data bag um But basically you have those recipes and then you just tell the master, you know, you just say hey Install that install that recipe and then you have you've configured your node. Okay, uh, let's go back here. Oh, yeah Our adobe clusters is finished So let's ssh to the master So again here. I'm in the cloud in switzerland And hopefully it should have a hadoop cluster That's s forward slash Yeah So I'm saying yeah, but I actually know that there's a little bug that I'm not going to show you but It it installed it's installed adobe. I have an issue with the the security groups So map reduce map reduce doesn't work But now you have a 10 node adobe cluster in the cloud You can start putting data and and run map map reduce job I have I have a full tutorial on how to do this and uh, and I think that's a world record I think we did that in 15 minutes Okay One more question for the road No You guys were interested in cloud stack or we're interested in tooling both Cool. Well, thanks a lot. I'm sticking around. So if you want to ask, you know, additional questions, don't hesitate and otherwise twitter These days is probably the easiest way to to ping me and I'll post the slides even though, you know, there's not much content in the actual slides But I think they'll they'll put the presentation online and and you can look at all those packages I I maintain the the cloud stack driver in lip cloud and sold stack And knife the knife cloud stack plugin is maintained by some guys in Amsterdam, okay, so if they are in issue if you start using those those packages and you you run into issues because You're you're doing a scenario that you know is not yet covered by the tool we can we can easily Fix it quickly Cool. Thanks guys. Thanks a lot