 Okay, so everyone look down at your watches. Does anyone's watch say it's not 9 a.m.? Raise your hand if it's not 9 a.m.? Okay, welcome. Cool, so welcome to the earliest, most thought-provoking, brain-squeezing thing that you've ever done after a night of partying. Was it Wednesday? It is Wednesday. So, my name's Colin McNamara. I'm the chief cloud architect at a company called Nexus. Actually just got bought by a giant company called Dimension Data. We're here today to share with you some of the tool sets that some amazing developers have made. We have Mr. Mark McCallana, and I'll introduce him a little bit more just in a couple slides. And Mr. Amy Bustardo. Amy and Mark are on our upstream development team over at Nexus at Dimension Data Company. So today we're gonna have a workshop. So if you have your laptops with you, I shall ask a question. Who came in for a talk and who came in for a workshop? Talk. So who's just expecting to sit and listen? You can do that. That is acceptable. But who came in with their laptops ready to be able to do stuff? Awesome. So keep your hands up really quick. Okay, loose count. It's about 20 of you, 30 of you? Okay, close enough. So, if my clicker would work, maybe if I turn on my clicker it'd work. So one of the goals that me, the community and my team have is lowering the barriers to consuming and contributing to OpenStack. So I've been in the community for two and a half years now. Came in about 2011, minor contributor since then, co-founder of Project. And one of the biggest challenges that we have is people who wanna contribute, who wanna fix a bug, who want to submit a patch. That first, the steps to that are really, really hard. And so what we're gonna do here is I'm gonna, we're gonna get things started of transferring some files over with. We're gonna introduce Amon and Mark. And during the time it takes to transfer the files over, so when it has the data, we're just gonna talk a little bit about what and why we've done this and do a code walk through. And then we'll go ahead and help you guys through kind of a self-paced workshop. So let's go ahead and start transferring files. So first thing first, we have I think 10 or 12 USB keys. Nine. Nine, we have nine USB keys. We have 30 people, so that means three to four. So group up in teams of four if you can please. So everyone raise your hand who brought your laptop. Okay, look at each other. Look left and right. Those are your new best buddies for the next 90 minutes. Okay, you're gonna have to learn to share. You're gonna have to learn to communicate. This is a relationship building workshop. It's for building love and friendships in the open-sat community. So it also is because to be perfectly honest, I submitted this as a workshop. I thought it was a presentation and I spent half of Monday finding USB keys and realizing they're really expensive. So please also, if you can return the really expensive USB keys to me, I would really love that. I love storage, but I don't want to own, I don't have to pay for this out of my pocket. Okay. Who are we got? Come on, why don't you guys huddle up here a little bit and, oh, sorry. Okay, so it takes about 12, so part of that's basically doing five and 12 minutes to transfer files, right? So just want to transfer the files on these folders there is what, on the USB drive? Yeah, everything, put it in one folder in your desktop and then we'll work on that. Speak louder, please. Hello, yes. Yeah, speak louder. Yeah, so copy everything that's in the USB key and put it in one folder in your desktop that way we can access it easily. Okay, and don't delete it and no viruses please. So if you're a Windows in a Mac user, there's the, excuse me, virtual box is on there. If you're Linux users, about 50 billion different files or 50 billion different builds. So if you're a Linux user, please go and download virtual box on your own and start updating your dependencies, is that correct? Double check though, I think there's a Linux installer for virtual box in the USB key, I think. So all that you would really need to download would be the VB Guest plugin for Vagrant. Cool, oh cool, so most people don't follow the prerequisites, so I really thank you for reading. Yeah, if you have virtual box installed already, you don't need it obviously to reinstall it off the USB key, you still need the rest of the files, the project files on the USB key though. So who pre-installed virtual box? Oh my god. Perfect. That is so awesome. You guys are great. No, it's okay, you don't have, okay. Look at the people who raised their hands before and see what you're supposed to do. No, it's fine, I never do the prerequisites, I'm horrible. There is however, virtual box for like seven flavors of Linux on that USB key as well as Windows and OS X. So you should be covered, if you don't have it, it will be on that USB key unless you're running some very obscure version of Linux or running something that is just even more obscure than that. I have another USB key that we can pass around by the way. Oh wait, no, no, I see how much individuals, everyone who needs one, come up and fight for the death. Groups of four, groups of four, we all get together. No, no, we all get together. Okay, two more people come over and sit next to them so we don't have communication issues. We don't have a USB key waiting. So if you are done transferring, raise your hand, say who's next. I gave the last one away. So because you have crappy internet access at a workshop or at a conference like this, we basically pre-warmed up the controller images, all the different nodes and set it up so it'll work on offline mode. Yeah, and the project files are separate than the installers as you'll see when you get the USB key, so it's a fairly short copy if you don't need the installers. The larger chunk is the virtual box installer and all that junk, so it's a relatively small copy if you just need the images. And the repo is public, so it's on GitHub, so when you go at home, there's whatever, you can use whatever you want to patch it to. So let me introduce your mentors. While we're transferring and fighting over USB keys, again, it's all right, if there's a really good fight, document it, share it. Makes it all have fun. I want to introduce people that are really important to me. They're important to the community and I think a lot of times we don't get a chance to validate these people. So first and foremost, Mark Maglana, he's standing right here. He's a QA engineer at the Nexus Dimension Data Company and is absolutely amazing. He's a great tester and great developer that understands the value of test-driven development. He's a, there you go. He also does more than QA testing, actually wrote our multi-cloud bidding code, which is really cool. The gutter damn commoditizer run. We can talk about that later. But that really does some really interesting things. That's outside of the scope of this discussion. He's an OpenStack Active Technical Committer creator. Did I screw up your airframe again? Yeah, it's aviator. Dammit. Okay, it's aviator. Okay, please, I need you to create a project named airframe so I get it right. He's also the creator of a really cool tool that we use for our own project management, or Rosetta. If you use Trello, anyone just scrumb on, use Trello, come on scrumb. Anyone try to pull data off, it's absolute crap, right? Because it's optimized for speed and security, not accuracy. Here with this awesome tool that is in preview release of the test bin update, some preview release that actually normalizes data out of that. And he's just a really awesome, interesting dude. And so the code that we're going through today is Mark's code that he had to do, Mark and Aiman paired up on, but I think you were the lead on it, right? To be able to troubleshoot some multi-node depth stack, or multi-node neutron API timeout issues, and submit some patches and tempests so there's better coverage. Now Aiman Bustardo, who I couldn't find a good picture of, I'm gonna be taking some selfies with you, is also another amazing guy. A horrible picture. You like that? Yeah, no. That's okay. I never like pictures of myself anyways. Can you take permission issues? Huh? Can anyone else having permission issues copy? Guard is a copy. No? Okay. How would you have? Pair programming, if you're friends. Troubleshoot it, send it to another. There shouldn't be any permission issues copying. Yeah, especially copying. Copying should have it. If you can see it, you can copy it. It's read, baby. So we use Mac, as most of our friends do. So Mac should be, this should work a lot on Mac. We haven't tested it on Linux, but it should, right? It'll work on Linux. If you're running Windows, I'm very, very sorry. Okay, so Amy Busardo is a really awesome Ruby developer who hides his awesome Ruby developer world under like an amazing DevOps engineer banner. So he's the number two contributor, I believe, so a puppet open stack. Used to be. Used to be? Probably gonna be again. Hopefully. Yeah. He is an undercover ninja, I believe. What's your martial art? Oh, Shaolin. Shaolin, okay. Is that ninja? Or is that like? Wrong country, but that's okay. It shows how worldly I am, right? And he's a development engineer. They're all architects in the highest level of skill that we have in our company. But he's a developer engineer at Nexus. And so work on our internal and external projects. I'm Colin McNamara, Chief Cloud Architect, Director of Cloud Practice. Been in the open stack community for two and a half years now. It's really great to see everyone joining it. I am a core reviewer for open stack manuals, co-founder of open stack training. Bit of a heavy Linux user. And I am the Chief Architect of an amazing beard and an amazing family. So what's that? No, okay. So why do this? I think that, so I talked a little bit about this. So is everyone transferring? Who's got the first files transferred yet? Okay, did you pass it over to a friend? No? Not done? Okay. So we're only like 10 minutes into a nine minute thing. And by the way, there's some videos available so you can self-pace. There's a self-paced guide so you can self-pace. There's the full upstream code so when you're connected, so you can self-paced. So why did we go ahead and have to solve this problem? I'm gonna talk a little bit about the problem statement for the customer. I'm gonna hand it over to Mark and Galana to talk a little bit more about the challenges you were trying to face. So we had a customer that's a server-srider customer that we were engaged with. And they had chosen to create an Ubuntu-based OpenStack cloud. Most of the work we do at Nexus is infrastructure-based. We upstream selectively, we try to partner with our customers. Customers facing a problem with escalating API time-outs when they're using overlapping IP addresses in Neutron. The requests for, the requests in the APIs are escalating to what, 12 to 15 seconds? And honestly, it didn't wanna fix it. They kinda thrown their crap into, they thrown their cloud into production. I'm not in the business of throwing really, really smart people to respond to other people's emergencies. I'm in the business of helping them and helping others. And so what we did is I made a decision to say, okay, well, yeah, we'll help you, but we're gonna help you by improving the tempest coverage. So this doesn't happen to you again, and this doesn't happen to the community again. So whenever we can find good for our business, good for our customer, good for the community, we wanna do it. And so the goal was to be able to improve the tempest suite. And who knows what tempest is? Good, okay. If those who don't, tempest is quality control suite that allows us to develop OpenStack safely. It's as Mark puts it, the seat belts allow us to drive fast. And that area of development, Neutron was driving without seat belts. So we went ahead and entered into development on Multinode Neutron, or specifically QA, improving the QA test suites from Neutron in overlapping API addresses and a Multinode setup and had obviously challenges. Developing on SingleNode DevStack is easy-ish. Developing on something that looks like a production environment is hard. So I'll give Mark the mic really quick and you can talk a little bit about what you're doing and then maybe you can do a code walkthrough. Sure. Yeah. Okay. Let me get you to this one. It's fine. So while he's setting that up, okay. We got a USB-K? Oh, it's corrupt. It's corrupt. We killed one. Okay. Don't, don't, don't crap. Yes. Then if there's, if you have a spare USB key, once you're done with your USB key, stand up. If you're, your team of four, we don't need USB keys sitting around. And one of these days, we'll find a good way of running a workshop at a conference. And maybe we'll do like the, you have to show up three hours before and get your data or something like that. I don't know. So as Mark's logging on right now, trying to, and I'm gonna go basically talking to Apple so he doesn't have to worry about focusing on both. So internally in our development, we use Amazon a lot. We build OpenStack locally. We just get Gary Jenkins, just like everyone else. And there is Mark. So you wanna talk about the why you were doing and the probably like the challenges you faced and then kind of what your decisions were and then you walked through. Yeah. So, you know, I do a lot of, I used to do a lot of development on top of OpenStack. As Colin mentioned, I created Aviator, which is a library that allows a Ruby developer to communicate with an OpenStack installation. I also, you know, in my past previous employer, I developed a, I helped create a dashboard that, you know, talks to OpenStack. But I have never had experience until I joined Nexus in setting up OpenStack and actually, you know, setting up OpenStack to develop, help develop OpenStack. And this was my, you know, first opportunity to do just that. So I heard, you know, to get started developing an OpenStack, it's a good way, it's a good thing to have DevStack in your local machine, you know, and that's where you code with, or that's how you test your code or integrate your code with the rest of OpenStack. Now I'm a fan of readme-driven development where I basically first, before when I start a project, I first write out the readme to, you know, tell the next developer how to set up the software in his local machine. And then after writing the readme, I use that as one of my guides when I code. When I started writing the readme for, you know, setting up DevStack and setting up Tempest, it was, you know, it was lengthy. So I decided, well, maybe I can automate this. So I started with Vagrant, which is a great way to automatically spin up VMs, you know, in your virtual box environment or your VMware environment, your local machine. And then I, you know, went ahead and wrote some puppet manifests. I'm not here, a full disclosure, by the way, I'm not very good at puppet. Aiman here is way, way better than me on that one. But, you know, I did my best to write something that was easy to follow. So the prerequisites for this, you guys already have that. If you went ahead and looked at the readme PDF in the USB, the prerequisite is virtual box, Vagrant. There is also the VB guest plugin. Instructions are also in the readme. And then we'll go together, we'll, you know, we'll walk through the whole, launching the controller and the compute node at the same time. When you guys, when most of you are ready. So a little walkthrough of, it's a bit small, but I wonder if we can adjust the, you can embed in it. Control plus plus, right? Control plus plus, yeah. Think. Yeah, you should be able to zoom in. There you go. Excuse me? Go to Windows, go to Windows Zoom. So right now it's gonna go inside and go through the, no, that's not it. This is the only node. And I can't hear how to assume this. That's the readme index. Oh, well, there you go. Oh, thank you. Man in the back. Here you go. Alrighty, awesome. You guys are awesome. So that's. All right, so, this is the entire DevStack dash tempest project that I created and put up in GitHub. So a little walkthrough of the contents of this one. Yep. The GitHub location, it's in the nexus is. So if you'd have .com for slash an exercise and then it's for slash DevStack dash tempest. Yeah. This, the ones that you have right now that you copied is slightly modified so that we don't have to download the source of open stack from the internet. It's all gonna be local. So it's a bit different, but most of it is the same as what I have there. In the GitHub repo. Right, so the main thing about the main file here in DevStack tempest is, of course the read me because we do RDD here. And then here is the vagrant file which we will use to spin up those two VMs, the controller and the compute node. I've defined two nodes over here. The controller. And then basically this definition tells vagrant what's the RAM of the VM that it needs to spin up? What are the provisioners that it needs to use? Down here I've specified two types of provisioners, the shell provisioner and a puppet provisioner. So once the VM is up and running, vagrant will then hand over the control to puppet to configure those VMs. So the puppet manifests are inside the puppet directory. And then the shell provisioning scripts are there under the shell directory. There will be, there is a tempest directory which is not immediately available when you download the repository from GitHub because that's part of the provisioning process. So the cool thing about this project that I really like is once you spin up those two VMs, puppet will also configure tempest inside of your controller and then put the tempest project in a shared directory. That way you can, and the shared directory shared between the local machine and your controller VM. That way you can still edit tempest using your favorite editor in your local machine and be able to run the test inside of your controller VM. So that's my favorite part. All right, status of everyone. Everyone, who else is still copying their files? Okay, cool. So that's not gonna be a problem. We have a video walkthrough that we will share online. And also I have a screen, my fault. I also have screenshots in the USB disk that you can use to walk through each of the steps. Alrighty. Cool. All right, so let me get started on this one. Okay. All right, so one of the things that we need to make sure is set up over here is the virtual box networking. You go over to the virtual box preferences window right here and just make sure that you have this host-only network with the following settings. Its IP address is 192.168.42.1. And- So the same IP address. Yes. That way you don't have to mess around with a code because it assumes this IP address. Of course, later on you can modify this and also modify the vagrant file settings. Sorry. What was that? Sorry? You click on, yeah, you can't see it from there. It's click on the virtual box menu on the top bar just right beside the Apple logo over there. Preferences. Yeah, virtual box preferences. And then that's how you get to this one. Sorry, what was that? It's gonna ask for people. Yes, yes, exactly. It's gonna be exactly the same as the screenshots in the USB drive. So once you get to the preference pane of virtual box, make sure that you have this host-only network with this settings. And also make sure that the DHCP server is disabled. We're gonna make a vagrant assign the IP addresses of our two nodes, the controller and the compute node. All right. Oh, sorry. Address of the network is 192.168.42.1. And the mask is 255.255.255.0 and leave everything else the same. Disabled. Just disabled. Uncheck the enabled DHCP right there. Moving forward. Yeah, we'll leave everything else alone. You probably don't have this yet because you haven't started your compute nodes yet or end controller node. The next step really is just to say... I don't know, Zoom. I have to look at that one, not this one. Okay. Alrighty. So make sure that you have vagrant VB guest. The instructions on how to install VB guest is there in the PDF. So it's vagrant plugins. Vagrant plugin install VB guest, I think, is that it? So vagrant plugin install vagrant dash VB guest. And... No, go ahead. Sorry, sorry. Sorry, what was the... What was the step that you... The whole network. Yeah, I apologize for that one. Yeah, I apologize for the networking part. Apparently I didn't do very well with a read-me-driven development this time around. And I'll call out on this. So it's also my fault for basically being totally disorganized and deciding to go to Hawaii after the company was acquired instead of working on this. Yeah, we can mal you instead of doing this. It is my choice. So yeah, how we like to approach community is this is what we share, what we know. We're not trying to share, we're perfect. We're just trying to share with people some of the lessons we've learned. And thanks for the feedback on that. Yeah, it's community. We're not paid to be here. We'll get the read-me properly fixed. I need to do some automated testing on the read-me file. Can you do that? Cool. Is that possible? Yeah. Yes. Yeah, there is. It's called an intern. Yeah. So if anyone wants to volunteer to test our testing testing of testing code, we have job openings like everyone else on Earth. Is it anybody else having trouble with the networking bit? No. The question was if your network name matters, no it doesn't. It's just done. It doesn't. Okay. Yeah, network name does not matter. Yeah. Cool. Downloading this. Vagrant. Yeah. So do you want to continue moving forward? Yeah. Well, so the ever, Vagrant was, well. Yeah. The Vagrant installer is also in the USB disk. Yeah. We should not download anything. Yeah. For Windows? No. Yeah. Yeah. No. Latest would be great. So to be cognizant of the time. No promises if it's super old. Yeah. So probably, we'll keep on going to make sure, again, we can do this self-face. This is videotape. We want to make sure everyone can at least reference the working stuff here. Yeah. Okay. Yeah. So the code's online that you can do it yourself? Yeah. Yeah, we will. Yeah. We did. It's nexus.io. Go to github.com for slash nexus is. And it's the Vagrant DevStack repo on here. All right. We'll make sure to post it at the end as well. Yeah. All right. I'm going to go ahead and fire up the compute node. So, you know, once you have your environment set up, it's, you know, just say Vagrant up, compute, and then dash dash provision just to be sure that it actually does the provisioning part. Most of the time you don't need this last part, but it's good to have that around. Very good. Yes. That's me just being an airhead. Yeah, it should be Vagrant up, controller first. And then the compute. So let me clean this up. All right. Vagrant up. Excuse me? You could destroy the compute. Yeah, we'll do that. Controller. I believe so, yeah. No, he's in a bash shell right now. It's just a terminal in your local machine. Then you will want a sig win or something similar so that you can run in. Oh, there you go. Yeah. It also installs the windows for get and includes. Some kind of shell. Yeah. And in full disclosure, we don't use windows. Yeah. Yeah. I literally, a friend of mine brought in a Mac and opened and booted windows on it and I cussed him out. And well, he wasn't a friend then, but it's bad to start a friendship. I did this about two weeks ago because my arm was heavy. Okay. Like, well, some of our business process guys are out of functions. I think the tools are getting better on windows because some people have to because of the jobs they work. Yeah. Some people choose to work places where they don't have to. We'll welcome your feedback on the windows. Hey, Eamon. Yeah. You have a bring your own Apple policy. So BYO and A. Sorry? It's definitely the reason why you didn't make the network nodes separate. Network nodes separate. Resources. Just use the controller and the computer. Yeah. So I forgot to mention for this demo, we're just going to do the simple DevStack we're setting up, you know, not, it's not new Tron based yet. It's Nova network to keep things simple. One of the reasons is because there's very limited route in my local machine. So, you know, I wanted to keep it simple just to demonstrate, you know, you can automate your setting up of DevStack using vagrant. But you mentioned about multi-node deployment. Yeah. So, well, this is multi-node. Yeah. It's multi-node. Just the compute and the controller for now. Yeah. So if anyone's, the challenge that we've seen is if you actually look at, so we also do some development for Open Daylight and doing DevStack multiple nodes. So if you're going to do like Open Daylight ML2, Southbound and so Neutron ML2 ODL, to actually do any testing on that to actually develop on any multi-node. And on the ODL side, when we're trying to get Open Daylight into the gate, so, because Open Daylight is a reference SDN controller for OpenStack, it's not in the gate, so you can't really develop into it. And so what do you have to do there? All run tempest against DevStack, right? So there was a, it's getting working with Mestri, who's also the, now the PTL for Neutron, Medivador Batal, Florian Hotel, and Brent Salisbury. You know, there's a lot of, especially on the ODL, so they're like really great networking guys, and obviously Kyle is a great OpenStack guy. But it's just a giant cluster, right? It was really, it was just a whole bunch of focus on getting people set up, like they come new to the project, and getting multi-node DevStack set up. We kind of take for granted that we got it figured out. And so this was one to make it so we can bring more, bring it easier for us to develop. But then also so the community wouldn't face the same darn challenge, right? So whether you're doing multi-node Neutron, multi-node Nova Network, multi-node whatever, the same challenges are getting your stack, your C files, mapping out the virtual box images into like a local, so you can use your local IDE, stuff like that. I don't want to interrupt Mark unless you're waiting for something to do. Yeah, I just wanted to show, you know, this part over here, the VM is, the controller node is ready, and it's installing DevStack. And I can SSH to the controller using Vagrant SSH controller. And then I can follow the installation of DevStack by just tailing the logs over here. This also, this step is also included in those screenshots in which you copied. And so that's it, this is puppet, putting DevStack inside the machine and then running it. And then once DevStack is done installing, you know, the open stack services, it will then add a, an entry into the log, into the configuration file offline equals true. That way the next time you run DevStack, it's not going to download everything from, you know, from the repost again. If you want to make it download from the, from the original repost, you can just take out that entry from the config file. So over here, DevStack is done. And then it's just setting up additional stuff inside of the VM. That way any instances that you spin up in open stack later will be able to communicate outside. There were some tiny little stuff that you need to play around with, you know, in virtual box and the VM itself. Can you show in the, in the puppet code where the stack RC file gets populated from? Or is that an amen thing? The stack RC file? Yeah, yeah, so you populate, you populate the DevStack. Oh, yes. So first of all, that's really all that gets added to DevStack offline. It goes through and in the manifest, where did I put that? I think it's in the controller, right? I believe it's in nodes. No, I think you're right. It's not controller. There it is. So it's just the file line plugin. It adds, you know, whatever is the entry in offline.txt, adds that to local.conf. And then the passwords and IP addresses and all that. Which file was that inside on here? You know what I'm talking about? You're passing configurations to DevStack itself. Yep, those are static files right here. So for the controller, if you want to set up, you know, if you want to add some more stuff to the DevStack installation. This is really just, you know, DevStack configuration file that Puppet, you know, loads into the machine and then DevStack sees this and that's what it uses to configure OpenStack. And for us, you know, it's like, okay, how do you actually take a development team out of a single developer and get them consistently using the same environment? How do you make that consistent between your laptop, your build environment? How do you make it so that you can bring people in? And I think one of the challenges we have in OpenStack and Open Daylight Development is that we bring people in and out of projects and development environment setup is actually a big barrier and a lot of people leave. And so that's one of the goals of sharing this is that I hope each of you are either new to a project or a quarter of a project or bringing people in and can start understanding the code and hopefully maybe improve it. So continue. Are we up? Sorry, there's a question over there. Speak loud. He's speaking in the microphone so the tapes can hear you. In the virtual box examples, you have a controller and a compute node already configured with Fedora 20 or something like that? It's Ubuntu. Ubuntu, okay. Did you have to specify the network adapters in those VMs? Yes, but it's already configured over here in the files, I believe. Yeah, so for if you're going to- You don't have to manually specify anything. The vagrant file specifies the IP address of the VM. And also- You don't have to do anything in Vbox to tell it to use the host-only network? No, it should be automatic. Mainly because right here, because of that entry inside of a VM definition controller. So when you define the Ubuntu VMs, you didn't have to set up the ethernet one, ethernet two adapters? That's what vagrant does over here. So you may, if you have DHCP on the segment though, it may, then you might be hitting a conflict. You might need to set up a private segment. Thank you. Okay. Do you want to show through? Yep. I ran into this. I don't know if you want to throw that up, but if you have a newer version of VBGas that'll fail to mount the shared folder. There's a workaround if you vagrant SSHN to do a sim link. So that does seem to work. Well, thank you. Okay. Is it this problem? Amen, amen. Same problem. Oh, hey. Same problem. Oh. Ah! Like a... Oh, so let's not crowd. Let's not crowd. So, hang on a second. Sorry. Pop it up and show it if you want. So the same problem was simlinking the files. Was the sim link in the folder share? Oh, cool. Thank you very much. So the question or repeat the question statement in the problem? So the problem was when you run Vagrant, it couldn't mount the shared folders. So you just need to do this step inside of the VM and then rerun Vagrant up controller dash, dash provision. And that should work. So while everyone's doing that, do you want to show how the why of the mapping of folders, so of how when you have multiple nodes? So as Mark's shown that, who uses RMate or Rsub to be able to use Sublime or TextMate or whatever your ID is from DevStack? Do you use DevStack inside of a VM and you develop using an ID on your laptop? OK, I do. Who goes and just use Emacs if you're Evil or VI if you're awesome to do it on DevStack itself? OK, who develops on DevStack? OK, cool. So when you do start developing on DevStack, you'll probably find that because DevStack is basically able to clone the repos locally. And you can make your changes. It runs everything inside of a screen instance. And so you can basically restart that process, make sure it works, run your tests. There are some times when maybe I love Sublime because I'm not as good as these guys. And so being able to have an IDE that has a whole bunch of whizbang is really cool for me. But since I'm developing on another VM, I either have to transfer files. I have to do like a reverse SSH tunnel as a pane in the rear. In a single instance, you can kind of do it. You can basically set up your reverse tunneling. And when you have multiple instances, it gets really dirty really quick. And so one of the things that Mark found was ability to basically do folder mapping. So you're always working in your local file system. And so you don't have to set up those hacks. It's just another thing that allows you to develop faster. So to show that, I'm back inside of the controller. And I just go to vagrant tempest. And here is my tempest directory. I can put stuff inside of that. And it'll show up right there. And this is my local machine. So I think that's a very handy thing to have. You don't have to fumble around with setting up our mate or the equivalent of Sublime or changing your IDE just because your code is inside of that VM. Really? Another note, too, is a lot of people are having problems getting the VBGas plugin installed under OSX, depending on which version you have and the underlying libraries. If that's the case, the only thing you're going to lose, obviously, is the shared file system. But there are other ways around it, as someone just mentioned. So don't let that stop you from continuing. Skip it if it's that much of a problem for you today. And you can work on something like that from home so that you can progress with the rest of the workshop. So anyways, yeah. So VBGest, if it's killing you, just skip it. The only thing you're going to lose is that shared file system. And I'm just going to repeat what you said, so it's on the video. And let me just make sure I'm making it back right. So the hardsum links aren't supported in the fault folder sharing. But inside of vagrant, you can set it to set an NFS export, so you can mount that NFS share from those VMs. And this is not pushing backstands, but if you have any links to that, or maybe if you want to throw a note or a pull request into the Git repo, OK. Yeah, it probably is. OK. But yeah, that actually sounds really, really cool, because that gets around some of the issues if you want to do this inside VMware. So if you have Fusion on, because VMware doesn't, at least I haven't been able to figure out how to map, how to specify what folders get mapped in. It just provides a guest folder mapped in. I have Fusion on my stuff. Although I like to try to make sure everything we do in community-wise is virtual box. Because you never know, like someone in India, Saigon, who's really, really smart, but doesn't have a lot of money. We don't want to basically say, this is how we do it, and then can't have them join the community. Yes, sir? Hours. It's theirs. I mean, they work for me, but. Yeah, we do. Yeah, yeah. But I mean, for this, the goal of this was to make sure that we're improving tempest test coverage so the community and our customers and ourselves didn't hit this fault to make. By the way, if you want to drive a hole through gaps of test coverage in the tempest suite, you can probably just close your eyes and drive in direction. You'll find it. I'll go to the VB guest. So OK. So guys, if you are going to skip VB guest, you'll need to comment out two lines of the vagrant file. The one there can just highlight it. Yeah, that line that Mark's highlighting right now, if you can't get VB guest installed for whatever reason, comment out that line that he's highlighting in your vagrant file. Otherwise, it'll complain that you don't have the plug-in. There's another one down there. Yeah, there's one for the computer, one for the controller. Make sure those are both commented out and you can continue booting your vagrant images. And then you'll be able to continue with the workshop. OK. All right. So once your controller node is up and running, then you can access it from the browser. We look at hypervisors. We don't have anything there. Can I put one safety statement here? Sorry to interrupt you. One thing that I think I had in that note that we're supposed to talk about when we had breakfast, this is not a Pock environment. This is not a demo environment. This is a development environment. If you're planning on taking this back to your shop and showing you having your developers or your boss or your peers test out OpenStack doing this, it is probably going to be a very bad experience in the long run. So this is for developing for OpenStack. Sorry about interrupting. I hadn't said that. That's a very important thing to make. If you want to actually go and just do the installation, there's other repos that we've put up that will allow you to install it and some other really awesome stuff. Or you can just use PacStack or whatever. So what are you doing here? OK, so I'm just cleaning up the compute that I started earlier. Because I don't want any dirty stuff in the file system. So the controller is up. I'm cleaning up the compute. And then I'll start it up again once this is done. So once we get the compute up and running, then we'll go back to the browser and check if the controller can see that hypervisor. And then we'll spin up a VM inside of it. So we'll give Mark a little bit of time. It always sucks when we're all on stage in the awkward silence. Again, absolutely my fault for realizing this was a workshop not a presentation. So who's gotten to the point where they got vagrant installed? Who's gotten the VMs compute starting? OK. It's starting. OK, look to the people next to you. The people that don't, you can go shoulder surf. Remember, pair programming. We're teams. Give them the verbals. Uh-huh, uh-huh, uh-huh. Right, don't be a greedy pair programmer. Don't give feedback to your buddy. It's all right to look over the shoulder. It's all right. You learn more. We all do it faster when we help each other. Those of you that had the, in the corner there, we had a lot of the folder sharing, right? And the, excuse me, had some Ruby gem dependencies. We're able to resolve that? Yes, no? No? OK. Let me see. So we're waiting for this. We've got 60%, just making sure we can actually complete through the video or the workshop. So there's the self-paced videos, the feedback we're capturing. There's the screenshots, and everyone has the code. While we're waiting for that, oh, still going. There we go. May I pass it over to my friend? Excuse me? OK, yeah. So I'm bringing up the compute note. Is this on? Yeah, it is. OK. But I've got the log on screen. I think I can log in before the compute comes up, or I should wait. That would be kind of one question. And then I don't know the credentials. It's admin, and then a password is password. Awesome. OK, thank you. And that's also in the stack of CFA. Yeah, you can later on, when you have time to play around with this yourself, you can go to the local conf of the controller and the compute, and then you can set up, well, mainly the controller. You can set up the admin password over here, and also a bunch of other passwords. I just make them all the same, since this is just for testing purposes. Yeah, so that's how you customize that. For those of us who are struggling with the Nokogiri gem, and that was messing with the mounting the file system locally, can you repeat the bit again? Where is it mounting it on my Mac host? Repeat that. Sorry, can you? So the SIFs or NFS mount that you were talking about, with the shared on the guest in the host? Shared folder. Yeah, the shared folder. Yeah. Where's the location of that? To comment it out. Line 18, 20, 2. It's in the vagrant file right here. There's the second line, I think. So you customize in the VM to set shared folder inside the vagrant, so don't do that. So that basically just tells vagrant mount the entire thing inside of the mount this entire project inside of slash vagrant in the controller node. Do you want to just leave that up there for a minute while if anyone's needing to comment that out so they see it? Does anyone have trouble finding your vagrant file or understanding where to comment that out? Line 26. Yeah, just. Is that the only one? Yeah. OK. Yeah. Yeah, you have to vagrant reload. Yeah, vagrant reload. Instead of up, vagrant reload controller dash dash provision. Can you repeat the question that was asked? After commenting out line 26, do they still need to reload the controller? OK, and the answer? And the answer is just do vagrant reload controller dash dash provision. OK. Line 39, that's just line 26. So you have controller? All right, so the compute node is up. And if I refresh this one, it'll show up something over there. Eventually maybe. There we go. That's the compute. Before that, let's run a simple tempest test. So I have a sample inside one of the read me's in this directory, not the PDF. You can go into the controller node and then run this test. So let's do that right now. Vagrant SSH controller, pseudo as root. Because I want to run tempest as root, cd vagrant tempest. And then just copy and paste. I'm just going to copy and paste this test over here. This is inside of read me.md in the dot get folder. I'm going to get the dev stack dash tempest dot get directory. So I'm just going to copy that one and paste it inside of the controller node. And that should run a few of the tempest test. And it'll output the results of that one shortly. Yes? This one does not configure Neutron yet. We're going to push that configuration change sometime after this. So I'm still having trouble authenticating. Maybe I don't know how to spell password. It's spelled like forward or just like Linux or something. No, it's just forward. OK. What file is that defined in? You can go inside of the puppet directory. And under puppet is files. And under files is controller. And then there's local.conf over there. And that's where the admin password is set. But it should be, did you put in lower case admin, lower case password? Is anyone else having problems logging into Horizon? Who was able to log into Horizon? No, actually a lot of people. Yeah. Is that? The only other thing is if what, databases are running or something? I have trouble. So that's setting up the default username and password. Yeah, so if you have a failed step, then you need to assume that other things are going to fail after it. And not assume that things are going to work after that. Yeah. It's the first time we've done that. So we're debugging in real time right now. I believe the default password is secrete. Yeah. S-E-C-R-E-T is the default password for Horizon. You can try that one. Admin and secrete, yeah. Basically, secrete with an E at the end. Yeah. OK. If it sounds like what is happening is if the virtual or if the guest additions plug-ins not added, then the permissions aren't being set for Horizon. And we don't have an answer, because we haven't run into that before. So that was one of the prerequisites is to have that stuff set up for the class. So we don't have an answer for you right now for that. Yeah. So if you look at the code for DevStack, they have secrete secret E in there as the default password. All right. So let me move on. Anyway, Tempest Suite was able to run over here, and it outputted the execution times. I'm just going to go ahead and launch an instance here as well. Test. Going to boot from a Cirrus image and the security. Just one, two, three, four, five, one, two, three, four, five. And it's launching the VM inside of our compute node. Viewing the logs over here is the progress of that VM launching. I'm going to, I'm uploading that right now. I will share the link inside of the Git repo in Nexus IS once it's ready. Sorry, the question was, is the video going to be available online? How to find it? We'll post a link in the GitHub repo. Yep. And we'd love floor requests. Yes, especially on Windows because we don't get the chance to test this on Windows at all. So that would be. Can you repeat the question louder? Yeah. And then run it off. So the statement was, if you're using Windows, don't use Windows. Boot it to Ubuntu inside of VirtualBox and just use the Ubuntu to manage the entire environment. I absolutely agree with that. You need a lot of RAM, too. Suppose you have them. 16, I think. Yeah, we have a reimbursement policy for any developers who choose to use Mac. We just take what we'd pay for the crappy Dells that everyone else has. But anyone in the company can do this. And as long as you have support on it, if we're going to spend the money, might as well give it to you. And you can be awesome. Look, he's awesome and looks great at Starbucks. All right, so my VM is up and running. I was able to associate a floating IP to that one. I'll go back to my local machine over here in SSH to that. Oh, wait. I have to make sure you can actually SSH. We're going to set up the SSH rules over here. It'd be kind of cool to extend the Denica stuff so that automatically populate. You're like, oh, crap. Colin just put something in the backlog. Yes. So do that again. Yes. And I'm in the instance inside of my compute node, which I can use to ping. What did I do? You're in the Cirrus image. Yes. So he's in the Cirrus image, which is inside the compute instance, which is inside the virtual blocks images inside of his Mac laptop. And if you're using Windows, that's going to be one more level down with your suggestion. So you're doing the VTARDIS lab? VTARDIS? It's bigger on the inside? It's four levels, we call it our cake. Oh, there you go. Hey, yeah, that's absolutely it. That's actually how the vXperts do the virtual labs at VMC and VMworld. It works. Yeah. Yeah. Qmoo is a pain in the butt. Yep. That's basically it. That's just two commands. And you have a multi-node DevStack environment in your well, two commands after you have the prerequisites. So we've got about 20 minutes left. And yes, we have questions, and let's go ahead. You have a question, ask it. You want to open up the new, do you have the neutron code version of the set? Not in this one. No, but you can just log on to repo and pull the new, if you log into our Denica repo. No, it's not on there. Yeah, I need to reshare that code again up there. No, just to visually show it? No? No. You can log into work. It's fine. Don't show the passwords. I need to clean up some of that content. OK, so it sounds like it's like I just asked them to show the dirty underwear. Yeah. Yeah, so it'd be a center of internal repos right now for multi-node neutron. But the challenge is multi-node neutron stack RCs are out there. It's on DevStack. It's impossible to do. That make sense? Yeah, because I'm a tron for about what we're going to have. Yeah. Is the multi-node neutron one on our Git repo publicly right now? OK. So then we're going to clean that up, because I suspect there are some settings there that are not really necessary. Once that's run ready, I'll put that up there. Yeah. Yeah, absolutely. Yeah, I have to. I'm still learning neutron. Can I ask a question? Is this like, did anyone learn anything new with this? Yes? OK. Yeah, yeah, I'll go to GitHub real quick. Yeah, so, and that's, by the way, the goal. A lot of times in the community, when you're at a conference, there's a lot of stress, by the way. There's really, really smart developers who are really, they share in the community. I'm a really, really dumb developer, and I overshare in the community. And so a lot of times when we don't share the lessons we've learned or are learning, it hurts everyone else. So it's kind of our goal here. So the goal here is to share the things that have helped us develop. And if there's, if you've been able to take one or two things or take our code and either use it yourself, you can take it, you can contribute to it, you can share it, don't care. But just to help other people lower the barriers to developing OpenStack. Puppet and Reel OpenStack? Ayman, you are the expert in that. Well, that's a much larger topic than anything we could tackle here. And there's plenty of talks at this conference or at this convention for that. It's something you can't even get into in the 20 minutes we have left. I mean, there's plenty of information out there. And actually, Christopher, that was here earlier, would love to talk about that, I'm sure. But you need to start attending some of the meetups. Start reading some of the documentation. The Puppet Labs OpenStack modules are getting pretty awesome. You can even do something like PacStack, which is puppet-based. You can try Formin, which uses the Puppet Labs modules. There's a lot of stuff that uses the Puppet Labs modules. But if you want to just start learning about it, I'd just engage the community. What's your goal? That depends on what you're trying to do. Well, so honestly, you start using configuration management, like use PacStack for single node. That's just to get one node installed. You need to figure out how to. So everyone gets stuck on it. And honestly, I'm going to focus on lowering barriers to development for OpenStack here. And we can engage with the community. But this is for helping. This is not a we're new to OpenStack talk. This is, hey, I'm trying to be a developer on OpenStack. And I'm trying to develop for multi-node. And I want to make it easier. So OK. One more question. Actually, you guys mentioned about the automation world, the configuration management for the OpenStack server side. How about the switch, the physical switch side? Actually, recently, most of the switch vendors, you mentioned about the base support of Puppet Shack. That's kind of interesting. So there's a guy named Jeremy Shulman, who wrote the Puppet NetDev code. He's an automation leader, Juniper. He's not here right now, because he's been traveling a lot. There's also a guy named Matt Mike Cohen, who was on the NCME team from Cisco. I think Mike's floating around. There's also the PTL for Neutron, that's known as Kyle Mastry, that wrote the VXLint code for OBS. If you are having to configure stateful switches upstream, some of the things that we do on the Open Daylight code is we have, actually, like a year ago, we wanted to answer a question of, let me answer your question. It takes a long time sometimes. Because they're different. No, no. So you have your old switches. I'm trying to say, you have your old switches that you might have to do expect scripts. You might have to do a NetConf over SSH. You may have to find whatever plug-in you have and write a provider. You have to write code to do it. Some of those are newer switches, like the Cisco stuff as Puppet and Chef agents, but they're really only for local package configuration right now. We're actually talking about funding some upstream work to extents and providers. If you look at Puppet NetDev for L2 configuration, Jeremy wrote it. It's on GitHub. It works on multi-platforms. It works on the Juniper stuff. If you look at the new Cisco 9000s, there's really good extensions to the NX API for provision and configuring. If you have a Rista that runs OBS, or maybe you have an open networking switch, OBS configuration. So if you have something that speaks OBS, it supports OBS database, OBSDB. And that's our RPC protocol. So you can pass simple JSON in an extensible schema. So if you're automating up to that physical network layer, you really have to expose APIs. By the way, top and bottom, because your network is the interface on your server. So things like the cobbler hooks into UCS, and the stuff that you're working on right now for B-Series, as the platform that we have to develop on at the moment. You have to basically hook in the hardware provisioning north and south. It's easiest if you have OBS, my opinion. If you can write into Cisco, there's a lot of network Cisco network stuff out there. It just is what it is. I would not bother writing into 1PK. NX API is pretty cool. There's actually some sample code on GitHub. Look on Mike Cohen and Mark Volcker. Mark Volcker is here, by the way. But you can basically, you have to write your own providers right now. Hopefully they'll come out with some better providers for stateful configuration network devices. The problem is that a single network switch you can statefully configure, especially like VLAN moves. So if you need to say, I need provision upstream VLAN, OK. Now one thing is that we found out when, so we put in actually Cisco's first open flow network into production and customer base in last summer. Yeah, and it was all open flow based. It was for a specific software development need for, I can't go into the security-centric customer. But for basically like monitoring or spanning or sitting monitor ports and reflecting ports. And so what we found was really interesting is that you can do test-driven development for networking by talking to a controller. So like open daylight is really, really cool because you can go ahead and pass in the intent or pass in basically the network definition or application definition in your positive and negative test around that. And then whatever that controller goes and all has all the southbound plug-ins that you're dicking around with all your network device, right? So it has the southbound plug-ins to whatever controller you choose, Ryu, open daylight, NSX, if you're doing VMware, MVP, I don't care. In my opinion, probably the most mature way of doing it because then you're not, that is the provider. And I like open daylight. I put developer time on it because I believe in that we need an industry standard collaborative way of managing networks. Did I answer your question? No? It's hard if you can, if you figure that out. Oh, you're an ERISTA. Yes, yes. Oh, cool. Okay, you know, you know what, I'm gonna call you out on this, calling bullshit. No, no, you need to talk to Bill Finner, he's one of the smartest guys in the world and he writes your networking code. Then you need to go talk to Doug Gorley, VP of Industry, and then you need to go talk to Anchol, and then you're gonna talk to all the amazing people at ERISTA, at Cisco, at Juniper, and you need to own solving that problem. Because guess what, I can't do it easily on your stuff. There you go. Yeah, sorry, the glasses. I mean, it's not an insult. It's just saying, you know, this network, this competitive nature that we face stops people from developing stuff. Your stuff's awesome. We win Jcery. Many customers mention about the best and the provisioning of the portal solution, like the profit shaft, but the problem is the server side, maybe it's quite good too. No, the problem is that networks are distributed systems. They're eventually consistent distributed systems, and that the order of operations and workflow, and the fact that we're influencing an algorithm, we're saying, hey, device, talk to this device, talk to this device, when you eventually get to the state, do this. Puppet and Chef are configuration management. They're not built to do that. A controller does that. Not a orchestration. Yeah, it's not orchestration at all. Many end users sometimes get confused about that. Computation management and orchestration are completely different. And you have to orchestrate a distributed system, whether you like it or not, or make it non-distributed and make it controlled. And just have the control plan in a controller. But I mean, that's not, that's out of the scope of this discussion, but I actually know it's within the scope, because if you are developing for multi-node neutron, you're probably gonna be popping into exit VLANs, or L3 gateways, with the L3 nodes being, with advanced L3 features being inserted into neutron right now, you're probably gonna be connecting to a core ag layer. I absolutely believe the best way to do that, and the easiest way to develop it, like the code that you wrote for the public code for deploying open-daily. Yeah, yeah. Right? And the MiniNet code for that, I think Outland wrote for passing topologies, and did you do MiniNex with the BGP, or was that Outland? For what, the public code for it, or? I don't know, yeah. Yeah, I did public code for it. So the challenge is that you need to be able to put it inside your CI pipeline. Everything has to be virtualized. I don't even know if you have a virtual image that is exposed to people, so that they can develop on, do you? No, you don't, okay. So, please solve your own problem. Yeah, you need better documentation actually, too, about the programming API, I remember. I think it's getting better, I used to, I had a pair of programming sessions with Ken Duda once, when you were still on Fedora 9 underneath, I think, the documentation was Ken Duda. So, yeah, you need better programming documentations also. And I would say to expose more inside of industry standard protocols like OVSTB, there's a lot of Cisco, Verista, Juniper, VMware, all fighting over stuff, I don't care. It's an RFC, it's extensible, I think it's really great. I don't like how people put a moving target in the Schemos. I think it's BS competitiveness, and it stops everyone from developing on it, but I think that, you know, giving everyone a solid, and maybe, hey, HP just joined Open Daylight, maybe Arista should, too. So, okay, questions, answers, we got 15 minutes on continuing to develop, it's continuing to get our development environment set up for multi-node. I agree, Dan. So, you're using VirtualBox, are there people who use lightweight containers, like LXE, is that a big container or not? The question was, we're using VirtualBox, are there people who are using lightweight containers using LXE? We're a small team, and so it was a mongolana's choice. I think, where's Mark? What was your, what caused your decision to choose VirtualBox versus lightweight containers or Docker or something? Portability, I mean, try running in LXE on Windows. Yeah, that being said, Docker is. Right there, it runs on anything, right? That's the purpose of this thing is you can, there's Windows people here that had a little tougher time, but they ran it, right? You can't run LXE on Windows. I can't run LXE on my Mac, for that matter. It's BSD, it's not Linux, so. You can control Docker on Windows, although it's better to work in Linux. Also, I think there's a seminar tomorrow afternoon that's maybe doing exactly that with Docker, where they're using Docker containers to put nodes up. So, and I will call out, like we have a very small focus team. PNL does roll up to me, and so we have to make sure there's a balance we can't solve every problem. We solve problems that are good for our business, our customers, our selves, and the community. It's really important to me to make sure that the barriers to entry of this are really low. Okay. Yeah. You can do Vbox, you can do VMware. Yeah. You can have a Docker provider as well. And let me make one more quick statement on this. We are entering, we're starting to develop some of our, some software I can't go too deep into, to deployment Linux containers. And so, we'll be expanding our code base and knowledge around it. And what we try to do is share what, if it's an unfair advantage in the market, if it's what allows me to pay developers to do awesome things. Yeah, we don't open source it. We just keep a matrix of things that support that, that share value in the community, that accelerate that, and then we share the burden of accelerating our business, right? And things that are, that are part of supporting that, we release those too. Sorry, I try to release as much as possible, but I gotta pay people salaries. But I run a very profitable business that we actually just got acquired, so I don't know what to do. But, I'm going vacation a lot now, but the, it's, there's a lot of, a lot of pure plays that are like, they just, it's a religious thing, and they take a religion out of business, right? I think it's really cool to like, we're adding eight more developers to our team in the next six weeks, I think. And they're all funded like profitable, right? And they're releasing stuff back to the community. And that's, I'm sorry about the, kind of going on a segue on that, but like I really want to actually release a lot more of the LXC stuff. Hopefully I'll be able to, or not I, I don't, I won't write it, but I'll approve releasing it. So, sorry about interrupting you though, I had a probably a really good statement. Just to close on Windows, they do it through Vagrant. It doesn't actually run on Windows, it brings up machines in virtual box. So I'm not sure if Vagrant will, anyway, work out the connections, but just. Yeah, I think Docker's amazing by the way. I have some friends that are, I really, really think are amazing and they're some of the early guys in past that are doing some interesting things in their platforms with Docker. Especially the ability to just sync file systems. Probably will enable our development workflow to be a little quicker. I will absolutely say that we haven't put a lot of development focus into it. But I'd say Docker and OpenShift are pretty interesting. Yeah, so. Who still has USB keys out there? All right, there we go. Okay, who stole USB keys? They all left. It's like the Tabasco sausage. Today, no, this is Colin's personal USB key fund. I really all wanted to create a USB Rateray. And so I said, you know, my excuse for doing that will be just create a talk. And anyways. So if you are done with the keys, return them to the front, please. If you aren't done with them, you can use them. Any other questions or answers or discussion points? You guys want to stand up and ask? I just wanted to say. Nice. So use positive feedback, use Windows, and it worked perfectly for you. Yeah. Okay. Cool. So you had to do it. You got it working? Very cool. Okay, can I ask you a favor? Mark, Ayman, can you guys come up on stage? Okay. No. You're gonna embarrass us now. I'm not gonna embarrass. I just want to make sure. So it's always kind of a risk emotionally. You know, we all worry about being an expert. And the reality is we are community and we grow by sharing with each other, being open and honest, and sometimes we've been taking a risk. Can you join me in just thanking them for all this work and being honest? Cool. And I want to thank you guys. And we're gonna stick around for the minutes, but thank you for participating in this with, hopefully you've gained and shared. And if you have shared, or if you have gained, please tell us and everyone, it's good. Thank you.