 Alright. Welcome everyone. I am the manager of the bedroom and also presenting this talk. It doesn't mean that I selected my own talk because that would be really very easy. I have a whole panel of people voting on talk. So, I'll be talking about Up Theatre. It's a project that we've been working on over the past year and a half. And today I want to explain to you a little bit about what we've done so far. I'm currently about myself. My name is Walter Heck. I'm doing technical stuff at Volidation. I started the company in 2008 and I decided that I'm better at technical stuff than running companies. So my beautiful wife is here now as the CEO and she does a much better job than I did and focus on the technical stuff. That works very well. I am a public influencer. I wouldn't normally mention that because I don't like bragging, but it's relevant for the soft because we have built the whole thing in public code. So, I get that. My background is MySQL. I've done a whole bunch of MySQL DVA stuff. I started my career seven years of Delphi programming. Do you like Delphi? Yes. Any only true programmer language? No. Inside your eyes. It made me love a whole bunch of functionality that I haven't been getting in many other courses. I love open source, which is also why I'm standing here. And I've been in IT for almost a decade. While writing that this week, I was like, sorry, but it is what it is. So, first off, the problem that we saw. So, there's many, many new tools. Every morning you wake up and there's a mentioning of a new tool that you want to look into and that you want to start using, et cetera, et cetera. But time is limited and so is your brain, so that's not possible. However, over time, we usually see a good number of tools that become sort of best practice and turn out to be working together quite nicely. However, as that second tool grows, it can manage parts of our infrastructure. The time that we are investing in managing the infrastructure to manage our infrastructure. Puppet is really nice, but now you're maintaining a Puppet master and you need to put your code into some kind of version control. You want to do some kind of a listing, so you need to make sure that your code is all great and all nice and delivers you good quality code, but it's all time that you're spending on not managing those web servers that run your company's front-end website is what you actually hide. And this has a tendency to get a bit out of control if you're not careful about it. So, I see more and more time spent on managing these tools. Is this microphone working or is it dropping out? Now we've had problems all day. The second part of the problem is that you're managing a redhead infrastructure, you're managing a redhead infrastructure, and you're managing a redhead infrastructure. A lot of the things that you're doing aren't actually the same. There is not much difference between your, let's say, young mirrors and mine. Just that we want to keep it private and inside our company and on premise and all of these things, whatever your requirements are. But in reality, we're just doing the same work. I don't particularly like doing work that other people also do. So, we started thinking about this. We all want centralized logging. It shouldn't be something that you have to think about and you want to set up when you're starting it for. Obviously, the larger the infrastructure that you have to manage, the more you have to think about these things. But, let's say, some 100 servers, you don't really want to think about what do I need to do to get centralized logging to go. You just want to do it. So, we saw these problems in practice because of my company. We do consulting, so we go from customer to customer, and we see the same. Maybe we should turn on the screen. It's not my laptop because it's my laptop I cannot use it because it doesn't connect. Was that seven minutes on the same flight? I'd elaborate too much. So, doing all these consulting projects with different companies we saw over and over. We kept doing the same thing. This can be a style. So, we started thinking about, okay, what if we come up with a stack of these tools that we keep using over and over. We convince our clients to use that tool stack or convince us of reasons why they shouldn't use that specific tool to use something else. And we worked on making a well integrated solution for the basics of infrastructure management. And so, what is the idea for off-theater report? So, the idea is that we are using Puppet to stick together a bunch of other tools. In this case, we have... So, because we're managing our infrastructure with Puppet we are also in need of some kind of urgent control. Lots of companies, unfortunately, are still scared of not on-premise. So, we get that. It seems to be a good contender. They are making really awesome software really quickly. You know, unless you're on GitHub.com this week. For those who didn't last that you either have no sense of humor or you didn't know, there was no... They managed to drop their production database and all of their five backup sources were not working. So, they had a bit of a fuse this night. By the way, on-premise GitHub. Compared... Coupled with the CI and convenience integration solution from GitHub, it works really well. I have a session about GitHub CI tomorrow morning and it's 10 minutes to 11 in the testing and automation room if you want to see what it does. Then we have server monitoring where we chose Isinga. Isinga is originally with an Nagios clone. By now it doesn't use a single line of Nagios code anymore but it's still compatible with Nagios and it seems to be quite a popular server monitoring tool. Then centralized logging. We have the ALTSEC. I think they are trying to get us to call it something else but I think that everybody by now knows it as the ALTSEC. Elastic search, log sash, Kibana. We added Grafana with that because I'll show you in a minute. And on the front file needs to push logs to log sash. And then last one of these shadows. In this case we chose MatterMos. MatterMos is an open source Slack alternative. They have been doing really nice things as well in the past two years. MatterMos comes integrated with GitLab so if you're installing a community edition of GitLab it's literally a Matter of One configuration parameter and you are up and running with MatterMos. Surprisingly, that was not another seven minutes. That was four. Surprisingly communication in many companies is the source of many problems. So maybe I'll just use actual keys. Surprisingly communication is the source of many problems in many companies especially as they grow larger and surprising amount of people have not jumped on the Slack, whatever bandwagon MatterMos is a nice cost free easy entry solution to get that going. And just creating a room called engineering that anyone from any team that's interested in engineering can join, can do wonders. We've seen this in a number of our clients where all of a sudden people started communicating and before it was over we'll submit a ticket to the networking team and you know how this goes, not very nice but so just providing a little barrier place for people to communicate is quite useful. So that's why chat ops is a part of those fundamental pillars of Sierra. We are all supposed to be experts on one or more of these topics so why not share our knowledge and make things together so that we all benefit from the standard open source. How does this look in a picture? This picture doesn't actually mean products but it is very easily one-to-one applicable to products so we have orchestration and provisioning is puppet in our case. Sorry, at the bottom layer we have your infrastructure, it doesn't really matter what kind of infrastructure it is. Ops Sierra is the part that comes in after you have your operating systems up and running because you need a puppet agent installed in order to start configuring stuff that's kind of the level where we go in. So you have your infrastructure and then we use puppet here in the middle puppet here in the middle for configuration management, some orchestration we're currently researching putting ensemble in for the orchestration part your puppet code lives in version control this is GitLab, whenever you check something into a version control it automatically triggers a build in GitLab CI to check if everything is fine if GitLab CI for instance finds a problem it automatically follows the message in your matter most rooms saying hey this build is broken so that everybody who is interested in it can see. We have our monitoring solution here Icinga which monitors all of the infrastructure and we have centralized logging analysis with log-elastic-search and then kibana-sutana as a go-e for that. So the point is that all of these tools are not made by us and I don't want to be spending time on making these tools the point is that we make these this set integrated nicely so that you don't have to go and spend a ton of time on setting up your plastic-search and logging session make sure that it all logs to the right place and etc etc So the vision is to have a combination of publicly acknowledged best practices open source software I was considering putting Enterprise back in there because personally I think it's quite important that tool if you're using open source tools in a company that is larger than a few people at some point somebody is going to say I want to pay for support Regardless of whether you think that's a great idea that is a reality and therefore we see that the open source projects that have some kind of Enterprise backing generally are more successful in the Enterprise environments because people can go out and buy a checkmark in a box So we use all of these pieces of open software and we glue them together using puppets so that it becomes easier to manage it all I have a demo after the slides are up but it's going to be a bit limited because the demo is running on my laptop and this is just espaging into my laptop but I'll show you as much as I can So basically we're building an abstraction layer that hides or makes easier all of the things that you need to do that are not specific to your business So there's a few examples here if you push a new code to GitLab it needs to be tested by GitLab CI and when that goes well or doesn't go well the notification should go to chatroom, et cetera, et cetera this is standard it doesn't matter what kind of company you're in or what you're trying to do this should not be something you need to think about setting up and so goes for all the other items if you can think of probably a hundred more I would even say that you can get to like 50% of all the stuff that you're doing is actually not specific or unique to your company Think of it as building packages versus compiling software compiling software was the thing before but then we figured out the package management is actually quite convenient and takes away a lot of hassle so why not do the same on operations level That said, who's it for and who's it specifically not for If you want to use it, that's totally fine but for infrastructure with less than 10 servers it doesn't really make sense because as it stands now with all these pieces of software you need about four to five virtual machines to run all of this so if you have less than 10 servers to manage then you can wonder whether that overhead actually made any sense Non-puppet environments if you're already using Chef or Ansible or Salt you can figure everything then probably recognizing all the theater into your environment is going to be a lot of cursing and so a discussable benefit Pure SunCloud environments if you're doing Amazon for instance the Amazon way like you call it then puppet is probably not your best tool of choice and you can wonder whether you want to dive into the wormhole of trying to fit Opseter on Amazon it'll run and you can do it if you really want to but I wouldn't say that that's necessarily the best tool of choice and the last one if you already have fully implemented an automated environment then it's probably also not very much of a use On the other hand if you have already used one or more of these tools so now I come up with a client and they were already using GitLab they were already using it in Navios phone or monitoring so it wasn't a very big stretch to start introducing the new tools one by one until we had the full Opseter sector completely new environments that's obviously where it becomes the most useful sorry completely new environments where there is nothing yet that makes for an ideal candidate because then you can get up and running fairly quickly and if you already have skills in one or more of these tools then your life is going to be considerably easier because you don't have to learn a whole bunch of new technologies very quickly the current state of Opseter we just released version 2 this week which I'm very happy with it was a while in the making but we're there you can check it out Opseter.com which will just redirect you to GitLab the new things in version 2 is that it is easier to get started so before you have to do a whole bunch of stuff to even check out what it does now you can just Git clone go to the right directory and call the right fagrentup command and wait 45 minutes because it takes a while for a public master using the form and analyzing MI SQL and GitLab and all of these things to get installed but it does work without the interaction we have started to move towards smart parameters in the format as opposed to HiRA somebody mentioned it in the questions after the last talk HiRA is really nice but if you have large infrastructures it gets fairly complicated and the other thing we've found is that very often you have people in your operations teams especially if you have application administrators they don't want to learn how to Git commit clone who brand you have learned whatever they just want to make sure that this application points at it or the load balance of points at a different URL, whatever so using the form and that becomes a bit easier basic stuff like hasn't to be in LDAC if you want to use them you have to fill them out in one of the complete files throughout the whole environment all of the tools will be configured with LDAC and hasn't to be that sounds like basic stuff but again this is stuff that we are all doing that shouldn't be specific to our environment the address that you want to be using yes that's specific the active directory group or the LDAC group that you want to be using yes that's the same but the fact that you want to be able to log in to GitLab using LDAC unique for your environment that's basically the first version where we have basic integration for all the tools that we have and now it becomes a matter of making it more smart where can we add more integration there's more things we're thinking about for instance a Grapana dashboard that automatically shows groups and servers and automatically creates graphs when you add new servers to your infrastructure it automatically shows up these kind of things are up for the next reasons so that's the road ahead we're we're just in one of the deployments we've enabled the MatterMost notifications from LogSash so LogSash has a Slack API sorry MatterMost has a Slack API LogSash has a Slack output so you can enable a LogSash output for certain long messages so that they show up automatically in your MatterMost chat rooms for basic stuff but let's say a failed software run you want to know about it or maybe you don't and that's also fine so we're playing around with that more chat bots more interactive chat bots so a bot where you can say show me this Grapana dashboard and give me a Kibana URL that automatically takes me to all of the queries regarding XYZ that kind of stuff is coming up package management we found out that in the deployments that we've done so far almost everybody wants to use on-site YUM repository not the international YUM repository so either Catello or some other way to make it easier to have your own package routing easier deployment someone in our company has been working on backer and terraform stuff to be able to terraform into whatever cloud or infrastructure you want securities can support when it has nice plugins for all this gap in the plans backups I'm not sure about I'm welcome to hear your ideas about that later if you have them whether backup is something that is generic enough to have an opinion about because most companies that I've seen already have a certain backup solution in place and I don't think that I'm not sure whether it is a great idea to to automate that into Ops Theater as well so that is the road ahead how can you help? first of all just try it out if you want to bring up the full stack I warn you it takes about 12 gigabytes of memory but there is a vagrant environment for it so if you go to github.com Ops Theater and then there you can find the instructions to get it working on the running right now it's hosted on our Olandator.com github and that is simply because it's still a fairly young and small project and it doesn't make sense yet to have it hosted on its own however I would prefer to be couple Ops Theater from Olandator I don't we don't intend to ever make money off of this other than selling services or something like that but the code itself has to be open source because I don't believe that such a project leans itself to proprietary code or proprietary add-ons as a functionality if you are very skilled at server form please take a look at what is there now and see if you can improve it there are some requests for being able to not use tool XYZ and be able to use tool ABC instead so let's say that you want to be able to use Splunk instead of Elastic Search and be able to use Google Mario instead of Xenia I want to make that possible so if that's your interest then please go ahead spread the word the Ops Theater Twitter account I created it gets to me because I realized there wasn't one so the internet is full of conversations with it but not we need a website I am not a web developer so right now if you go to Ops Theater redirect you to there was something else last week but I decided that was so horrible that we just finished all of it so in short any help is welcome even if it's just a vision or coming to me after this session if you like it or you don't like it or you have different ideas or you're doing something similar or when you work together etc etc very open and then I was going to use the rest of this time for a demo however this is going to be a bit more challenging we'll give it a try anyway so what I have done what I have done here is this is a git clone of the Ops Theater there edit the session make the background white except for the yellow I changed my yellow either way right now I am sitting in the top directory which has a bunch of stuff in it so basically the Ops Theater repository as it is is a public control repository with some extra directories that have some extra stuff in it mainly here in the Deploy directory has some directories the Packard Terraform and most notably Vagrant Oscar let me show you Vagrant Oscar a little bit so Vagrant Oscar is quite nice so as you see here it is a plugin for Vagrant originally developed by I believe the Puppet Labs support department to quickly create Puppet Enterprise environments but it is very useful for a whole bunch of other stuff as well so in our Vagrant environment you will find the master which is the Puppet Enterprise master which is an open source master Isinga instance, Elastic Surge, GitLab MySQL Elastic Surge will also run Kibana and Grafana so it is quite busy if you want to deploy this in a larger production environment you are going to have to split it up but all the Puppet code makes that easy MySQL and then we have four test machines for all sections so right now we have a bunch of them running however normally you are probably used to a Vagrant file that has a whole definition of all these stuff you won't find that here what we do is we set some environment variables that we will use somewhere down the line and then here this is the Vagrant up command so we load some Oscar come things and configure with the it comes out of Oscar and to show you what Oscar can do is quite nice so in Oscar you have two main files you have the roles of YAML and the VMs of YAML VMs are the actual machines that you want to configure and roles is a list of different roles that each VM can have so we see here for instance a role for a Puppet Enterprise master for a managed host for different machines different amounts of memory so all this does if I assign this role to a host it gets two gigs of memory from a virtual box one gig of memory etc down here this is all the code that is needed to turn a machine into a Puppet Master an Enterprise agent a Puppet agent that will register itself in the foreman a REL7 agent etc etc so here we have all of our roles they are small pieces of code and then in our VMs of YAML we have VMs so here you will find for instance we give it a host only IP address so that it works in any network that you are in we determine which box that needs to be deployed on this so it is working on a CentOS 7.2 box at the moment we give it a host name and then these roles are the roles from the roles that YAML showed before so this becomes quite nice because instead of having a bunch of repeated code in each machine each machine only has a number of roles assigned to it so you have here for instance our elastic search machine needs to be registered with the foreman after it is deployed I think a machine also needs to be registered with the foreman instead of repeating that piece of code or putting it in a somic script we just do that here anyway this vagrant environment is mostly for development so I want to go to the defensive so we have our vagrant status so we have our the whole stack up and running on that on this laptop here the downside is that it gives an association to that laptop I cannot actually show the web interfaces at the moment because they are running on a host totally virtual box here but that said I can show you something else which is let's say we want to add a new machine to this and so imagine that if you are in a production environment this you wouldn't do this with the vagrant but you would bring up what I am doing right now you would be doing this in I don't know VMware for your infrastructure management so let's call this false client bring it up by saying false client false them so while we wait for that to finish here on our puppet master I cannot actually show you the format because of things but I can show you that right now we have a bunch of certificates so we have our elastic search machine our false one of the test machine GitLab I think the puppet master itself and my SQL machine up and running and after this it's finished running and log into it and then it will show up here as well so it will automatically be this machine will automatically be registered in our puppet master and you'll see in a minute you'll hopefully see it automatically get the Icinga client installed and it registers itself with Icinga it starts file-based and starts automatically sending logs to log sash so all of these things are things that I don't want to necessarily be thinking about Vagrant is very colorful with its output I'm going to need one of the things you see here on the right hand side where it says all of these machine names it says updating hosts this is actually a vagrant plugin so the vagrant oscar comes with a plugin where it is able to insert ETC hosts on all of the machines that are already running inside the same vagrant environment so if I go on my puppet master now here we'll see falset.obz.vm here obviously ETC hosts is poor man's DNS so we wouldn't actually do this in a production environment but this would be managed in your actual DNS environment so we didn't want to make ETC hosts part of obz.vm because it's really not available outside of anything else oh where's he at kept the IP address to see let's see what happens so 7 so we'll destroy the other machine so we don't get an IP conflict I don't know if you're familiar with the way puppet works but in order to have things happen on one machine that you did on another machine it requires what is called exported resources and that means that one has to happen on the first machine that is exporting its definition and then another puppet run has to happen on the other machine in this case we have our new client that runs puppet and after that it has run puppet and exported the fact that it exists to Isengun or to the puppet master actually and then when we run puppet on the Isengun node we'll see automatically the node appearing we should be able to get into this thing but as you see up here at the very start of the puppet run at the very start of the puppet run you'll see an error message saying hey I don't know this machine because when this machine first connects to the puppet master it says hey I've never seen that machine before but because of the way the format is set up it automatically registers itself it just doesn't get any configuration yet so as you see here it's immediately it's immediately done configuring all that it did is sync a bunch of plugins if I run puppet agent it didn't actually sync any plugins normally I would show you here in the web interface of the format how the machine shows up here but for now you'll just have to believe me or try it yourself so now it's run once it's made itself the moment the puppet report arrives at the puppet master the puppet master knows of this machine so if I now do puppet search you'll see here the machine is registered however it doesn't get any configuration yet because I don't want that I don't want random machines getting random configuration so what I need to do is go into my form an instance and assign class to it I'll just assign a basic class to it there so I've assigned a standard class to it that deploys the Icinga agent and the final beat etc. so now you'll see starting the machine getting configured with those specific classes that will take a little bit of time and after that it's done we can also run the public agent on the Icinga node and then watch it appear from there you have any questions? so the thing is the question is if you search for a tool, you'll find a github repository that has nothing in it the problem is that for a while we were hosting on github but then we felt ourselves okay we were bringing out the message of up theater so we should eat our own dog food and host on our own stuff and then for a while we were hosted on github and mirrored on github but that is a total nightmare I don't know if you've ever tried that but it's a no go so ignore everything on github sorry? he says it's there maybe the google link goes to a 4 and 4 it doesn't read yes, thank you issue we'll fix that the thing is that we just released 2.0 this week so some of that stuff still points to an older version probably that's a link to 1.7 anyway our public run here is finished now and you'll see that you scroll up a little bit oh you have your scrolling set the other way than I have so you'll see that the x bar is a bunch of sorry created a bunch of endpoints you'll see up here that it created some other icing of stuff the file.yaml you see here it's all being created so if we go on our missing a node and we run a public agent here now you might be thinking hey but I have all of this stuff already so what's the point? the point is that all of this stuff that you see here is not specific to any environment and therefore anybody can use this and if you start thinking in that abstraction layer where anything that's not specific to your environment you put into a theater in this case you will arrive at a much cleaner configuration of the stuff that is specific to your environment so the last couple of environments where we deployed this the public code that is specific to the environment of that specific environment specific to the environment of that specific environment that is actually very little compared to all of the others so here you see so you'll see here our positive oh it has been created automatically and the same goes for Icinga it shows up in Kibana it starts putting its logs here so if you look at the host itself you can see that depending on what is on the host itself so it starts for instance putting it's Icinga logs to you see this whole so you'll see prospectors for the Icinga logs etc etc so it starts automatically sending a whole bunch of logs that you care about to file bits so that you can there start doing things it has automatically created in the Grafana dashboard a data source for Elasticsearch and for InfluxDB Icinga sends data to InfluxDB so that in Grafana you can make logs you can make graphs coming from the data coming from Icinga and data coming from Elasticsearch so if you go along this will be more and more I think that's it for now I see that there are five minutes left so let's go and see if anybody has a question does anybody have a question the question is now how are we why are we putting behind public we are at the moment we are currently using PublicDB there is also a plug-in for the foreman so you can check from the foreman what is in the PublicDB some of the plans involve also in pulling those parts separately to separate VMs if you want so they become easier to manage but for now that's a rather standard state-of-the-art logs one more question unfortunately there is a different operating system so you saw that I brought up a REL instance the false image that I just brought up is a REL image at the moment we support REL 7 REL 6 but it's fairly easy to because of the servers themselves they all run REL 7 or CentOS 7 actually but that's only the opposite of servers so the GitLab I think of my SQL instances however the servers that you manage with that you can choose your operating system quite a long time there are some windows that we will be adding in one of the corners that we have so at that moment we'll probably have a role or a client so for all of us here we have things like windows that we have to register with we'll never be as fully integrated as this if you're leaving a room that's totally fine but can you do it quietly because people still have questions that's the demo manager not me so what I showed you now is actually the development slash test environment for testing ops theater functionality itself the point is that all of this brings up a control and then there it's up to you to bring up a test environment and connect test machines to either the basically the puppet master that comes included or even bring up a second puppet master which is also totally fine you can have a second machine that also has the same ops theater public role or a false master so we'll just get the same configuration people would get feedback here because so far we haven't been really speaking in public about it because until now it wasn't really something I would recommend other people to use and now we're getting to a point where it's really adding benefits for other people as well so I would love to get feedback thank you very much