 Thank you everybody for showing up to my talk this morning. I'm going to be talking about how Dream Hosts, our dream compute, which is our OpenStack cluster, how we use the Elk stack, or Elasticsearch, LogStash, and Kibana to ease our OpenStack pains. My name is David Walter. I'm a Cloud System Engineer at Dream Host. And if you care to get a hold of me afterwards here, a couple ways you can find me. So first, a quick overview of what Elk is. LogStash is the L. It's a log line indexing. The idea is that as your applications generate logs, they send them to LogStash. LogStash then kind of chunks up that log line, indexes it, and sends it to a database. Elasticsearch is that database that stores it. And then Kibana is used to visualize the data that is in that database. And we'll go over some of the ways that we can visualize that data. One of the, so being Tim and Toc, I don't have too much time to get real technical into these things. So I wanted to kind of give an idea of how we deployed this at Dream Host and kind of how, maybe try to inspire people on how you might be able to use this deployment to actually maybe get some more data out. So we've used containers to deploy our Elk stack, which as you'll see in a minute, has just made it super simple. We ship our logs to LogStash with RSS log, which is a pretty industry standard way of doing things. We did have at one point the centralized logs being mounted inside of the LogStash container, but we noticed a bug with that, which we believe is due to the way Docker uses AUFS and log lines are being duplicated. So there are some gotchas here and there, but for the most part, this deployment has been just amazing for us. So here's an example of how we've done our deployment. We use Chef and there's a Docker Chef cookbook that is kind of what drives all this. Some of the key things you can see here is we use a restart policy of unless stopped, and this allows us to be able to manually stop the container and do some inspection, but if the container crashes for whatever reason, Docker will try to automatically restart it. We also expose ports for Elastic Search so that way the other containers can talk to it. And then we do some volume bind mounts, which allows us to have persistent data if the container does die and we need to restart it, or if we need to redeploy it or do upgrades and that sort of thing. Similarly, our LogStash one is similarly simple. There's a few notable changes in this one though. The extra host field there allows LogStash to use the host name Elastic Search and it maps to that IP address. And this allows our configurations to be essentially baked into the container so we don't actually have to have those particular configurations changed. We do have some configurations we are passing into it. As you can see, the volume is the xtlogs-conf.d. These are the patterns and LogStash configurations that tell it how to index those files. One other real notable thing here is we use log driver none, which does make it complicated to troubleshoot this thing kind of live, so to speak, because you don't have any logs. Essentially, Docker is just dumping them to DevNol, but we had to do that because LogStash is incredibly verbose and we were filling our disks very quickly with just the Docker logs of LogStash. And again, we're also binding the ports. And I'll explain why that's important in a moment. Then the one last one is the Kibana. We do provide a YAML file to give it some very basic configuration. If you're interested in what that looks like, you can hit me up later and I can show you what we've done. But this is to give an example of we do dump a config file onto the file system and then we mount it into the container so we can see we're putting the template at xtkibana.yaml and then the volumes line, we are actually binding, bind-mounting xtkibana into the container. And this is so the Kibana application can actually see the config file. So the one interesting thing with Kibana here is it uses environment variables to find various things. So we're able to use this ENV line to set an environment variable inside the container so it can find the other Elasticsearch container. Then again, the port binding. So the reason we're doing these port bindings or may not be the reason, but an upshot of using these port bindings is essentially all of our containers run on a single host, a physical piece of hardware. And this allows the containers to talk together and it allows clients outside of the container overlay network to actually hit the applications. So we are using the hosts network as opposed to the Docker overlay network. So now we've got the whole thing set up and with Chef that was what, maybe 50, 60 lines. Incredibly simple. We can actually start making some graphs once we ingest some data. So Kibana is what we use for interacting with this stuff. It kind of has two concepts. One is visualizations and the other is dashboards. Visualizations are kind of like single graphs that kind of give you a scoped view of whatever you're researching if you're looking for how many VMs were created or whatever. Whereas dashboards allow you to put multiple virtualizations on the one screen and you can kind of get a fuller picture of what's going on. So here's an example of what a visualization looks like in Kibana. What we're looking at here is log lines. I don't know if you can see that, but log lines that contain instance was spawned successfully or log lines that contain instance was destroyed successfully. And this kind of gives us an idea of how much churn we've had on our VMs. This is by no means like perfect. These aren't necessarily the best ways to even get this data, but it does give us some idea of what's going on and kind of directs us the direction we might need to go if we wanna do a little more research. For example with this, if you're doing live migrations, technically the VM is destroyed. So again, it's not perfect, but it gives us an idea. So now that we have one or hopefully multiple virtualizations or visualizations, we can create dashboards. And those dashboards give us that higher overview. Hopefully you've created visualizations for multiple data points, multiple applications. And we can use filters on the dashboard to filter across multiple applications. So here's a dashboard that I use just about every day. You can see a bunch of, pretty much every one of those boxes there is a different visualization. We have like how many active users and how many active tenants. And the idea here is if the users or tenants kind of grow at odd rates, that might be a key of something to look at. We have API endpoint activity, but again, this is by log lines. So some of the API is just log a bunch more than other ones. So again, not perfect data, but it kind of gives us an idea. And the other interesting thing here is the tag cloud, which I'll get into in just a moment. So we can see in actually go back there in the tag cloud, we have the largest tag here is that 9924 guy. That looks a little suspicious. So we might actually want to look into that and see why this guy is being the noisiest talker across all of the log lines, across all the applications. So if we actually do a filter on the tenant keyword for that specific guy, we can see there is a spike here in the API activity, which gives us a hints that, there might be something going on here that we should investigate a little more closely. So again, the kind of bad side here is that these are based on log lines. So things like the VM created successfully and VM destroyed successfully don't have the tenant ID on them. So if we actually go back, we can actually see about that eight o'clock, 8 p.m. hour, there actually was quite a bit of VM churn. And so looking at these two graphs and doing some interactive comparison with these things, we actually were able to determine that this guy was a spammer on our network and we were able to clean up the mess that he made and not have to deal with that. But the nice thing is that being able to just look at this overview, we have the tag cloud, we have all the data right here that we needed to track this down. And instead of having to log into a bunch of hypervisors or look at the APIs and kind of iterate over hosts to see who may have stuff, it was just all right here, just in one easy to see screen. So the tools that we used here, obviously elk. We used Docker, which has been amazing to make the deployment super simple. And then as I mentioned earlier, we're using the Docker Chef cookbook. Anybody have any questions? So the question is how do we ship our logs? We use our syslog on the clients to ship it to a centralized our syslog server. And then we actually have the syslog server collecting those and sending it to the log stash container. Filebase, I wouldn't say there's no reason, we just didn't choose to do that. It's something we can certainly look into if you suggest. All right, thank you very much.