 So, oh, OK, it's all working. Hi, my name is Zascha Wadesh. I'm working at T-Systems as a cloud architect on their open telecom public cloud. And today I have a talk about inspect and how I can use inspect as a compliance tool and automated and monitoring it with Prometheus. So that larger audience than I expected. So I have a little cold. That is my first presentation with PowerPoint since 10 years or so. So I'm a developer. I'm hoping I make it all clear what I mean. But if I'm talking too fast or start talking gibberish, then please make notice and say something. OK, so what is inspect? Inspect is a tool for compliance from the Chef ecosystem. It's a tool focusing on readable compliance config. So the typical way I knew about compliance is to create a lot of paperwork or Excel cheats. And if the auditor are coming, you have to fulfill a lot of paper or your Excel cheats and said, OK, this system looks like expected and system look like expected, and so on, and so on, and so on. And it has a lot to do with a lot of manual work. So inspect is a solution for most of these problems. It's like you written your test for your software and test development. It's more a view of, OK, before I start to change something in my environment, I write an inspect test for it, and then later if I make this change, I can check with inspect. OK, are all these systems as expected? So inspect is a compliance tool. It's open source under the Apache v2 license. It's more than for all easy to read and write DSL. It has the possibilities to locally start checking or via SSH or VINRN. It's mostly platform agnostic, so I can start my checks on Linux, Ubuntu, Red Hat, or SUSE. I can start it on Windows. I can start my checks on Mac machines. I can scan for Docker or cloud platforms like AWS, Azure, or Google Cloud Platform. What inspect look like? As you can see on your right, it's in DSL that you can easily read if you understand English. And if you understand a little bit of development, you can also easily write it for yourself. It's parted in controls. And these controls have the Scriber and the Scriber have resources. The resources we use, I come later on to it. It's a lot of predefined resources you can use to accomplish most of the things you need for compliance purpose. So as you see here, in this little example, you can check if the packages are installed or, in this case, are not installed. You can also check your infrastructure conflicts with that. So you can see, OK, in this case, it's your shadow file. There, it should be a file. It should be owned by the root, and so on, and so on. Or in the test case, I present later. For example, you can check if the Zudros file has the right permissions. Or the Zudros file are exactly like the file you expected and no additional lines on it. It's easy to read. And the important case in it, you can easily extend it. You can write your own inspect plugins or you can write your own resources. So if you have some dedicated purpose, a dedicated file that's only used by you and it's not an open source tool, then you can easily write these resources. The only thing you have to know is that it's written in Ruby. So as I said earlier, you can also check for provisioning purpose. So I know it's not the right addition, but you can search, check for AWS or Azure or, as I said, Google Cloud Platform. At this moment, there are no official plugins or no official tooling for OpenStack, but as far as I see and as far as I debug this, it should be really easy to fetch most of the APIs into InSpec. So OK, what is a run look like? In this example, I used the Linux baseline from Sectorf that is an already predefined InSpec profile for in hardening OS. So as a default, if I start InSpec, as you can see on a new created machine, it's looked like something like that. You have a lot of checks that are making, and most of them are OK, but some of them are not OK. And on the finish line, you see you have about 26 successful tests and 53 failures in one skip because it's not the right address to check. So now you start your debugging or your AnsibleChef or SaltStack start or maybe your best script and deploy your default baseline. And if all are good, you have something like that, and all is good. And you have now check your rule system and can set, OK, all the predefined tests I have on my environment are good, and you can go to your auditor or your product owner and can say, OK, all the things are as expected. So the next thing I want to talk about are the resources I talked earlier. InSpec have a lot of predefined resources that are only a bunch of it. So you can see what you can use are the APT packages that are installed. It's my firewall rightly configured. It's my ZFS pool as I expected. Are the right groups if the file has the right check so it's my elastic search up and running, and so on, and so on, and so on. You have a lot of predefined resources and you can easily add your own resource. So I want to show some dedicated resource for it. That is an example to check if the login devs are looking right. As I said earlier, that is a test case from the DevSec Linux baseline system. And on the right, you can see a test about how to check your kernel parameters as ICMP are performing the right way, the way you wanted. Now, the same for the cloud resources. As I said earlier, at this moment, there are no official OpenStack plug-in available. So at this moment, you have only the way to check for these three major cloud providers, but not OpenStack. But one of my wishlist is to add these plug-ins. So the other thing is, OK, you have seen what looks and responds. You can also start inspect with other reporters, so-called reporters. As you see on the left, it's the reporter progress. It's look like the J unit or PHP unit reporter. You can use a YAML exporter, or you can JSON, and that is a really big blob about it. So now we came to Prometheus. Why you want to use Prometheus with inspect? The typical use case, you start with your OpenStack system starting to deploy something, and then you have more customers. And you grow, you grow, you grow, you add something. Maybe you have a change from Puppet to Ansible, or from Ansible to SaltStack, or so and so on. And you have a huge growing database of checks you make, or changes you make. And you have a system that has a state. Some of the machines you can rebuild from ground up. Some of the machines are totally static and have state. And are only updated on iterations. So you have a widely changed environment. So the problem is, well, not the problem. The task is to check the systems at the moment at the state you expected. So typically, you said, OK, if I have Ansible script, and the Ansible script deployed the sudoers file. And over with the whole sudoers file, then you have now, OK, my sudoers file at the end should look like exactly like that. But what if you do, you said, OK, I only want to add some line, or I want to remove some line. And in this particular case in this VM, I only want to add this user. And on other machine, I don't want to add this. So you have a wildly changed environment of sudoers files. So that is a point where Prometheus is a common point because typical, that is my use case, I want to run InSpec at the end or at the beginning of an environment change. So if I said, OK, I want to add a new database in my system for the customer, I said, OK, at the final state, the system should look like. And then said, OK, that is my InSpec script. And then I let it run at the end. But as I talked at this moment, it's not every time possible because you have a wildly changed system. And maybe you don't know the final state of your environment at this point. So that's the point where Prometheus comes in ground. I think most of you should know Prometheus. So I make the Prometheus instruction really short. Prometheus is some time series database with the possibility to monitoring things. So in short, it's easy to fetch or to script exporters with Prometheus that export the data and then interpret it with alert manager or something like that. So what have I done is I write in exporter for Prometheus. These Prometheus exporter look like that. I defined a script interval. If I have a huge InSpec profile that's checked the system, the MySQL or the process, the Rebit MQ, and so on and so on, I have a huge book I want to run. And this book took not about five seconds to accomplish. It took more than three minutes to accomplish because it's a rightly range of checks where accomplished. So I said, OK, my script interval is only one time a day because that is OK for me and my time of 30 minutes. And then that is needed for my exporter. I said, OK, I want to check following a profile. And I want to define my targets where I want to check. And then the other case is one is via SSH and one is over local. So maybe you have a system where you don't have installed Ruby or you don't want to install Ruby or you have a dedicated version of Ruby which is incompatible with the InSpec runner. So InSpec gives you the possibilities to also run these checks via SSH. So I can SSH or WinRM into the target machine and then run these scripts. And if I do that this way, I don't have the requirements to install Ruby. So in this case, our two examples about one is for SSH and one is a local running target. So now we come to what my config for my InSpec exporter look like. The InSpec exporter are defined by SSU on the lower ground or defined, do you need SSH? Then you have to define the SSH key and identify on which port are you listening. And if it's need to be sudo, then you can add sudo and the check are running via sudo. So if you check your sudo as well, you need sudo to get the root rights. And last but not least, the path where the InSpec profile are been. So you can add more than one profile to my exporter. And last but not least, the prefix. The prefix is only used by Prometheus to make it easier to distinguish which tests are running right now. So if I have more than one profile, I have an InSpec for my sudoers. I have InSpec for my Linux baseline. I have a profile check for my engine X or my local answer. Then I have more than one profile. And then I prefixing them to find them easier on Prometheus side. So last but not least, that is what I get in response from my Prometheus exporter at the default. And as you see, you see something like InSpec. That is my default prefix. Then my prefix I defined in my profile. And then the tests that are running. After that, I can easily create an alerting rule for that. So I can say, OK, if not all of my baseline checks are OK, then I want to create a major alert. So what are the limitations or the NICIPIC details? Let's go ahead and remove the NICIPIC because, as I see later, there are some bigger issues with my exporter. I use a JSON-min reporter to I start InSpec with a JSON-minimum reporter that only gives back a small bunch of information from this one. The description of the check that are running. Is it passed or is it not passed? And I think it was. Only two or three things. So it's only a little bit of response that I get from InSpec. I have to change this description to a Prometheus readable format. So that is a bit of normalization there. And one of the things for Prometheus, you only can add unique values. So you don't can have two values for the same key. So if you have two times the same description in InSpec, I only write the first one into Prometheus. So if you have a double description, a duplicate, you have to change it. And last but not least, as I said earlier, I use the JSON-min exporter or reporter. I would like to change that to JSON. That's a lot of more data. But with that reporter, I also can use the title. Most of the time, the title is a little bit shorter than the description. So OK, now I have talked a lot about how I accomplished it. Now I want, in short, show how I write such InSpec book, because I think most of you never heard or only see it once what InSpec look like. So typically, you write a profile. You don't have one file. You write a profile for your checks. So the only mandatory files you have to need is in InSpec YAML and under Controls, your control that you want to use. You also can add readme or you can use additional files that you need for your InSpec run. You also can add a library where your own resources are implemented. And you can also add more than one control, more than one file. Look in this folder, look for the InSpec YAML, and then go through the controls and see if there are some files linked or something like that. So what does the InSpec YAML look like? And typical, the only required fields are the name, the title, are the only required fields for that. You can also attend a maintainer, your license, your version. If you are at rate over more versions, you can give support platforms. These profile are running on. So you can say, OK, I have a profile that's only running on the latest LTE Ubuntu. You can also add one for Red Hat, also, and so on, and so on. The other cool thing is you can add depends. So if you have a baseline Linux OS and said, OK, that's a baseline, but I have additional things I want to add, then you can say, OK, I have a depending on an already existing profile, and that's my overwrite. Then you use the depend fields, and you can use attributes. Attributes are used if you have some passwords. You don't want to have directly in the controls Ruby files. So that now goes to the controller code. As we see earlier, that is what a controller could look like. You have a lot of criterias. You can also add more text to each test. You also can add more than one text to uncheck to give them more meta informations. You can define the impact factor that are interesting to you for your overwriting. So if something has a lower impact factor, it's more important. So if I have something with a higher impact factor, it's only overwritten if I have something with a lower impact factor. In this case, I have write an example script about, it's the Zoodooers package in the right version, I expected. Does it have the right credentials? And it's a hash write. There's a demo hash part. Later, I feed it with the write hash. So the same goes for the Zoodoo ST folder. So the next step is to make one manual start. So I started on my local machine, and I get it all green. And that's my new default compliance check for my Zoodooers file. So the next step is to get these profile into my Zoodooers exporters, and also add them to Prometheus so the new file are gathering by Prometheus, the new data. So what is the change on the Prometheus config? It looks on the left. You see the InSpec Linux baseline. That is my baseline that you show earlier. And now I have my InSpec Zoodooers. I also want to get it in every minute, and a timeout of 30 seconds. And my new module called Zoodooers. And my target is my local host. And on the other side, I have my InSpec exporter, where I gather the Zoodooers file as new profile config. So my InSpec exporter get to this file, past all the entries, and for each profile config, the Prometheus exporter start these check. And if you don't forget to add them in the alerting system, you can see on my system my example in alert. So I see, OK, my Linux baseline have a major error. So there is something completely broken. And my Zoodooers only have a warning because some of the minor change I add are not OK. And now you see that I'm only deaf and don't have much experience with Rafaana. So I only have this little gouge about my Linux baseline things. But you also can get this information via Rafaana from Prometheus to build up your dashboard of compliance checks and add them to Rafaana and show your auditor. OK, that is the actual state of my system right now. So another interesting thing is there are already a bunch of profiles for Linux-based systems and typical used services like EngineX, MySQL, Postgres, Apache, most of the Linux-based systems, you can find them on DevSec. I only can say please look there to start with inspect. And the other part you can look is at the so-called Chef supermarket. It's something similar to more from the Ansible perspective, the Ansible Galaxy. So there you find a lot of predefined books you can use. You only have the URL of supermarket in your book in your profile. So that is what the DevSec look like. So you have a lot of predefined can then get from GitHub and can look here what are their definition of a good-looking EngineX. So I have what are the future of my development with inspect. At the moment, the outer reload config or profile, the outer reload on config or profile change are already done. So if I add something to my inspect config, then my exporter, I get them at this moment and reload internals. So if I change something there, I already can get this information from the exporter. The other thing I have to add that is not done yet, as I talked earlier about, is to add the title and not only the description. So at the moment, I only get the description field from my profiles that are a little bit long and sometimes not really clear. And I want to use a titl as well. So one of the important thing I'm working right now is at the moment you have to dedicate it, you have to directly say the inspect exporter which profile should be scraped. So if I have a profile so Zudos, I have to add the parameter, OK, the module must be Zudos. At the future, I want to make it that you, if you call the root metrics path, then all of the profiles should be fetched. At this moment, you must be parameterize them one by one. The cross-built is nearly done, so I can not only run these locally on Linux machines. I also run them on Macs and on Windows machines. So I write this and go, and that is no problem. It's only more convenience. Side from my side, I have to add these at test because some of the normalization things I use are not completely tested at this moment. And what I also want to do is I want to implement the impact value of the defined book. So at the moment, I get the information from inspect. Is this passed or is it not passed? If it's passed, I give the value 1. If it's not passed, I give back to 0. So on Prometheus' side, I only know if the tests are passed or not passed. But I don't know, was it an important test or what is the less important test? So I also want to add these impact value I have in my inspect books. I want to add in the Prometheus fetching. So that is the thing I'm thinking right now how to accomplish that the best way. And last but not least, if my time at these systems are, if I find the time to write it, I want to write in a plug-in for OpenStack. So I can get this information like how many machines are running in my tenant at this moment, how many, if the bucket I defined have the right credentials, and so on and so on. That is on my to-do list, but it's not really dedicated to this point of time to my inspecting portal. So to make these long, long talk and short, if you have a system which are highly changed, please have a look on inspect. It's a tool, an easy tool, an open-source tool of outcast to check your environment. And if you have the possibilities to add these tests inside your deployment process. So if you have a CI-CD system already running, a GitLab or something like that, you can add these tests at the end of your deployment process. So I update to a new version at the end of these tests. I also want to run these impact tests. If you don't have this process or you have too much processes and not all of them are so good as you wish for, then you can use the inspect exporter and use Prometheus to monitoring on a daily base or in a minute base or something like that. So at the end, I would like to present something with the internet and the live preview, but I have problems with my network, so I don't can show it to you. So are there questions? I'm actually working for Deutsche Telecom as well, but as the group is pretty big, we do not know each other yet. I'm also co-maintainer of DevSec project. It's really great to see these examples. I enjoyed it. Regarding inspect and the idea to have OpenStack integration there, I think it's a good idea to talk to the chef and inspect developers. I'm pretty sure they are interested on that or maybe even working on this right now. And I have actually another question. Chef has a project or product, a Chef Compliance, which goes actually into this direction to get the compliance testing of your entire infrastructure. So I see you know it. I'm interested on the reasons why, what are the reasons why you choose to go this way with Prometheus instead of using this Chef Compliance. It's, as always, a think of time. So the thing is we are, at the point, I start with the development, with the inspect look up. We build a new monitoring system of base of Prometheus and at this point of time, has decided to use Ansible as default for playbooks. And we have to say, okay, at this time, the automation script to report directly to the chef cookbooks are not there. So the only way to get the information out of inspect was to run that manually and then put these informations to your Excel cheat or something like that. So on this timeframe, the best way for us to accomplish recurring monitoring of these statuses we get back from inspect was the best way to write these exporter. But yes, if you already use Chef, then that may be the best way to go. Thank you. Hello. Has there been any work done to tie or to develop a standard set of tests that ties into the stigs or ties into PCI or so on? Say a test that's labeled with the actual stick number and so forth. Okay, I'm not really sure if I understand you right. There are a lot of already built up baseline as we have a colleague there who are building up the DevStacks. You have a lot of, or not a lot of, you have some profile that are getting this information. If you have more directly, if you want to check more dedicated tasks you have, so you have a check for PCI or something like that, you can add them there. If there are not resources to get these informations, you can write resources with Ruby to get these informations from your system. So maybe there is a profile, but that is a point we can talk about later. Maybe we can help you to gather these informations and put them into your InSpec profile. So I hope I understand your question right. Did we get another question? Doesn't look like. So thank you for your patience with me and I wish you a pleasant day. Thank you.