 Hi everyone, so my name is Paul Tchaikovsky. I'm a cloud architect at blue box and IBM company I'm Rachel I am an engineer on the QA automation team I'm Zachary slice. I work on IBM blue box So we're gonna be talking about a few things First of all, we're gonna talk a little bit about Why we're doing compliance and how we're doing compliance? You guys are a little bit of history of where blue bus came from what we did before acquisition and then what we've done through acquisition and You know what we're kind of doing now and where we're going towards as far as Compliance and how we're applying DevOps principles to it So, you know open stack is kind of installing open stack is kind of a solved problem at this point There's plenty of tooling out there to do it There is it's Still pretty difficult because you've got a like the architecture and deciding how you want it to work is still pretty tough And then the the day one operations is still quite tough There is a bunch of dev ops tools that are out there for installing open stack We use Ursula, which is blue box rotor. It is open source. There's open stack ansible There's chef puppet, etc And most of these will help you with like the day zero installation And then they'll also do some amount of you know ongoing operations helping keeping things in a working state, etc And then a couple of them the two that are involved actually also help with compliance and security If you if you don't if you're not currently subscribing to any sort of you know base compliance for your OS I do highly recommend the Stig they have for most OS is that it's fairly rel specific But a lot of it applies to Debbie and Ubuntu as well And they also have stigs for a lot of applications and other stuff as you got the stack my sequel and stuff like that And it's got a good lot of good really good defaults To help you make sure you're not really doing anything really dumb from a security perspective on your OS The relationship between ops and security is can still be pretty contentious You know, this is kind of how the security team sees themselves. You know the dev ops are up here You know shitting out all this beautiful Unicorn poop and they're trying to shovel it up and actually make sense out of it And it's not that unusual to see like your automation tools from an operations perspective and you're tooling from a Compliance perspective actually fighting with each other over, you know settings in an SSH config file or or whatever or you know As we'll go we'll go and set up one way security team will go and like change it next time as we'll run It changes again, and that's that's not really cool and so we're trying to bridge that and bring security into the sort of the dev ops model and and help give them ways to contribute to our automation and provide us specifications in ways that we can more easily consume and more easily test And yeah, as I said earlier both Ursula and OpenSack for Ansible Ansible for OpenSack, whichever way it is do some stuff Towards compliance OpenSack for Ansible does is based on stigs The blue box stuff is kind of based around some of the internal IBM Security, but we want to start looking at making it more like stig so that we can hopefully Open source a lot more of our compliance bits And like if you if you're but if you're buying OpenSack from someone or getting an OpenSack distribution from someone You really need to be making sure that they have some support for compliance stuff You know even like getting stuff like automatic password rotation and stuff like that that's pretty important to a lot of enterprises and OpenSack doesn't easily support that you know native in a default install And so it's important to make sure that You're able to do that sort of stuff if you do care about compliance, which if you're in here you probably do And then of course even if you have your OpenSack itself is compliant You also need to make sure your workloads are compliant, you know Just because your OpenSack is compliant to stig doesn't mean the random Wordpress sites or whatever whatever it is you run on cloud these days is Compliant so make sure you do the same sort of compliance to your workloads if you have to do disc encryption There are ways to do disc encryption Even if you don't want to go to you know all the way down the Barbican and low-level disc encryption there You can still you know encrypt LVM's of Cinder volumes and stuff like that So a little bit about blue box. We sort of we're I know it doesn't or more years old We started a blue box cloud as a project a couple of years ago And have had pretty good success with it In June last year. We were acquired by big blue Which was an interesting process But one of the things we got out of that acquisition was access to all of the soft layer data centers And so we went from having kind of two data centers We could deploy into to having 30 plus data centers we could deploy into and that was that was pretty cool But also as part of that a lot more interest in compliance and stuff came in because that's kind of one of IBM's big things is security and compliance And we did we were interested in that but not quite as formally as IBM is so we have We basically had primitalist security. We always considered putting hosts on the internet first So most most of our hosts always had a public IP. So we were very focused around Firewalling at the OS a lot of IP tables rules and stuff like that And then if a customer wanted something different it was easy to add perimeters firewalls Stuff like that, but it was to take them away and be surprised by them And then we've always had security as part of our testing pipeline like you know, we're running the elk stack We always check that we've got port 9200 on elastic search firewall off from the public internet and that sort of stuff Yeah, but we never actually had like really good formal compliance and testing So part of I being acquired by IBM is called bluewashing and That that brings in like doing your open-source software license audits making sure you're not bringing any GPL in and poisoning your code base Making sure that all the authors of the software you're writing our IBM employees or have Signed the appropriate licensing agreements, etc A lot of code scanning port scanning sort of going down the stack of security compliance And that's kind of my bit All right So now we'll talk a little bit about tech specs, but before we get into that So I was with IBM pre-bluebox, so I've seen plenty of changes over the past year and a half or so I try not to break things because if I do Paul gets very mad at me, so Gonna try not to break this presentation So what is a tech spec? So before blue box? you know the tech specs were a way of ensuring generic config settings and as well as compliant settings for security folks they were Excel spreadsheets in a shared storage space and Feast your eyes on the glory that is a tech spec so I had to compile these when blue box was acquired and That was very fun. So Here's a small example this one So it sets the no anonymous user access within my sequel database You can see the initial value. I would have to go in by hand and run those on every single machine that we had so Yeah, it was interesting times Of course tech specs were run by hand. So all the man hours wasted They're usually written by rote. So, you know with little understanding of the why and the how They'd also cause conflicts with running our software Just because you know stupid little checks and Last but not least the commands were in Excel spreadsheets So, yeah, so after blue box was acquired the tech specs Came down a little more So they're limited to just compliance and best practice settings Written in yaml stored in github. So, you know so easy caveman could do them We use server spec which are just our spec tests to run our tech specs which then report back to sense you and Page your duty alerts us if anyone changes anything they're not supposed to So yeah, needless to say things got a little better when one of blue box joined So a couple of the rules yaml based document. They needed to have a specific ID code Remove the complexity behind, you know compliance and the best practice guidelines as well as you know Just make making sure that they're easily readable and understandable A couple more rules So they should be linked to at least one ansible task or a server spec test They shouldn't be generic config settings. So none of that, you know making sure user exists They should describe the expected outcome and the reason behind it. So actually Understanding what you're doing and why you're doing it and they should have a link back to the original compliance document So here's a good example Written in yaml it has the unique ID code ops w1 and Has the good description Explaining what like why we're why we're writing this one? Here's a bad example. So this one It looks just like a basic generic config setting doesn't explain why the setting is important in the first place and Unfortunately, these were all written up in our new text spec checks, so We just you know went in did the same thing that we usually do and wrote it again So here's what it looks like in a Enansible So you can see that these are a couple of server spec checks Gfn 12 just stating that should be a file and gfn 13 is Stating that the password should be 16 characters long But does anyone see what's wrong with that? It's just checking for a word. It's not the best of password checkers So here's an example of what it looks like in server spec in sense. You sorry So I'm sure y'all are familiar with Uchiwa dashboard As you can see I did not fix this one. So I have to fix this after the presentation Here's what it looks like in our Kibana Dashboard so since your client logs every failure to elk and so we'll have that audit trail of every pass every fail for every hour and Elasticsearch is backed up to our object storage. So Maintaining our long-term audit requirements something like 270 days And some future work. We're gonna move away from server spec and move on to in spec So it's essentially, you know like server spec. It's based on our spec But it's much richer compliance DSL richer focused compliance on DSL So it increases our DevOps culture to include security and compliance folks allowing them to do test-driven development Here's one example So much like our tech spec rules. We have the unique ID code it has plenty of metadata so required by security and compliance and The description, you know explains the need for this check. So it's just as a sage protocol and a couple URLs so chef in spec and Paul is graciously provided the rail six six stigs But he's working on more to come so Get ready for that and Rachel Okay, so I'm gonna talk about what kind of testing goes on in the development and testing phases So this is a really blurry cartoon. I'm sorry. I couldn't find a better one But we can all relate to it right like Catch a bug see a bug fix it resolve it and then we're just gonna see a bunch more like that's just part of life And so during development or testing stages, we're just constantly deploying our environments With the latest packages testing upgrades deploying with or without certain functionalities in order to find any kind of bugs and address them as soon as possible so with every deployment we need to make sure new changes don't break anything and Regression tests are run to verify this they're made up of Tempests and extra end-to-end test cases that we've created for blue box cloud and After that we run smoke tests to figure out if the cloud is functional Are the basic functions of the components working? if not then the pipeline is paused to verify the build or the environment and Then finally we do performance testing can it perform up to standards? is it doing what we expected to do and Jenkins helps us automate all these test buckets and lets us know when it's passed or failed and We run this slot or this pipeline continuously through every deployment So now that we have a product and development stages is over. We're ready for production and it's time for a new set of testing and Before we get into testing tools. There's itc s104 which is IBM's own security policy standard and It's very important to IBM that we use it because it provides a consistency and security for all of IBM's product line and And sorry for this process. I use itc s policy for Nessus and AppScan So Nessus is used to scan for infrastructure vulnerabilities It focus on focuses on the network and supporting services it also scans for vulnerabilities during port scans and I like it because it has a simple GUI that we can import our profile into our itc s104 profile into and it can interpret that That profile and scan for the infrastructure based on its requirements So here's an example of Nessus would look like like for results you can see that it ranks the vulnerabilities by severity and You can also see so there's a history tab right there and you can also see the history of these scans and kind of compare your results and You can also export it and give it to someone maybe like a security expert or something and So if we go more in-depth into these vulnerabilities Nessus also provides us a description and a solution So right here. We have SSL certificate cannot be trusted It's description in depth and then a solution to the issue AppScan is a different type of scanner It scans UI and API and it looks for common vulnerabilities like the OS top 10 web applications And another cool thing that it's able to do. It's able to record and remember login credentials So that's kind of automated like that. And when it when you want to scan an API you kind of have to Set you have to use it as a reverse proxy And what it'll do is the test runner will Request or send some requests. It'll go through AppScan, which is a Reverse proxy it'll record those requests and then it'll test those requests But the thing with this is that it's only limited to Windows machines And it's not a perfect scanner like you're gonna find false positives so Here are some examples of results that you'll see so you have your security issues that are arranged by severity as well as advisory for each issue and Then fixed recommendations. How are you gonna fix it, right? And so me and AppScan really don't have a great relationship like We run into a lot of problems and the next couple slides are gonna be The kind of problems that I ran into So especially when scanning OpenStack APIs We kind of had this problem because AppScan is only on Windows so we kind of first we were like, okay, let's try and set up Tempest on a Windows machine and I Did it but it was weird because I had to disable some things and it just didn't feel right So the next thing that we did our next approach Was we kind of put Tempest on its own separate Linux box and then used putty to create an SSH tunnel from our Windows machine to our Linux machine and Continue to use AppScan as a reverse proxy So you go a Tempest SSH tunnel reverse proxy and then to your cloud and that's just that's a lot That's a lot of connections that you know it didn't feel it still didn't feel right like it felt kind of like not right so We had so okay, let me go back. This worked for one quarter It didn't work for the next quarter and then we had we were we had a deadline so we had to kind of figure out like What is an alternative and that alternative is Centrobus? And it's really easy to use like much easier to use than with the reverse proxy in the SSH tunnel and it's specifically for OpenSack API security testing and it aims to detect common security defects like SQL injection, LDAP injection, buffer flow, etc And it fuzzes everything and it's a new project And hopefully we're gonna give feedback in the future. I actually have some that I've been postponing it but and then also we want to add additional test cases to kind of match our ITCS104 profile so This is what it looks like to test on one template. It's called policy endpoint get Just this really simple one It'll show you how many errors how many failures your progress It'll also it also shows you what it's testing. So buffer overflow command injection integer overflow LDAP injection SQL injection, etc, etc and You can also pipe the results and it'll show you the errors and the failures. So in this example, we have the failures Defect type 500. I think it's really cool because it'll tell you it's confidence in it So it has a high confidence that this is actually like real and then the severity like it's low severity So whatever but not really whatever and then stats an overall stat So you have one low vulnerability and that's the end. Oh Does anybody have any questions? All right. Thank you everyone