 Okay Good morning Everyone is okay Okay, you do this small session as you see probably because PCI compliance is very close Topic and very let's say hermetic so Not many people doing this especially not many people doing this in the cloud and I think in OpenStack. It's only you know a bunch of people that actually did something with PCI Of course, there are some huge institutions financial institutions that are Currently deploying OpenStack and they have needs for compliance. That's why this talk was born actually Few words about me I have system administration background Like for 16 years. I'm working as CIS admin and DevOps Also for financial institutions for big companies huge companies. I was leading an afford to Implement a PCI in one of the biggest European payment processors I was a tech league on that project We did that successfully. We passed our first audit, but that was not on the cloud That was 2011. No one was even thinking about OpenStack Also Certification on public clouds was kind of unknown ground. So I don't actually have experience passing out it on On the cloud. So that's the That's the main information But but as you know PCI audit is a painful process. I don't know who of you actually went through the audit Okay, one two three kinder which means that you didn't pass or Okay, so how many people four people? Okay, four people. That's interesting So those of you that went through the process, you know that This is probably more unpleasant than visit to the dentist. So We are suffering a little bit about Mirantis usually I Showed this slide in Europe in America and probably in Asia in Hong Kong. Everyone knows Mirantis We are Mountain View based. We are probably the biggest independent OpenStack Consulting company or services provider Main office is in Mountain View. We also have offices in Harkov in Moscow, Saratov in Russia and the last addition is Poland when I am based there's a bunch of engineers in Poland working on OpenStack not only for European customers also for American customers because OpenStack presence in Europe is not that strong that in America, but it's changing one of the signs its Mirantis Presence in Europe So that's the slide for Europe. That's the slide for America. I hope everyone knows Mirantis We have together about 400 engineers and we did 60 successful deployments of OpenStack from medium small scale up to Very large scale and large. I mean thousands of hypervisors Okay, a little bit about Agenda I asked who of you passed the audit so not many of you So I bet that you're interested how to do it with OpenStack So who of you actually runs OpenStack in Production that's cool and who wants to run or who runs financial applications on that? I Know Jonathan runs. I'm sure Who wants to run financial applications? All right, so we will have like a common common ground here It won't be easy to do and I won't give you like You know golden way to do it because there's no golden way to do it as you know Also my from my experience European auditors tend to be more stricter than American auditors. So it might be easier in America to pass the audit than in Europe That's my experience from discussions with our customers how they went through audit how they implemented Compensating means and so on in Europe. It's really a little bit complicated than America. I don't know why Maybe because PCI is from America. So European institutions are trying to be More of a king than a king itself Okay, short agenda We'll talk a little bit about the state of compliance in the cloud. So what we have what we had in the past What is now what are the Approaches now Then I will show you what we did for one of the our customers for this project that I'm presenting We'll go through all the modules we provide and all the tools we provide and at the end. I'll try to Provide you with practical tips Practical tips by practical tips. I mean Our experience when analyzing entire open stack stack, let's say because at the beginning of the project We took all the internal projects of open stack. We analyze them according to the PCI 12 requirements and we made remarks what's implemented what's not implemented what can be done easily What needs patching enough of opens tag or using external tools and so on and so on I Will not cover securing the VMs and by the by VM. I mean your application So because Miranda's provides clouds we build clouds. We're not application specialists So we don't know what you run inside of hypervisor. So our Experience ends at the hypervisor level. So what's inside VM? That's totally Your responsibility as a cloud operator except maybe for networking which is like which is very The hypervisor networking is connected very much to the VM networking. So that's one of the Areas that we cover a little bit Okay, PCI itself all of you know what PCI is So For us what's important here is that we're covering PCI DSS version 2 obviously and We'll try to cover all 12 requirements. We have in in the standard General approach is to protect customer data. So the or cars data So the biggest impact is on protecting the data itself so processing data takes place inside of the Open stack infrastructure and the main point when the data Well, it's stored the process is probably are probably images glance and snapshots Unfortunately, there's no easy way to protect that part. It very much depends on the back end that is used in glance for example So it's really up to the implementation we don't we don't cover protecting the data at the glance level so what we had In the past is that like pre 2012 Let's say when I was going to the audit 2011 Everyone thought that it's probably impossible to be compliant using the cloud Well, then we didn't have open stack in 2011. There was no open stack really There were other clouds like open Nebula and so on but I don't think anyone was trying to go to the audit There were some Institutions trying to move their processing to public clouds like Amazon But I think that the the the assumption was it's not possible. So everyone was moving like Not crucial parts the cloud and no one was thinking about actually moving data processing can't processing into the cloud In between 2011 and 2013 Open stack was born and many institutions started to think about moving their processing To to open stack for example from VMware or from other virtualization tools But still there was no easy way to do it then in July there was Famous blog post on right scale blog by Phil Cox. I don't know if you know it Do you know that blog post about going through the audit on a cloud? if not, I Strongly encourage you to to to read that post. It was Phil Cox and he provided a way to actually Be compliant on public clouds which sounded then which sounded really amazing and actually I think they went through the audit with one of the customers. So it started to be possible Also this year PCI DSS published Cloud computing guides. So it means that the council itself sees the need for for Standardizing running financial applications in the cloud. This guide is mainly about public clouds. It introduces the the Something called shared responsibility, which means that The compliance responsibility is shared between you as a custom as a Customer using the cloud to process the data and the cloud provider. So if you read the PCI computing guide document Certified cloud provider would be you running your cloud. So in terms of open stack cloud We are CSP if we are running operating the cloud. That's our responsibility Usually if you are running your internal cloud you and your your data You are the same the same company But for for that document in open stack world CSP is just you or your CIS admin team operating running your your cloud So with our project where do we position ourselves? So in that blog post Phil Cox analyzed all 12 requirements and every requirement Or every his every advice starts with Statement rely on CSP to provide hypervisor or hardware Related compliance in every single requirement. He just moved the responsibility to the cloud provider, which means that the team or people running open stack are responsible for the security of hypervisor and hardware and your application is From the hypervisor level up to the data database and so on and so on. This is very crucial and To see how important that is that it's 12 times mentioned in the end that's a blog post that document So this is how usual infrastructure looks so we have some some kind of physical hardware server we have Networking layer storage layer. They might be on the same box or on other boxes when using quantum Then there's a hypervisor, which is probably most important part to secure and there's the VM So what our tools are doing is only ensuring that this part Is as much compliant as possible. We don't as I said before we don't cover what's Going inside the VM. So your app is your responsibility except for networking where we provide some Some means to secure. I'm afraid this is not readable Yeah Unfortunately, there's a lot of requirements as you can see so there's 12 requirements and what we don't cover is requirement one four six Nine and twelve because we can't we just can't do it. It's it's impossible. It's not the responsibility of the cloud operator the rest is Covered in with our tools in probably 50% So a little bit about the project so the project started because one of the customers Was deploying opens tag with our help and they were actually running financial Financial application on top of that they were trying to move from traditional virtualize a visualization into the open stack. So we started looking at the open stack and It's compliant and the state was really bad really really bad because usual way to deploy opens tag was to get Like a fresh operating system install opens tag on top of it and that's it Apart from that Open stack itself is not very secure. It's not written with compliance and security in mind. It's changing now After two years of development the developer started thinking about the security. There are For example, one of the biggest problems is skipping Passwords inside of the services. So all passports are plain text now there's an effort to Develop secure store for that when we started there was no secure store So we had to develop our own way to keep passwords and the same for Cryptographic keys and so on and so on After I think few weeks Mirantis Realized that this might be very interesting project. So we moved a little bit more engineers By the way, one of two engineers Were actually me and Bartek could be durah who is on this on this bar. So in case you have questions He's like a main technical person for the project So we we just true more engineers in sight we Extended the scale of the project Some of the parts like our logging infrastructure and lock correlation infrastructure We also used in other projects and we still using them. So this means that the project is is modular So we can take parts of it and use you don't necessarily have to use the entire project Originally this project was meant to be used as a fuel extension fuel is our deployment tool So it still can be used as a fuel extension, but it's also just a library you can use Not using fuel of course I strongly encourage you to use fuel Up to now two clients are using this this library to secure their their infrastructure Obviously, there are limitations. We have to be aware of this project is completely red hat or center as centric So no one to support unfortunately This is because of lack of time and at the time we started we were only supporting Red Hat because our customers running red hat also some of the tools we are using are only Available for red hat. So that was also a problem As I said, this is only for infrastructure as a service cloud So this is for cloud operators if you are interested in running Secure application this project is not for you. It's for securing Infrastructure, so it this is actually for infrastructure guys We don't cover any procedural stuff As you know PC is has a lot of procedures or policies to be implemented We don't do that in one of the documents in a checklist In every place we know that that should be procedure and we cannot do it in technology We state that that this should be covered by the procedure So we do we give a little help, but we don't write procedures So we only write code and and puppet modules and there's This might be a showstopper for some of the people But we assume that every machine inside of the open stack cluster is in the PCI scope So we secure them at the same with the same way. So all the machines are secured Probably controllers are the most secure part of the cluster which is natural But we also secure compute notes and so on and so on There's no redo. So if you apply the the puppet modules, you can't do the step back Of course, you can just enable In puppet you can enable backup of our configuration files But the tool itself does not provide any way of redoing things During the projects we had to develop some some patches for open stack for different components At this stage, we don't provide those patches Mainly because we were working on a custom open stack distribution So it would not be easy to apply those patches to Upstream open stack now we might do it in in the future We'll be thinking about it. But at this stage, we don't provide the patches And also we don't provide firewall management So we think that using security groups is not enough for for security in the cloud And everyone should also run host firewall Preferably with some kind of orchestration, but we don't provide that Okay, so what's in site? So we provide tools for basic operating system hardening Like password policies Hardening ctl hardening other parts of the of the system. We also had to write Proof of concept hardware security modules. This was written Because we had to test it. We were running open stack with plain text passwords So we wanted to keep those passwords somewhere else. So we worked this hardware security module POC if it's of course software security module because we don't provide hardware, but If you put it on very secure host, you might use it as normal hardware security module We also provide extensions to auditing system mainly in form of Around hundreds audit the rules for all crucial parts of the system and open stack and parts of open stack system Uh, we also implement log collection system, which is in this case lock stash with 0mq and kibana We provide our Rules developed by mirantes like filters and and And parsing rules rules for open stack We also implement secure communications inside the cluster. So you can do it yourself on your Networking girl, but you can use our modules to set up mesh of tunnels between between Controllers this is actually one of the biggest problems of this implementation because those tuners are point-to-point So every controller has to has has to have a tunnel to Every other controller. So in like in reference Mirantis open stack implementation. We use three controllers, which is okay, but if you use Imagine hundred controllers It might be a problem to set up the mesh of tunnels. It will be it will work, but it will be unmanageable probably We also provide audit tools So you can do a little audit before you implement the rules and then do another audit after you implement the rules the rules We use open scab the scab for that So there's just an extension You just run cli tool and you can see what's the state of your of your Infrastructure and we also provide documentation So tools itself it's The tool itself is a fuel extension in form of puppet modules Open stack patches. I hope it they will be available Also openscap profiles, this is called system readiness review So it's just well the tool produces html file with All the requirements we need and the state of the requirements on the system So you just you can just run it before applying After applying if you output xml you will have like Nice diff to see what changed We provide a documentation, which is an analyst of the state of open stack itself. So you'd not only have Compliance tools that you can apply but you can also see what's not Covered by us by our tool and what you should do yourself or using external tools or some other providers and we also provide compliance checklist so For your security team, you can provide a nice documentation that all that stuff was done actually The compliance is Is pc ideas as point 2o, but in some cases we apply stricter rules like with password policies We use nist recommendations with which are a little bit more strict In the future we would like to extend this project so you could choose what kind of compliance you want to apply So if it will be pc i this will apply pc i rules if it will be nist This will apply nist rules and so on and so on So in many cases You should have some external tools Especially the ldap It is possible to implement password policies and General access control inside of open stack, but it's very hard It requires patching of open stack. So the best way is actually Use ldap or all the catalog And I think that's what most Like meet size to big size organization do and then obviously you just push the The responsibility out of the cloud to the guys managing ldap, but that's probably the wiser way the best way to do it As I said we have Something called software security module, but it's not production ready. So In most cases you should use hsm or Use cloud keep this is a new project developed. I think by right scale and rack space uh, which is Secure storage for keys passwords and so on very interesting project. I think it was announced Last summit in portland Oh, all right cool. So Well, you're welcome to just change the room and go and and and hear about the cloud keep So obviously instead of using our Let's say crappy software security module Uh, you should use probably you should use cloud keep My cql or all the database we are concentrating on my cql During the audit is one of the crucial parts to be secured In this case, we don't store passwords inside those that the or current data inside that database So it's not that crucial but still SQL alchemy which is the uh, sql abstraction layer in open stack So sql alchemy, uh performance using ssl is not known very much. So that might be a problem So You should check that, uh, if you If you see my cql and ssl you should check the performance after applying the rules Okay, I think we have only five minutes left. So I will go through the uh, quickly through the modules So puppet modules we provide is uh, Integrity through id With the rules pre configured rules We also extend odd. We provide many many many interesting rules to To monitor the system activity Those of course those audit data goes into lockstash and you can just parse it or Using kibana you can correlate the deluxe There's a problem inside of open stack that uh, what's the state of logging framework for now? Not known Yeah, so one of the biggest problems with open stack is that there's no way to Like to keep track of the entire workflow. So if you launch a vm It goes through different apis and there's no easy way to track the workflow So what's happening like there's no one tag with every request that goes through all the apis There wasn't there was a project to to implement that. I'm unfortunately, I don't remember if it's working or not But this is one of the biggest problems in case of keeping your logs and Keeping eye on what's going on inside the inside the Infrastructure We also provide baseline module for security We disable some Many services actually after the after the default installation of red hat dot center There's plenty of services that are not needed We also tune cctl mainly ipv 4 stack networking stack We also disable interactive startup We set a password for single mode We prof we tune profile Default system profile And also provide how nice we provide issue or issue net Whatever you use and also ssh banner. This is needed. So you have to see the if you log into the system You have to see the banner that this system is PCI compliant We provide module for clama for for Virus scanning clama is not very good at that Last I think last stats is it it finds about 65 70 percent of of Threats, so it's not the best tool, but that's open source. That's what we provide you can easily replace it with something else Controller ip sec is the module for creating automatically creating tunnels between controllers We also tune system limits limits We add log stash plus Kibana web UI and zero mq. Kibana is in version two because version three Is html html five base So it needs to open a connection actually to open a connection to the browser Which in pc i world is probably impossible Most interesting part are predefined open stack inputs and filters We also tune pam stack password policies We add ssl support for rabbits Uh Root login Securing systems securing ssh And also change changing sudo entirely Like in our implementation, we don't we don't Enable sudo su for root because it means that you lose tracking of what's going on in the system What we don't include we don't include secure images for vm's that's totally up to you But usually that falls into internal procedures how to prepare images. So So that's why we don't do it. We don't provide glance protection in form of scanning images or scanning snapshots Etc etc, but we provide some guidelines how to do it And we don't provide swift encryption But now swift encryption is possible easily when we were working on that it was very hard to achieve now It's uh, it's just a future of swift Okay, like few tips Like you all probably know that compliance is not only technologies technology is one part I would say it's 10% the rest is ongoing process and your procedures the awareness inside your organization So these tools will not Give you compliance. It's probably you and your organization to develop all that stuff And Applying Your experience from virtualized infrastructure. It's not the same in the cloud. It's exactly. It's not the same because of the Dynamic nature of the cloud In my opinion is completely different approach. So I would not apply Experience from virtualized infrastructures onto the cloud. I tried because I have an experience in virtualized infrastructures with PCI and it didn't work with cloud Because of this dynamic nature automation is something you have to have it's just It's a must. So you have to have chef or puppet in our in our case. It's puppet I think it's impossible to achieve compliance in cloud infrastructure not using Some kind of automation If you're implementing that yourself you should get an expert on PCI and on cloud obviously And you have to find Cloud our auditor because cloud for traditional auditors. It's like Like a nowhere grounds. They just don't know what's going on. So this might be crucial and this will be hard I know it's changing but Year ago or half a year ago. It was very hard to find someone Actually understanding cloud cloud internals how the applications are behaving in the cloud and so on And obviously you should use quantum For networking because with nova network It's possible to achieve compliance, but very hard and you need additional layer of Automation just to manage networking Some notes from grease release Uh, I was I was talking about the security groups. So security groups are useless for Aggress filtering because it's buggy and greasy. It just does not work. We have a patch for that I'm not sure if we submitted it. No, probably not. Sorry Also, there is no tless support in vnc. So your auditor might not require it But if it's required you have to take care of that Uh, no image scanning shred again. So on so this is this might be very crucial So if you use snapshots inside your infrastructure, you might be leaving tracks of the data inside your infrastructure If you're not shredding images like snapshots Of machines and so on you might be leaving data inside the infrastructure. This is this is uh, very crucial Uh, also user management, uh after um removing user from From keystone or from From open stack. There's a lot of stuff laying around. So we have scripts for cleaning this app That's something you have to Take a look at The problem with ldap there's no granular access lights right in uh in keystone no easy granular access lights You can achieve that using additional tools Editing all those json policies with in every api, but there's no one point when you can manage that so ldap is probably Wise choice and there's no default zero access policy which is required by the pc i but that's actually easy to achieve This this is just a Few actually two slides on requirement 8.5 So you can see how much is left to you After using the tools. This is the analysis of the state of the compliance And all that red stuff is not implemented. That's something you have to do The same with 10 1 And okay, so the tools are not yet published. I hope they will be published When i'm back from summit So it's next week Not end of 2003 this is very Like a conservative date We have to clean clean them up a little bit because they were only used internally. So we don't want to be, you know ashamed of the of the quality That's probably all any questions This is probably the biggest problem the dynamic nature of the cloud and uh well automations Automation is the solution because what you do to your auditor you just show him your rules in your in your tool in in a puppet And they accept that they say okay, you uh, you just Make sure that all the agents are up everywhere Puppet agents or chef agents and the rest is inside your automation So that's the way to to to fight with the nature of the cloud the changing nature of the cloud So that's why we say automation is something you have to have any other questions We probably don't have time right Okay, if there are no questions. I'm available somewhere. I will probably at the mirantes boot So thank you very much Have a nice stay at the summit