 All right guys, if I can get your attention, please! Eyes up front, please! Eyes up front! Come on, come on, come on! I know it's coming close to lunch, Eric. I got you all hungry. Set me here on some very important things to say to you. I want you to listen clearly, and I want you to be ready, potentially, to answer some questions and answers and everything like that. I know you're going to be a good audience. If you guys choose to get up during the thing, or take off in the end, quiet please, be considerate of everyone else. These doors make a lot of noise, okay? So, I'm decided to move that right there. I'm going to leave it up to you, Setu. And thank you for a round of applause, Karthik. Thanks. Hello, everyone. So, welcome to Las Vegas, welcome to Defcon, and welcome to Cloud Village. So, myself, I'm Setu, and I'm a cloud security architect, working at Tides of Cops, and at the same time, I work for a couple of financial organizations as well, taking care of cloud native infrastructure security aspects. So, before going to the talk, this is my first presentation at Defcon, and if I'm not clear, and if you're not able to follow my talk, you're always welcome to reach out back to me, and then I can help, because I'll be standing at the end, and I can help you with your questions. So, in my organization, the most important tough thing that I felt, and our organization felt, is we have too many accounts, and achieving complaints across all these accounts is a huge complication for us, until or unless we have a $300,000 cloud security product, or a huge security engineering team who does all the automation and other things for us. So, we wanted to make sure all the security policies across all the accounts are standardized, and there should be a single central source of truth, and we wanted to automate it as much as possible, and provide all those reports to the complaints team, and at the same time, we wanted to be secure. So, at the same time, the number of accounts are too many, right? So, auditing each and every individual account at scale is a huge manual audit process for us, and we faced it as a... We were doing it on an ad hoc basis, but we were not able to automate it, but this tool from T-Mobile is a pretty open source and enterprise-ready tool, and we were using this for the past three years, and I was contributing for the open source project as well and thought like giving a presentation on this open source product. So, the solution is pretty much... We wanted to automate as much as possible, so we write all those policies and rules and controls as a code and give it to Packbot, and Packbot can go ahead and run all these rules across all the assets in your fleet of accounts. So, we have automated the compliance rules, and we have automated reporting on a periodic basis. We send out the reports back to our compliance team for our regulations requirements. At the same time, the best part is we don't just report it, we take actionable insights on top of our reports. So, if there is any public history bucket, Packbot will be able to automatically close that and make it private using automated patches. And quick introduction about Packbot. So, it's a platform for continuous compliance monitoring and compliance automation, where we do auto fixes for the vulnerabilities and compliance violations at scale. And you can always write new rules, and it's very pretty flexible. The REST API is open source. You can write your auto fixes as well, and you can customize it to your enterprise requirements. Coming to the architecture of Packbot, so we pretty much have a data collector and a rule engine. What we do is we use AWS Lambda function and cross-account rules to go ahead and log into each and every account and then grab all the assets from every account and store it in history bucket and then ship it to Redshift. And then what we do is we run compliance rules against all these assets. So, each and every asset will be having like 10 or 15 rules run against it. So, we'll be going ahead and monitoring the compliance at scale, and we won't be missing any resources that are non-compliant. So, initially we were using AWS config. It's like we have like 40 rules per account and each rule is $2. And just for the compliance of one account, it costed us like $200 to $400 for just one one account. And we use like DevTest, Prod and UAT for each application. We wanted to make sure in case of a breach, we wanted to isolate our... We want to reduce our blast radius and we segregated it on this production work environment-based and application-based. And then all these results will be sent to Elasti Search. And on top of Elasti Search, we have the UI which will be running using Angular and Node. And it will be a very powerful UI. And it's one of the best open source UIs that I have seen till now. And there's an application load balancer so that most of... 80% of the application is completely serverless. The cost for running this is also very less. We use like ECS, EKS, AWS, Fargate and all this container and serverless services. And at the end, we will be able to see a lot of cost savings because once I onboarded this tool at a couple of organizations that I was working, they were able to get rid of their enterprise secure cloud security rule tools, which were costing like $300,000 per year for them. And it's a huge cost savings for them. So you log into the dashboard of Packbot and here you will be seeing a single view of your entire cloud environment. So here you will be able to see the overview of how many policies are available and how many policies were failed, how many audits were failed, how many audits passed. At the same time, you will be able to see the total number of violations. And these all violations have been categorized in terms of high, medium, low and critical. So you'll be able to see the number of critical vulnerabilities and take immediate actions on top of these policy violations. And all these policies have been categorized across as four categories. So we use compliance, tagging, cost optimization and security. So an example for security is if there is a public read write bucket, it will be mentioned here. And the cost optimization is if there are any EBS volumes that are not used or any underutilized instances, you'll be getting on under those things. And then moving forward, you'll be able to view the overall compliance portion of it. And we use tagging. We wanted to make sure all our resources are tagged and then across all our accounts just for compliance. And because we have few applications under PCS scope and few applications under non-PCS scope. So we wanted to treat those applications with extra compliance requirements. So we add tags and then you will be able to see another number of vulnerabilities and how many assets were untagged in it. It will be a graph of number of compliance violations over the period of time. So you'll be able to see some pretty good visualizations and you will be able to sort the policies based on the categories that I have mentioned earlier, cost optimization, security and all other things. So when you click on one individual policy, you will be able to see how many resources and how many issues are under that policy. And also the compliance trend of it. So there are like 564 assets that are being scanned for S3 buckets, whether they are public or not. So we wanted to make sure we won't be another capital war on either any other organization breach due to S3 misconfigurations. So we will be able to sort them out based on the application tag as well. So we have internal tags based on the every application. And then you should be able to view all the admin panel and all the statistics. And it's a complete open source product. So pretty much I liked it in terms of an enterprise use. And if you go to assets, you will be able to see all your configuration management and number of assets that we are monitoring using Packbot across our 200 plus accounts. So this is a pretty... So based on this, we will be able to say that Packbot is not just for one account. It's not scalable or something because we use serverless and containers and it's all and ALB as well. So auto scaling is being used. So it's completely scalable. So we were able to monitor as many accounts as we can. And there is a only search bar here. Like you can search for assets using a single pane. So what the issue that we are facing is initially we used to use cloud custodian. And it doesn't have a UI, right? It just shows us the resources that got failed as per the compliance audits. So we were having tough time in searching for all these resources. Which account does this resource belong to? We were having the tough time. So using this, we were able to filter down based on the resource and then based on the regions. And then we were able to isolate the right assets. And we were able to search, like for example, if we will be scanning out all our EIP addresses externally, just to make sure no ports are being probed or being exposed since two ports, we use shod and monitor and then we scan them continuously the entire external IP range. So we find an external IP on shod and monitor and we wanted to know to which account that IP belongs to. So we were having very tough time in doing that and using this only search place, we were able to just put that IP and then hit enter, we'll be getting all the information about that public IP in which account it is and which server is using that and we'll be able to take immediate actions on top of it. So this is a place where we are able to see, like if there is a resource, we were able to see like how many compliance rules were being audited against that resource and we'll be able to see a security posture like overall it is 98% compliant. So we understand that there are few rules that are failing. So we'll see what are the rules and then we'll take, we'll ask those developer teams or the DevOps team to take necessary actions to make it compliant. So that's about the compliance pain and we tag it based on our internal requirements. So we have, tagging is something very tough even for developers, they just run a cloud formation tag and formation script and then they wanted to make sure they have to tag each and every resources. So initially we had tough time in making those developers to tag all their resources based on application and stacks, but we were able to achieve it and then we were monitoring the systems manager patch compliance. We have so many servers and we wanted to make sure all those servers are patched at scale. So we even have a trend. Initially when we deployed Packbot, there are so many unpatched servers online and then later we went ahead and we patched all those servers and you can able to see the graph trend increasing the more of unpatched servers or the servers that failed the audit reduced drastically. All these reports we will be exporting to the cloud console team internally so that the developers will take immediate actions instead of giving excuses. So the other thing is like if you want to export all these resources, policy violations, you can just click on the download button and then all those results will be exported in an Excel sheet which you can later share it with the developer team and other teams respectively. So this is an statistics page which always my compliance team goes ahead and watches all the time. So here we'll be able to see how many assets we are monitoring, how many accounts we are monitoring and how many policy violations are there and everything. So that's a quick demo. And when I mentioned about the policies and rules, they are highly customizable. You can, it's written in Java so you can always write more rules as per your organization requirements and as of now we have 100 plus rules out of the box. You don't have to write anything new. All these things will be installed by default and the deployment is also very smooth. So we have a Terraform script. If you run the Terraform script, the entire entire pack board setup will be up and running for you in like less than 15 minutes. And we have the overview dashboard which I shown in the demo. And the other thing which we noticed is we wanted to make sure we want, there are some accounts. For example, there is an application which is being shared by two business units. So we wanted to group those assets based on those business units and then we found the pack board assets group being very helpful for us and we run targeted rules as per that resource group or asset group. If you see here, we have resource groups and asset groups based on the application here. And the configuration management is as simple as that. You have a single pane of UI. You don't have to go ahead, log into the RDS server or log into the shipper and then make all the configuration changes. You can do everything from the UI itself. So we have the... There are three main configurations here. One is the batch and the second one is the rules and the third one is the API. So the rules... I highly work on the rules because I keep writing more and more rules for my organization. So I use a... This feature is very helpful for me as an engineer. And then you will be able to see and you will be able to also send an email to the respective business unit or the distribution list saying that this is a policy violation and you can send an email. You can just export it as an image and then you can share it with your business units if there are very immediate reactions that are required. And other things are like... I always get back from the developer saying like it's an... They wanted to have an exception for the next 10 or 15 days 10 or 15 days to patch it. So as a security engineer, I have to make sure like... After 15 days, I have to do a follow-up with them and then I have to make sure they remediate those security findings and it was a huge manual process and I wanted to automate it. I added a feature and Team Mobile Team is like damn awesome. They released it as a feature like you can add exceptions to each and every policy violation. And when you click on add exception, you will be able to set a date. Till what date that exception is valid. And if it passes beyond the date, again, it will be coming back to our policy violation queue and we will be doing a follow-up with the security analysts. We'll do a follow-up with the business units. Other scenario is like we have a security group or we have a DMZ network and we wanted to make sure the servers inside the DMZ network doesn't get policy violations just because they have a security group with 0.0.0.0.0. So the purpose is if it is in a DMZ, it should be open to the internet, right? And we don't want the pack board to keep on hitting it as a policy violation. So we can write sticky exceptions like you can configure it and you can create, you can select the asset group and then what is the exception name and then just write the expiry date expiry date till what time this sticky exception should be valid. You can set the date and then if you can go ahead and choose your target types, like for example, in our scenario, they are the easy to service and other container clusters. So we added the exceptions as well. And this is something among these five jobs, the last two are available open source and the first three are my company, the organization where I work, we wrote customization. So it's like we have used like a couple of container security products and we wanted to import all those findings into pack board for monitoring purposes. And we have a static code analysis tool and we want to make sure all those issues are imported as well. And we have a containers vulnerability scanning and host vulnerability scanning tool. We import all those things. So these are like some custom in-house built plugins that I have worked on and built at my organization based on the organization requirements. But you can always build these kind of plugins for your organization as well and then use them as a jobs in your pack board. So you have already seen this. It's like we monitor the patch compliance across all our server fleet and container fleet. This is an asset dashboard where we have all our assets, number of assets and we'll be able to get our single pane of view here. And the most important thing that the latest release is like we are creating auto fixes. So we don't have to just listen to those see those policy violations but also we wanted to make sure we take automatic remediations on top of those violations. We send out the pack board out of the box. We have like eight or 10 rules right now if there is a public bucket exposed to the internet. And it's not as an exception in pack board. What will happen is pack board will see the violation and send out an email to the business unit saying that hey this bucket is public and open to the internet. If you don't close it or if you don't request a cybersecurity team to add an exception this bucket will be private in the next four or five hours. So we will be avoiding huge data breaches using these kinds of auto remediations across organization. And as of today we have like EC2 instances, SSH ports or automatically the security group will be closed and the bucket will be made private. The redshift or RDS databases they'll be private. So we have a few and there are more coming in future. And you are always open for you're always welcome to collaborate and write some custom fixes and contribute to the open source project as well. And at the end I would like to just give a huge shout out to the T-Mobile OSS team who open source this fantastic cloud security product for the entire world for free literally. So a shout outs to them. Thank you everyone. If you have any questions you can always reach out to me. So you guys have you guys built an approval workflow around separation of duties for the exception changes or is that like an administrative passport? Yeah. So internally we are working we are using ServiceNow and then once that email is sent to an outlook that email will be triggered triggering a ServiceNow ticket and inside ServiceNow we do the Yes. We do the workflow inside ServiceNow. Thank you. Yeah. Close or there's a headache. Yes. They have close kind of Yes they are. So for example if there is a role which is saying like EC2 run instances privileges there is a policy saying like unapproved IAM role has EC2 run or unapproved IAM role has network privilege and next network permissions. So all those IAM roles I love that one actually it was very helpful for me. All those IAM roles will be listed under one policy and then you can export it and ask those guys to fix it. Yeah. Yes. So Yes. So yes and no. Yes in the sense we use Azure AD authentication and then oh yeah I can repeat the question. So he was saying like even the developer teams wants access to Packbot the DevOps folks already always want to monitor is that it as well rate. So the developers or DevOps they will request access for Packbot and this kind of open source tool providing access to every there is no there are some tweaks that needs to be done. So in my organization how I figure it out is like we use Azure AD and then we are using role based access controls using Azure AD and then I created asset groups and based on those asset groups I have literally isolated the business units and then I gave permissions to them. So we have to write as per our organization requirements internally. Yeah. Yes sir. Yeah. So actually we are not integrating with Sim but instead we are getting the alerts from the Sim and we are putting into Packbot. So in the other way. Yeah. Yeah. Yes. So we have a template the template will be in S3 bucket and you can customize the template and you can put your organization logo and then say this S3 bucket is violating your organization cloud security policies. You can completely customize it. Yes. Yes sir. Is there any limitation in policies? The limitation in the policies is like the number of policies actually. So if you want more policies we have to customize them and then we have to write it as per our requirements. Apart from that I don't see any limitations in the policies. Java. Yes. The data that you guys are acting on for enforcement when you find compliance issues it sounds like you're actively scanning all the resources and you find compliance issue. It's enforcing. Do you guys expose those issues also so if I have another system that is querying you and then I can enforce on my own is that possible tomorrow? That capability is not right. It's not there right now. Yes. Yeah I can explain that real quick in a high level. Yes I can tell it in a high level. So what happens is when a policy is kicked off a lambda function will be triggered and that lambda function will go ahead and initiate AWS batch job. The batch job will spin up EC2 container. It will spin up a compute instance and it will use this compute instance to run the batch job and the batch job will have a docker image which will be from the docker will be pulled from the docker hub and there will be two images pretty much one is the rules and the second is asset collection. First it collects all the assets for using the cross account IAM role. It collects all the assets from all the roles and put it in the put it in the S3 bucket and then the second batch job will be kicked off and the second batch job will go ahead and run all these policies and rules against those assets that it got collected. So and then send it to the Elasti search and then we were able to visualize it in pack mode. We have 280 so and T-Mobile said they have those in place. So yeah And how much time does it take to run? Okay that's a good question. So it took us 15 minutes for two 40 accounts so you have to calculate whether math. Yes. Whenever you write a policy any changes after? What about this thing? So how we do when you do the asset collection you collect all the you do you pretty much to describe instances describe EIPs and everything right. It will be collecting all those resources and then you will be able to you can set up the cron job kind of thing. So every two hours we run the public S3 buckets public as such ports. So based on those time we keep running. It's an automatic thing. It's not like that. No, we don't have to trigger it out. It's completely automated. I can show that. So you have to go to your admin panel and then you have to go ahead and pretty much use rules and policies and you can edit this policies and my bad. Okay. I should exit the PowerPoint right. Okay. So you just have to go to the rules and then edit those rules. So just click on rules and you have the rule frequency here. And once you click on the edit you should be able to set the frequency. How frequent you want to run it and then enter the value here. Okay. So there is a roadmap correctly in progress and they are going to support the next Azure and then they go to Google Cloud and I'm not sure about the VMware but there is always a chat available and you can always suggest ideas and then you can contribute to them. Yes. So it's pretty much it uses AWS SNS simple notification service and there if you configure the subscription to use an email it will send an email or else if you use mobile it will send push notifications to the mobile. So I'm not exactly sure about that but we can there should be some ways to customize the application. Yeah. Yeah. Thank you everyone.