 Back everyone, we're going to talk about GitOps in action now. Thank you. Hi everyone, I am Anil. I'm Billy. Hey, Nathan Kalbra. Hey, we are from Broadcom and we are here for case study on GitOps in action in Broadcom. So, we'll start. So, for intro, what is Broadcom software? What does SaaS platform engineering do and how do we use GitOps for all at one stage? So, this is our agenda for today and we'll go into details there. So, for the intro, Broadcom is a large enterprise company which was primarily into the semi-conductor market. So, anything that you do online, wherever you connect today, the more the chances are that your data is translating somewhere it's touching Broadcom technology. So, we have all the kind of semi-conductor tips into whether it's cell phones or data centers or connectivity anything which is out there. But then, you know, the company is about, you know, as of last financial year was about 24 billion in revenue and we invest about $5 billion in our R&D. Then, we transitioned from basically a semi-conductor, primarily semi-conductor company to a software company as well. So, we're diversified in there with the acquisition of CA technologies and Symantec and that's where Broadcom software came in. So, you can see our legacy like how we came in and then how the Broadcom software came in with CA and Symantec. So, in the software, in our Broadcom software, right, we are, you know, as of last financial year again, you know, we have $5 billion plus in revenue. And we invest almost like 15% of our revenue into R&D, which is, you know, again in, you know, hundreds of millions of dollars, which may be even larger than some of the company's total revenue itself, right. And we have, you know, thousands of patents and, you know, almost 80% of our workforce is into the R&D. So, in terms of our usage, almost 10 of our 10 global companies use us. So, we have, you know, SaaS software that is used by a lot of companies. So, we have infrastructure-specific SaaS softwares or security-based SaaS softwares with different portfolios. So, these are like, you know, you see there are AI ops and DevOps and, you know, network security and identity software. These are like software product portfolios which are sold as a SaaS service. So, you know, these products again, you know, portfolios have multiple products in there. And these are all revenue-based, you know, subscription revenue-based SaaS model. So, within Broadcom software, we represent SaaS platform engineering. So, what we do is basically all these SaaS softwares that we sell, right, for subscription online, right. These are basically all run in a standard platform. So, we came in from, you know, CA technologies and then with semantic acquisition, you know, platform engineering got expanded. So, we had these SaaS softwares, you know, which were running in, you know, at some point in Bay Metal data centers. And then we moved on to VM-based hosting as technology progress, right. Then on to, we started our shift to containers, you know, years back, right. So, you can reference our previous, you know, presentations in the OpenShift Common in 2017 and 19. So, we have been heavily focused on containerization. And, you know, so, you know, what we provide within Broadcom is, right, because there are so many software, you know, products and software portfolios for SaaS delivery, right. So, we create a common platform so that all the teams, all the products which are delivered, are delivering these services online, they utilize the standards, you know, based platform. And these, you know, for that we have, you know, our standard CD pipelines for spinning up infrastructure, whether it's a network or other resources, you know, containerized and non-containerized workloads and multiple application components that we create ourselves or we use, you know, where, you know, cloud provides the native strength we utilize those. But we still maintain, you know, security and standards because these environments for us, right. You know, can vary based on the, you know, compliance and security requirements, right. Something may be for FedRAM, something may be for, you know, payment security based, you know, highly sensitive stuff, right. So, in SaaS world, right, for customer data and customer privacy is supreme. So, there are multiple, you know, audits and, you know, things will go on. So, we try to address those as a standard platform and then, you know, your teams will utilize that. So, you know, this is our, basically, technology stack. You know, you might have seen, you know, all these icons as people had seen, right, there are like a lot of icons out there. So, we, you know, set up our own services plus we use some of, you know, we can't build everything on our own. So, we have to use, you know, some of the components and try to, but still we maintain the standards and how we're going to roll that out. So that multiple products can use it rather than everyone trying to reinvent the wheel, right. So, if there is one team which needs something, there's more than more chance that, hey, there are other product teams which will also need it, right. So, we try to create a standard and that prevents, you know, reinvention of the wheels, multiple people trying to do the same thing and then we have to clobber the solution, right. So, instead of that, we look at like, okay, what is it, something needs it. We collaborate and maybe they can, we are a first beta customer internally, right. And then because all these products are again, multi-tenant from their perspective, right. So, what we treat is internally within Broadcom platform in a team, we treat our product team as our customer. So, they are, for us, it's a tenant, right, in our platform, but that tenant can be again a multi-tenant platform which is out there. So, that's how, you know, we work out in that. So, yeah, so this is going to kind of simplify or repeat what I know was describing. From a product team, all the products that we support within Broadcom, this is what they get to see. They just need to focus on their continuous delivery pipeline. They describe how their application is going to be deployed and it goes into a SaaS platform. It has all those different pieces that this is just very simplified for what we're doing. So, yeah, but we're doing this for all these different product teams within Broadcom. 30 different product teams, some of them have different sub teams, lots of different developers on these teams, a whole bunch of different CD pipelines, checked with one of our team like over 7,000 regular jobs running on average per day. So, it's just a lot of stuff going on and we need to do that as efficiently as possible. You know, make this thing go very efficient. And so, some of the ways that we do that is using common platform foundation, make sure that the environments are as similar as possible across a whole bunch of different clusters and environments. Using reusable modules so that we can pick and choose because, yes, the truth is, not all environments are going to be same, production is probably going to be bigger than your dev or verify environment. Enabling as much self-service automation as we can, but we have to do that with guardrails. We can't give developers access to do everything in production. And then collaborating with the different product teams, things like using inner source so that if we're building a particular cluster and a team needs to scale a node pool, they can submit a pull request and actually get that change in, or maybe they want a new dedicated node pool for a component. So, here's kind of a diagram of some of the different components. We have a common VPC networking layer that connects various components. Depending on the environment, it may have an open-shift cluster, it may have database virtual machines, it may have a GKE cluster, and there could be even product-specific workloads, things like a bucket or service accounts or not everything is containerized. As much as we want that containerized, sometimes different product teams need to have a VM of their own. But through the continuous delivery pipeline that we provide teams, they can describe their application via home and get it deployed either to an open-shift cluster or a GKE cluster. But one of the things that we found is sometimes they need to deploy infrastructure. So, we gave them the capability of writing Terraform, putting that into the pipeline, and so they can actually deploy their infrastructure within that product-specific project and area to give them whatever either cloud-native resources or VMs or stuff that they need from their perspective. So, that was from the product teams. What about our team? All of this infrastructure needs to be created. We don't go through and manually create it. We wouldn't be able to scale that way. So, each of these components is built using a set of automation that's packaged as an automation so that if we need to build an open-shift cluster, we've got automation that can build that. We can specify how many nodes or stuff we need if we need to build database machines or a GKE cluster. Sometimes we can have one tool that will do multiple things. Sometimes we have to have specific tools specifically for something like a core networking. And those tools are built and run in the same pipeline method. So, we have an infrastructure's code configuration repo. So, in our pipelines, we define, okay, this is what we want that particular SAS platform component to look like. And it will go through a Docker-packaged container and will actually build that SAS piece. And then that tool is actually built using the same CI-type pipeline that you would see. So, it'll package either Ansible or Terraform or various tools that we're using as part of building the infrastructure. And then that then is used within the pipeline to build that. Now, one of the things you may notice is we've given product teams the ability to deploy to clusters and deploy their own infrastructure. But we actually managed database virtual machines, and that was one area that was missing in this pipeline. Teams would have to create a ticket and request a database, and it was a very manual process. So, I'm going to hand it off to Nitin that's going to describe how we solve that. So, if you look at the Kubernetes adoption phases throughout the journey of how exactly, from a manifest point of view, how exactly it kind of involved. So, we basically started with stateless applications, mainly backed up by replica sets, deployments. This was more than half a decade back. Packaging mechanisms were like YAML, JSON manifest, Helm, of course, coming into the picture, then comes in the stateful applications. Where, you know, the database applications or any of these, let's say, a Redis cluster or MongoDB clusters, anything around. Those were backed up by persistent volume claims with CSI drivers coming in with the new formation of the whole Kubernetes part. New packaging mechanism also kind of, you know, were adopted, something like customize, for example. Then comes in autopilot applications, operators. So, with operators brings in, you know, a lot of rich features such as like auto backups, app sensitive scaling, seamless upgrades. If you look at the packaging mechanism for the operators, again, going back to the same old YAMLs, but alongside, you know, with OL and bundles, which was a new thing altogether. So within Broadcom, these are some of the SaaS-provided operators that we actually support and have. And if you look at the whole breadth of it, we have starting from the database to PubSup, to security, something like a vault, you know, a service mesh. All of these are within the platform that goes into any of our clusters, whatever we have for the engineering teams to basically consume and deliver it or deploy it. But what Billy was stating, you know, we are the exact same pipeline. So the approach over here is, how do we basically take in the same operator method and use the same GitOps approach to deploy, you know, even if it is an operator or anything of that sort. So specifically for operators, we finalized and chose OLM to do the whole lifecycle management of the operators, which involves not just the initialization, but, you know, installation, seamless upgrades, anything of that sort. We introduced our own catalog, OLM catalog, that is. And we also wrote our own OLM bundler, which you can think of it as like an operator hub.io, but internal to Broadcom within the SaaS itself. And this catalog specifically has those specific operators, those certified specific operators which are scanned and being delivered, you know, through the same GitOps approach to do any of the deployment or upgrades. Effectively giving operand as like a service or think of it as like a cloud within a cloud wherein teams can now choose any given service and, you know, do the do the deployment and architect it around that particular part. So as we were scaling, here comes in, there was one of the slide earlier, right, 10,000 plus nodes and, you know, hundreds of clusters. We quickly realized we need to scale further, we need to do HA on the cluster itself, you know, communities, multi cluster or cluster HA. So as it gains adoption, there were certain of these particular queries that started coming in. How do I gain geo redundancy? How do how do I observe these fleet of clusters, right, from from a single from a single plane? How do I connect native community service across the clusters and that too, you know, from an internal native to communities, not like, you know, going outside, not we are the ingress or anything of that sort. How do we do that? How do we architect something like an edge computing? And a very classic example or use case that we recently saw was deploy a MongoDB cluster in US region, while have the edge nodes in Europe region, which is just read only. And, you know, yes, there were latency issues, which we kind of, you know, overcame, but how do we do all of that particular thing? Additionally to that from a get off point of view, how exactly do we deliver the same solution across multiple clusters? How do we, you know, adhere to multiple security policies? And how do we make sure all our clusters are adhering to that and the same solution is kind of, you know, being adopted by a specific cluster. There is no drifting anywhere within the cluster. How do we, how do we do that? So as a community, of course, you know, we need a solution for that particular one. Some of these solutions, what we identified have implemented and certain work in practice that is going on, particularly some Marina MCS, which helps us joining multiple clusters to have internal services communicate with each other. Something like open cluster management, which is still in progress, giving in, you know, observability point of view, pushing in common manifest that there are cluster level operators, which teams do not interact. But we basically as like, you know, within a team or maintaining a SaaS platform need to deliver. How do we do that? So open cluster management helps us around that particular area. Concept of her hub cluster with multiple managed clusters coming in, joining at the hub cluster, and you feed in something on the hub cluster, all of it trickles down towards, you know, all of the managed clusters. So just a pictorial diagram over here for the open cluster management on the left hand side is the same concept that I explained. And on the right hand side, you can see same security policies or any of the cluster manifest being feed it in into the same get off pipeline, what what any of the teams are using irrespective of whether it is for a communities manifest deployed via helm customized, we don't care, or infrastructure pipeline. It's exactly the same pipeline, which is basically meant for for deploying it across using the hub cluster and deploying it across the multiple clusters. And speaking of security policies. And I'll hand it over to another to speak further on. Hey, so, as we described right so our environment right basically as a team we focus on, you know, the SaaS side of things. So that means hundreds of clusters scale that you saw right thousand, you know, 10,000 plus nodes. And the environments are basically, you know, varied in, you know, because these are used with the SaaS applications customers are enterprise customers. So there is a strict or stringent requirements on security compliance and you know, regular audits and all that. So, whether that's a, you know, stock audits or PCI or like Fed ramp or like, you know, we are trying to go into, you know, like, say, I'll four, five kind of, you know, so depending on the environment type right there, you know, multiple environments that we spin up. So anything that we think of these these are like building blocks. So we think of, okay, we want to spin a one environment, second environment or whatever right and how do you, you know, put all the standards across all these environments or cluster one environment make consist of 1015 clusters all together right and the scale is humongous right so how do we do that. So, you know, we, you know, the DevSecOps approach there right because we think of this, you know, whatever platform that we create, we think of platform end to end right right from, you know, infrastructure layer that we spin up so it's not like okay we are just thinking of pipeline only for application deployment right infrastructure applications, you know, and on top of that security compliance whatever we can put that as a standard so that the it's easier for teams to adopt that right. So all the, you know, forward, you know, thinking organization would always plan for that because you think start small but you'll realize oh it's basically gone beyond a scale, then it's very difficult to handle at that time right that's where we try to stay ahead of the curve and try to adopt if there is one team needing as I mentioned right you know we think there's some other team which is going to need it so let's use that first team as a beta customer for us right so you know with the security same approach right so it was not like okay that you build it you run it right it's like hey if you're going to build it you are responsible for security too right so we think of shift left model all the way right you know even in the as Billy mentioned right you know whatever we provide we shift left in that scenario right we will collaborate will have a you know inner source will you know you can you know do a pull request you can look into what is being done right but it's a shift left here it's a shared responsibility model for the product teams right they'll be able to adopt right like you know operators or multi cluster services that we provide you can use those but at the same time they are standard right so it's not like there's not enough freedom there's freedom but there is God wills as well right so so from you know security evolution perspective right because as I said right we are SAS products used to run in data centers then we moved on to the you know we released hosting and then on to the hybrid cloud public private cloud and then on to being cloud agnostic so you know even security has you know evolved the same way right so we instead of you know someone some experts doing the scanning and coming out with the reports okay now teams could got into the scanning but they still it was like okay application teams or security teams so to know you know we have been sporting continuous and real time you know security and compliance for all our environments right up to the developer level right they can see what is going on right then the security champions can also take action on top of that and security teams basically have moved into more of oversight role rather than actually you know trying to you know implement everything right so we have built our you know DevSecOps you know for our entire SAS platform ecosystem by having a multi layered approach right hey where we gonna because we as I mentioned right we look end to end then we say okay here is you know where CA processes here is where CD processes right and then eventually this environment is getting deployed in production right so as you know like hey how operations teams would look at from the compliance from the compliance perspective some customers gonna look for here is my compliance and security needs how you are being meeting those right so we try to address that right right in there so we integrate security as well as compliance basically both are deployment build and deployment teams right like say you know with that mindset in you know we keep that as they are focused right that hey this is how things are gonna get deployed and then we try to automate right so because with hundreds of clusters if you don't do that then you know you have a lot of problems right then we offer teams because whatever tool we adopt right we try to make again shift left scenario included in that that hey teams will be able to access that and they will be able to manage and you know so you know to make life easier for them so basically result for us is basically we have a you know secure environments which are with the runtime defense in depth and you know we have granular vulnerability and security compliance for policies enforced so basically the one that does is basically our dev and ops teams you know they can focus on more strategic work than rather than you know mundane repetitive work so we look at a security at a holistic layer you know whether that's a you know network policies or whether that's a you know you know board security policies or whether that's a WAF policies TLS policies or SSL policies we push that out to the environments right and teams are you know can utilize those test those configurations and then roll it out to production so if it is okay TLS 1.3 is going to be standard then that policy is available in all the clusters they can just adopt to that and you know the policies already pushed out so basically with that basic still apply right because we create hardened images and that's where the all the stuff starts right we have okay you are going to adopt the standard images that we are putting in out there and then we use continuous posture improvement right with all the kind of scans and then here are some of the best practices right that we wanted to share with the teams you know we want to automate and enforce with automation so that there is less friction people understand what is being done and they are able to live you know make you know use of it rather than you know one sided you know it's like okay collaborative and then you know along with that right you know it's very easy to forget the dynamic running scans where you look at the policies and security because the containers will come and go right you may not realize what has happened so you need to be stay on top of that right like hey those policies prevent your you know or you know a security environment during the night also right like in that way something came up was it compliant or not right take action on that one if it was something which is not indebted and then more importantly right like there will be always some new technology tools API's which will deprecate and all that right like you know it's everywhere right whether it's a cloud API's or some product API's or even you know even within Kubernetes right you have to stay on top of that so that's our basically how you know some of the best practices we wanted to share with that thank you and you know we'll welcome any questions great thank you so much