 All right, I think I think we're ready to get started So first who who shot out of bed this morning said I really really want to see more Comcast stuff Yes Well, I'm here to fulfill your dreams. I think I'm actually the last Comcast talk of this summit So so we're gonna we're gonna shut it down after this My name is Charlie bomb. I'm a principal cloud engineer on the application platform acceleration team at Comcast Our team as you may have heard from several my colleagues given talks we're tasked with making developers lies easier developers experience is better and One of the many ways that we do that is by deploying and maintaining large scale rapidly growing cloud foundry environments Specifically, I'm here to talk today about our journey into open-source cloud foundry And specifically the Genesis tool chain that allowed us to finally be successful with deploying and managing an Open-source cloud foundry footprint that provides the user experience our developers were accustomed to with other distributions So we're gonna quickly begin talking about cloud foundry a Comcast You guys may have heard some of this stuff before But it provides some context around the scale size and speed at which we have to operate Which which allows us to move and force us to move into some open source We're gonna talk about why we went open source We're gonna talk about what Genesis is how Genesis specifically helped us and then I'm gonna hopefully push you guys into trying Genesis out and kind of guide you into a first initial deployment So quick overview. We're we're five years into working on with Bosch and cloud foundry a Comcast We currently have 23 foundations nearly half of those 11 of those are open source at this point We operate in private and public cloud. We're mostly VM. We're on-prem, but we do have a little bit of AWS out there And that's expanding we currently run about 40,000 ai's across and about 16 or 17,000 of those are in open source We run 5,000,000 500 million transactions a day. So it's very very high-scale high-demand Who's using CF at Comcast Well, we have several thousand applications in but some of the big ones that you guys have probably heard about or at least know from a product side Xfinity mobile Xfinity home security Comcast business and AI Q which is our virtual assistant and Jason Michener colleague of mine gave a demo of that yesterday in the foundry I hope you guys caught that Great demo there. So why open source? Well Comcast is trying to be and becoming a leader in open source communities We're really working toward that as we've seen in this summit and several others We've as we have established an open source practice led by Nithya Ruff who we saw speak at the keynotes the other day We have an open source fellowship program specifically For working on cloud foundry and upstreaming work into those open source communities and And lastly the demand for cloud foundry pot platform was far exceeding the budget allocations that we had come up with And we had to come up with a way to provide capacity to our developers in a real-cost effective manner So to give you an idea of our growth here We started this is just you know our journey is five years old But just less than three years ago We had about 7,000 AIs and we're now 40,000 plus so that growth is is extensive and we don't see that slowing down anytime soon So why Genesis Earlier in our CF journey During the Bosch one in two days. We made a couple attempts at open source. Yeah, but we didn't really find it sustainable Wasn't really sustainable enterprise friendly solution for the small team specifically that we have running cloud foundry We've seen other talks at previous CF summits that have addressed some of the issues here You know too many manifests to manage too large a manifest to manage No real easy way for secure secrets management and our renewal rotation You know a typical CF deployment is contains at least 19 CAs and 72 certificates So multiply that times are 11 foundations That's a lot of secret certs to manage The manual stitching of all the releases that make up a CF deployment You know numbers in the 20s, maybe pushing 30 for that And that's just CF alone. That doesn't even talk about you know Marketplace services There was real no simple way to create marketplace services that were consistent across all of our platforms The pain of on-boarding some new folks onto a team managing this type of ecosystem We found it very difficult to train people up on on some of the older ways that that open source CF was was deployed and summary it did not allow us to be able to really move quickly and Implement and scale CF as fast as our developers needed us to You know our CF has been growing at tremendous rates. We really needed to be able to respond quickly with a stable repeatable reliable CF paradigm So what is Genesis? Well, no, it's not the Peter Gabriel Phil Collins span from the 70s although land of confusion could really be used to describe a lot of what we were trying to do It is not the project used to make uninhabitable planets habitable in Star Trek the wrath of con although This case in this case the metaphor kind of works and you'll see why in a minute So Genesis is Bosch at scale. That's really the core here. It's secrets management. It's automation built in With concourse CICD Pre-packaged best practices kits And we'll talk about that in a second So what I mean by Bosch at scale Well, this is a typical breakdown of our architecture And Genesis allows us to use like a spoke type architecture model. We're a main Bosch. We call it proto Bosch That's the that's the name you'll see in a lot of the documents For Genesis and and the and the architecture. So we'll use that proto Bosch is deployed and its primary job Is simply to deploy other environment Bosch's So we have a main Bosch that handles that proto Bosch handles multi cloud multi CPI and Cloud config, but only for the Bosch deployments. It's supporting Each of these Bosch environments are sorry each of these environment Bosch directors deploy CF and All accompanying services needed for a CF and this even could even include CF CR So we currently have this architecture scale that to 11 So imagine prod Bosch times 11 and a dev Bosch up there. That's that's us 11 open source cloud foundry both on-prem VMware and public AWS all running with this architecture And I like to think of of proto Bosch sort of like Lord of the Rings, you know, there's one Bosch to rule rule all Bosch's Some so a secrets management some additional deployments from main Bosch proto Bosch Help Genesis live up to its potential one of those arguably the most important is vault Genesis integrates heavily with vault to create and store all secrets to every deployment an environment within this architecture Genesis handles this and creates them all on the fly and is part of the deployment for you All secrets are completely hands-off to the operator with the exception of AWS keys or other as an organization specific credentials and certificates, but for the most part Genesis handles it all and does it all for you Secrets can be rotated and removed and renewed easily using Genesis CLI and Vault is easily deployed with Genesis, and I'll show that in a little bit as well Automation Concourse is another Genesis kit Bosch deployment deployed from proto Bosch This concourse seamlessly handles upgrades to keep all of your environments identical Concourse using this spoke architecture allows you to have a runway Model of upgrades So concourse can push a new CF version or a new Bosch to dev Bosch if you like how that's working testing Okay, click a button push that out the QA Bosch you like that, and then you can click out and proceed to upgrade your The rest of your prod Bosch or CF environments All of these are based on github changes Back up backup integrates into this model as well We use shield shields a core component of this architecture This allows you to back up your vault your concourse your Bosch directors cloud foundry databases blob stores to webdab S3 GCP storage. There's a number of different plugins and features that that you can use to back up what you want where you want So what we're really trying to show here is that you have a central Managed control plane that facilitates the deployment and scaling of Bosch deployments at an enterprise level that that we need So we talked about kits best practices the secret sauce of Genesis is the kits kits are essentially a collection of YAML and Configuration files that provide best practices and defaults for deployments These kits are easily customizable and allow operators to have a say in how things are deployed So the Bosch Genesis operator is dealing with 70 to 20 line Genesis YAML manifests instead of 1200 to 5000 line, you know Bosch CF manifests Genesis handles all the YAML merging replacing and overrides for you Flexibility you can enable disabled features such as at the top of my head high availability DB for cloud foundry You can deploy autoscaler along with cloud foundry. You can create a Cloud foundry for a test demo environment with a small footprint minimal VMs You can do that all with with basic parameters within Within the Genesis manifest you have a lot of control over how your environment Over how your deployment despite the small size of these Genesis manifests Third you can build your own kits This is all fully open source fully available and Genesis has a command line tool to create our own kit and help you walk You through that so you can deploy other Bosch releases. You can deploy your own created in-house Bosch releases with Genesis It's perfectly flexible and available to to use any, you know Bosch deployment that's out there or any built in the house You can create it on kit and then push it to the open source Genesis community to add to that For example Comcast we created the squid Genesis kit We had a need for squid proxy and our product environment So we created a kitten and open source and it's up there and available So maybe you might be thinking won't they be opinionated If we simplify things down to defaults and best practices will they be opinionated? Well, yeah, they will be somewhat opinionated But that works for our favor here because they contain best practices and things that you don't really want to or how To worry about but it's easy to apply your own opinions by overriding defaults and the manifests with parameters or instance groups That will be merged directly into the Bosch manifest that is then deployed I encourage you to check out the Genesis community github repo for a full list of available kits But these are the ones we most like most most rely on so Bosch Genesis kit cloud foundry has a Genesis kit concourse The jump box is kind of integral. I didn't have it on the diagram, but it's essentially a VM Chalk full of every tool you would ever need to manage Bosch concourse cloud foundry shield vault and blacksmith so it's a it's a it's a VM where Instead of having tools and worrying about updating tools or installing tools on your own laptops or new hires laptops Everybody kind of works through a jump box and has common tools There's a blacksmith jump Genesis kit Blacksmith is a service broker that allows us to provide marketplace services such as rabbit reddus rabbit post grass Maria db And then there's a vault Genesis kit Prometheus monitoring kit and a shield Genesis kit for backup So how did Genesis help us? Well aside from all the heavy lifting that Genesis does on our behalf to simplify In scale our Bosch in CF deployments It drastically reduces our time to build out new environments Our initial deployment took some time to get a handle on Once we got our head around that though We were able since able to reduce our time to deploy an environment to less than a day about 16 hours So on the left is hours there so the first one took about 50 hours You know once we got a head around and went down and now we can we can We can kick out a deployment in less than a day So cloud foundry is really the best platform for stateless applications and demand for these services is growing exponentially in Comcast As you saw Genesis helps us quickly turn bare infrastructure into usable capacity for our developers in a simple and repeatable way Genesis also manages our infrastructure for us So we're able to look when we're looking to expand our team. We really focus more on developer centric hires So instead of spending time in Bosch, we're spending time scripting writing improving automating our platforms as Well as helping developers use our platforms more efficiently We really work closely with our developers So it makes sense to hire and have people on our team that have Developer background so we can jump in and look at Java logs look at heap logs look at code and help our folks Use our platform more efficiently Where can I get started well, here's a couple links that we can take a look at Genesis project IO. That's the real main landing page for Genesis Good documentation there unlike some open source projects. I've seen these are these tools are very very well documented The Genesis project I was great documentation on there has blogs has some demos and labs that you can go through good information The Genesis CLI is available at the github.com slash dark and Wayne Genesis community where all the kits are that's under github.com slash Genesis dash community and the safe tool Which I'll talk about in a moment is available sarkin Wayne slash safe So I'm obviously pretty passionate about this and and I'm hoping that some of you get inspired and go try this out And here's how you can kind of get started You can easily test and demo this this ecosystem. Unfortunately. You can't really be done on a local system You really do need some my ass So go grab some you know small chunk of I as some compute couple IP some storage doesn't matter if it's VMware OpenStack AWS Azure GCP whatever get a little bit of get a little bit of resources There's a couple prerequisite tools you'll need to really get started You'll need the Genesis CLI give you a link to that a minute ago Bosch CLI I think everybody knows where to get that spruce which is a yaml merge tool and then safe Safe as a vault wrapper and you're maybe wondering why why do we need a wrapper for vault? Well safe makes extends the capabilities of vault Allows your security to generate some random RSA pairs SSH private private key pairs auto generate random secure passwords securely provide credentials and Genesis requires it So next up fire up an instance of vault You can use a container if you want or even an existing dev test vault instance You could even use safe to create an in-memory instance although it only lives while the terminals open So I'd recommend it may be against that but just fire up an instance of vault could be when you have Target this vault using safe safe documentation is great for this from an empty folder Run Genesis init dash k Bosch What this does is is pre-stage is that folder and as a Genesis? Kit downloads the latest kit. That's on the that's up on github Pulls it down if you navigate to the Bosch deployments directory. You can run Genesis new ops And what that essentially does is it'll take you through a CLI wizard of sorts Asking you for your vault target IS specific information such as IP storage secrets credentials Clusters depending on on your eyes. You'll need to have that stuff available When that wizard is complete You'll have a Genesis manifest that is ready to deploy your first Genesis Bosch So simply run Genesis deploy ops and Genesis will fire up a VM Interacting with the CPI of what you chose in the previous wizard and you'll have a Bosch director ready to go You can then use Genesis to log into that director Once it's actually logged in it's like any other Bosch director you can start uploading releases stem cells Whatever you need to do You can even start creating deployments from This Bosch director using the similar workflow for other kits that you find you'll need to create CPI Cloud configs Genesis doesn't necessarily do that for you But it does handle all the deployments when that's once CPI in cloud configure all configured So this is essentially how Comcast built an architecture That allows a relatively small team to manage and build out cloud foundry at an enterprise scale to meet the growing needs of our developers and As you see in many sides We're big on open source of cloud at Comcast. We do have a hundred four already five currently open source repositories Our biggest win is patchy traffic control, which is part of our CDN Cooper healthy is another one. You can look at all the projects we have and contribute You know Comcast get up to IO And visit us at the booth while we're open the rest of the day I'll be there shortly after this if you guys have any additional questions We're hiring as you're probably well aware Comcast is a great place to work. It allows us to work and cutting edge stuff like this I'm pretty passionate about this stuff. I love this stuff. It works But Comcast also lets you fail and fail as long as you fail fast pick yourself up work on something else And keep moving forward and making sure you're focused on you know developer lives and experiences at least from our side Questions in a two-week period at one point we did three Once once we have basically hardware given to us. So we're mostly VMware But we don't control or manage the VM infrastructure. We essentially say hey, we need hardware when we get it From the minute they say it's ready, which means networks ready storage is ready computes ready We can build it if we start at 8 a.m We can almost have it up online by the end of the day and have people pushing applications in with it so that's how it's repeatable and people on board under our team quickly With this type of architecture Yeah, you can use that to rotate whatever you want you can target pass you can target Deployments you can do you have a lot of a lot of customization there what you can do By by deployment you can say if you're see a deployment Let's say West one rotate on this on this deployment It knows in vault all the past where those sorts are rotates them for you the shield so shield uses the Runtime config so it'll put shield agent wherever you specifically want it and then from a shield gooey You can say backup Bosch director backup bash on my vault based on the agent So postgres file backup whatever whatever it is you're trying to back up all through that all through a Interactive gooey, and there's a CLI to with shield So proto Bosch cloud config has all the IPs for all the Bosch director It's Bosch directors that it deploys and including like jump box shield wall concourse All the Bosch directors you then deploy have their own cloud config that handle all their individual deployments That's me Yeah Right Yeah Well, there's a multi CPI config within the proto and that allows deployment out to And that deploys a Bosch Proto deploys a Bosch director that knows it's going to be deploying to VMWare or it's going to be deployed to AWS That's all built in in the Bosch director itself So you don't have to maintain CPI configs on local on environment Bosch is everything's done in The multi CPI config on proto Bosch that gets pushed out to the Bosch director configurations Right or concourse, which we're you know which we do that as well. Yeah, you just say it goes through that that runway workflow Yeah, so new kits are released, you know regularly, you know look through them test them push them out into your you know The release notes are very thorough about what's changed You know recent changes I think they've moved away from you know console DNS to Bosch DNS So you push it out to QA or dev low environment test it around play with it And then once you're satisfied with that you can set basically a kit number in a manifest In a global manifest and you can start kicking concourse off and it'll deploy out to any ones that you choose Well, like I said, I'll be at the Comcast booth shortly if anybody wants to you know talk further about this stuff Thank you. Thank you for coming appreciate it