 Cool. Thanks, William. So thanks and welcome to the talk. This is about developer experience and how Salesforce built an extensible pass using custom resource definitions, saving developers 5,000 hours. Yes, that number keeps changing. So a little bit about Salesforce. We are the oldest enterprise cloud computing company. We also hash one CRM company worldwide according to Gatner. We also been voted nine times by Forbes as the most innovative company. And our employees follow a 111, a philanthropy model. What that means is we pledge 1% of our product, time, and resources in the service of others. This is a look at the agenda. I'm gonna talk about a thought experiment, some of the key observations from it, and how we use the controller pattern and the CRM model to create a pass. We're also going to look at the object model that served as an abstraction over our SDLC processes and how it helped our developers. We're also going to introduce you to a new word called extensible pass, which I think I have not heard that term before. And we're going to show how the extensibility helped the small pass team grow while the infrastructure was evolving. We are going to show you the growth and the return on investment and share a little bit about a small recipe you can take back to your enterprise and probably achieve the same result. So let's dive in. So one find that we basically thought about or not just thought, but basically wrote down all the steps that it takes for our developers to go all the way from an idea to production, including every little approval and reviews required from legal change compliance and everything that you need to do on day two, which is, for example, ship a small change to production. And we asked ourselves, is it really possible to automate all of these steps while keeping trust top of mind? Like is developer agility with trust a thing in enterprise environments? So we basically look at all the manual steps and we saw a lot of duplication, a lot of process overhead, and a lot of service configurations spread across repos. Our developers were basically modifying complex JSON documents by hand. There was literally more time spent on infra management than actually writing business logic. In fact, our machine learning engineer was spending time writing Terraform code and Spinnaker pipelines, then actually writing some cool machine learning logic for our Salesforce customers. We took a step back and said, this is not gonna stay. We need to look at it and automate it. What we really wanted to do was model all the steps that it takes to reach production using a declarative API. We also wanted our developers who needed to create a service to just create a CRD or a declarative API, call a declarative API that would take the data model or the desired state of the service and call into various backend APIs to get the work done. We also needed a CLI that would call the APIs. And most importantly, we were a very small team. So we needed an extensibility model so that we, when we created this pass, it doesn't lock our developers and while the whole infrastructure was evolving, it lets the developers do their tasks. Everything boils down to what we really wanted was to hide the infra. So what we did was we modeled our automation engine in the form of a control loop. The control loop is essentially the same machinery that powers the Kubernetes core controllers. And what essentially is a control loop? It's basically a non-terminating infinite loop that looks at the desired state and the current state and it takes a very small incremental step towards moving the current state towards the desired state. We basically implemented the control loop using Kubernetes controller pattern. And we definitely looked at CubeBuilder. So I'm not sure if everyone know, CubeBuilder is basically a SDK for creating Kubernetes APIs. We use CubeBuilder to create the controller code and the scaffolding that gets generated. Things like controller runtime and the reconciled stuff that you need to fill in. We use the custom resource definition to model the desired state of the service. And all of the various SDLC components. And when we started, there were no backend APIs. So what we used was Go GitHub. It's a library for automating processes that are GitOps based. And I know this is not the best. This is not the we should aim for, but at the time when we were starting, this is what we had. So here's an object model for our abstracted pass platform. The first one is an application object which captures the minimal service configuration. It essentially, you can think of it as an index into your company system of records for pulling in more information. An application is also a combination of multiple artifacts that could be coming from different repos like Helm, Terraform, and Docker image. An application can be deployed into multiple environments like dev, test, prod. And we also model the act of creating a release which is basically taking a set of applications and their artifacts and bundling them together and taking them all the way from dev stage to prod in a desired sequence in a health mediated way. And for that we created something called a release object. So here is a look at the controller pattern. Our customers or internal developers would use a CLI. You can see at the bottom there's a command. The CLI would call app create and there were other various other commands. It would take a team name, which is a Salesforce team name, an app type which can be different types of services that you want to create internally and an app name. The CLI was used to create the custom resource definition or update various parts of it. The application controller or the release controller would look at the CRD and take action on behalf of the user. Like I said earlier, when we started there were no backend APIs so we resorted to calling into Git to get various tasks done. So what we want to show here is all the STLC components that the custom resource definition abstracts and automates and provides a default opinionated and secure configuration across all these components. So the first one is Git configuration. So this is basically managing what is in the Git who is allowed to access those repos and various other branch and other related configuration. The second is the source code. So you can use the pass to get a default scaffold of a specific language like a Node.js app or you can bring in your existing source code and use the pass to go to production. You can also specify or configure the CI configuration. So all the steps I'm showing are basically automated on your behalf. We also provide a default Helm chart that you can customize or essentially further choose from a catalog of Helm charts. We provide a CD configuration that is basically Spinnaker pipelines, configured in a specific way to meet the security posture of your company in our case Salesforce. We provided some default configuration for any cloud resource provisioning that they need to do and the service needs to be registered in our registry so that it gets the proper identity and various other Salesforce specific things. So that's also part of our continuous deployment pipeline. And lastly, we also talked about the release which is essentially how do I take all of this and make releases to ship code? That's essentially what every developer does, right? Each of these SDLC components that you see on the slide are also units of customizability or off-roading as you'll look in the future slides. So we created this object model which was essentially an abstraction that the developers, the internal developers interacted with. We were a really small team and the whole infrastructure that we were developing on was evolving. So what we, and we did not know all the specific requirements of all kinds of internal developers that were developing on Salesforce. And I'm talking about within Salesforce, not outside developers. So what we really wanted to do was create a platform model, platform as a service model, that did not lock in our developers into the platform. It allows the developers to break through the abstraction and take control of different pieces when the opinionated pass is not working for them. And for that, to enable that, we created an extension API. It's called module metadata API. So what are the advantages? The biggest advantage that we got out of the extensible portion of the passes that when we were developing, like I said, the infrastructure was evolving and we were a small team, right? So let's say the infrastructure team introduced a new feature and the pass team did not have time and resources to come up with a default configuration or a secure posture for that feature, right? So the developer is always empowered to unlock himself by off-roading on that component and making use of the feature. And later, the pass platform can see how that feature is being used and incorporate that in their default configuration. The other benefits are like, it obviously offers a composable pick and choose model, right? So the developers can choose the features of the pass that they love with the default configuration and they can further customize the features that they don't like the default configuration. It also allows the service owners to take on incremental complexity as and when they choose. So here's the layered model and extensibility using APIs. So the top layer you can see is the pass API which we just looked at as the object model consisting of the application and release CRDs and things like artifacts and environments. This is the layer where they get the least customizability and configurability and the most opinionated but safe and secure configuration. Now the pass as you saw is divided into customizable or configurable blocks called modules. These modules are nothing but SDLC components and they're basically divided into based on separation of concerns. So we chose things like in our previous slide, CI, CD, infrastructure as code, monitoring, security, approvals, anything. And this can technically be any arcane process in your enterprise that you care about and is hindering the productivity of your developers. So at the module API layer, there's a little more customization. Things that are not very common but developers might have to configure. And at the lowest layer, what we are showing is bring your own module. It's essentially a developer of pass that is using pass saying, I want to take control of this SDLC component. I want to configure it more. I want to be responsible for the upgrade and integration of that component with the rest of the modules within the pass. So in this layer model, you can see as you go down the stack, you get to take on more complexity. And that doesn't mean that you, the goal is that more and more people work at the top layer but they are not logged into the platform. So here is an example of an extensible pass modules using controller pattern. So what I'm trying to show here is that the SDLC components or modules that we just talked about are themselves implemented on the right using a controller pattern. So on the left, you can see an application CRD red by the application controller and it is configuring three modules, CI, CD and IAC, using default save and opinionated configuration. The interface that it's using is obviously the CRD for those modules and there is also an extension API called module API which allows the developer to say, I want to take ownership of the CI module or the CD module and then it can use the same CRD interface for that module to make further customization. So what is transparent abstraction? So in addition to building an extensible pass, what we wanted was a transparent way for our developers to interact with the pass. What that really means is that when the pass generates the configuration of the internal modules, it shows you exactly what it's doing. It shows you where the generated configuration lives and how to interact with it. And how that really helps is that when you want to have an opinionated configuration, when you really want to customize the pass, you know where to look. You know where to go for and what to do with that configuration, what tools to use to apply that and that's the beauty of transparent abstraction. So our small endeavour as actually growing by word of mouth, we really helped some critical teams reach production using the tooling that we provided. In fact, those teams have our tools in their runbooks for not only day one, but day two operations like shipping a change, monitoring. On the left, you can see the CD module, which is a customizable module of the pass. And the CD module, like I said earlier, right, we didn't have APIs. So the CD module was using Git boards to manage Spinnaker pipelines on the developer's behalf. What we are showing here is that the Git modules, bot, is actually the top contributor in our Spinnaker repo with 7,000 comments. And let's say if you all were to put a 10 minute time on every commit, that 7,000 commit amounts to around 1200 hours of a developer time. So what we really did was helped developers save 1200 hours of their time just doing Spinnaker pipelines. And note that we are a small team. You're only looking at 10% of all the services that we were targeting. So if you were to extrapolate, right, imagine how much time that would have saved. This is the same bot contributions across all SDLC components that we were trying to automate. It has totaled around 30,000 and it keeps growing. And if you were to do the same 10 minute math, it amounts to around 5,000 developers that are saved, or overall 2.4 years. Imagine if we could skip this time back to our developers. What would they do for your company, right? So we are still working and we are automating more and more things. Now, the hard question. How do you build a pass for your enterprise? There are lots of opens or solutions. OpenFace, Knative, Flux, Dapper, Open Application Model. I think CubeVela is basically an implementation of it. OpenShift, and there is a lot more. It's basically a build versus buy decision. Obviously, if you want to build, there are a lot of things you need to do from scratch. And if you want to buy, you need to figure out whether that fits into your enterprise. There are certain problems that the open source solution or the buying solution will fix, but it will probably create some more problems. So it all boils down to, at a given point in time, with the resources you have and the problem you're trying to solve. What is the best choice? I'm sorry, I don't have a good answer on that. What I'll do is share some of the things we did. Kind of a recipe in coming up with this internal pass solution. That doesn't mean we're not evaluating some of these other open solutions. Like for example, we are looking at Knative. But this is working beautifully for us. So, I mean, the journey how it started is basically look at all the processes your developers are doing. And when I say developers, I don't mean just the developer persona. It could be any persona, right? And see where are they spending their time. How can we make them more effective and are they really doing what they were hired to do or are they just doing busy one day in work that could have been automated? Identify how to automate those processes, right? Every automation technically requires APIs, but there may not be APIs. So we basically resorted to kit automation. But there could be other ways. Define a custom resource definition for the desired state of that abstraction. And make sure that you don't require a lot of inputs. I mean, if you basically created a declarative API that has the same number of inputs as all the underlying components, not really, you're not really solved or made it easy. Write a Kubernetes controller that understands that custom resource definition. And then for many of these things, unless it involves security, don't ask for permission, just do it. Divide the controller into extensible SDLC components with separation of concerns. So we chose like CI, CD, IAC, but again I said this could be any arcane step in your enterprise that is hindering the developer productivity of your internal developers. And the extensibility component we just talked about, right? It basically allows the pass to grow decoupled from the infrastructure. Try to assume safe defaults as much as possible. And this will change. Make sure, not only save defaults, but secure defaults, right? You want these developers to not have to think about not running as root or 3AZ maybe or whatever your enterprise wants to enforce as best practices. Expose a generic API for each of your extensible components. So what we saw in the extensible controller pattern for the module, we were showing a CRD as the extensible component, as the API for your extensible component. And these SDLC components that your pass is managing is going to grow, it's going to evolve. For example, your monitoring stack will evolve, right? So how do you expose an API that the customers can use and the pass can talk to without getting broken again and again? Expose an API to take over an SDLC component. So this is the module API that I talked about. And I have a slide at the end that talks about what that might look like. But it's basically just saying if the populated pass doesn't work for you, give me a way to take over so that I can customize it further. And then for the extensible components, make the API dynamically discoverable. So make an API to be able to discover APIs, right? Sorry, there are too many things in the recipe. So have knobs to turn off various components of your automation, right? So when you're starting this automation, things will break and teams will come after you. And you need knobs to be able to turn them off in the middle of night when your manager or somebody's manager calls you. And this is essentially saying that the controller needs to be config driven. And you don't need to like recompile code in order to fix things. So this is important. Be prepared to hear pushback from a lot of teams. So what you're really doing is automating manual tasks. And so those tasks or systems may not be set up for automation, right? They may not have experienced the amount of scale that you're gonna bring with your automation. And obviously you didn't ask for permission. So be prepared for that. And this is also going to help evolve those underlying systems to meet the scale of your enterprise. The safe defaults of your system are going to change. And if your pass is managing thousands of services, you need a way to roll out new, safe and secure defaults in a staggered manner. So you need to have a strategy on that. When we started this pass, we had a few customers in mind. We didn't really understand the landscape of every possible internal developer we were targeting. So what we really wanted to do was focus on a few couple customers and the services they had. And so talk to them in a closed loop and there's no really good solution. If you're not really talking to your customers, you're probably wasting your time. Think of all the tasks and especially the ones that are done by developers every day, every hour, make them really dead simple. One of the best things we probably did in our CLI was automate the process of taking something from a given GitHub Git branch all the way to production, finding what artifacts are living in that branch. And that's the most loved feature. Keep visibility into your past top of mind. So if you have created a pass that crosses all these various SGLC components, but you have no visibility to show using your tooling, it's not really doing a good job. The developers are running blind. And if they have to interact with your tooling to get the job done and then 20 other tooling to find the visibility into what's going on, then it's probably broken. And lastly, choose developer experience and customer feedback over everything else except security. So what I'm really saying here is that drive the architecture and design using developer feedback and customer feedback, developer experience customer feedback. So to conclude, these are the early days of our SGLC journey on public cloud and we have definitely gotten much further ahead. Kubernetes controller pattern is a very really robust way to automate processes. It's based on the time-tested controller framework code that Kubernetes is already using. In fact, one of the best things that has ever happened to Kubernetes, in my opinion, is the extensions API called Custom Resource Definition. It has technically created a whole ecosystem of infrastructure automation that people are using. Question your processes. Your extensible pass will drive the adoption of APIs and will identify API gaps. So this might be shocking. And lastly, empowering developers to build matters more than ever before. So with that, I wanted to thank everyone for coming here and I can take some questions.