 If you haven't heard of Intuit, just a little brief introduction about us. You may know TurboTax MailChimp, but we run services on Kubernetes modern SaaS and we're moving billions of dollars through our systems as well as billions of machine predictions and such things every day. Amongst the five platforms at Intuit that we have as teams, I mean, and myself are from the Productivity and Experiences platform and here are the numbers you can see to scale of what we attend to on a daily basis for our engineers at Intuit. And finally, Intuit's big in the open source community. We just received the end user award in 2019 and you may have noticed in the keynote we got it again this year. We're also very involved with many projects as contributors as well as maintainers and these are just some of them out there. And quick agenda. So we're going to go over a little bit about our history, what our landscape looks like at Intuit, discuss a little bit about the problem and then we'll go over a solution, show you a demo and do a Q&A. So with that, we'll go ahead and get started. So it's been on this journey for quite a while. Back in about 2013, we've got, all right, cool, sorry. So yeah, okay, we started very narrow, we declared we're going to be all in the cloud. So we were getting out of data centers at that time. And we started with just a few applications, a few of our services and once we had success with those, we decided about, let's say 2017 or so, dates are years are a little hazy at times. We shifted the scale. So we had such good success, we're like, we're going to push everyone now. Let's all go into the cloud. So there's much broader adoption within the company. So engineers were creating all kinds of cloud resources, AWS accounts, whatever they needed to do, they were pretty free to meet the needs of their services and applications. And then we just stumbled into the kind of, I think a problem many of you may have had working at larger companies and you end up with a large EC2 fleet and that gets to be a bit painful to manage, which about I think 2018 to around there led us to containers, which obviously led us to Kubernetes and scaling up on that. So all our services are pretty much in Kubernetes on containers now. We don't really run as much in EC2 anymore. And that was a modernization of our services. Now we've had great success with that. We were still left with a lot of stuff in AWS for lack of a better word. So we've been going down this path of modernization of that while applying similar principles that we did with our services. But in order to get there, we had to take a look at our landscape. If you deploy anything in AWS, you know, you just don't deploy one or two little things necessarily. You're going to end up with IAM roles or whatever other supporting types of resources. So for a lot of our services, they had to have RDS, SQS buckets, things like that to all the kinds of persistence and messaging you would need to do their job. So we categorized that as supporting resources. Then we had some things we were using called resources that weren't really container-based. These would be like SageMaker or Amazon Connect or something like that without, couldn't put those in containers. So we just had to kind of figure out how to deal with those. And finally, we still had some what we would consider legacy, not necessarily legacy from the perspective of the technologies, but legacy to us because we couldn't move these into containers necessarily. And we weren't ready to just completely turn them off or do something else with them at this point. So with that in mind, what was the problem we were seeing out there? Well, first, we found a lot of engineers were making various types of solutions for themselves, which is going to happen in a heavy engineering org. In this first column, we're looking at the orchestration column, if you will. It's custom automation, so from soup to nuts, doing everything they can possibly do to manage their cloud resource. And then they would get down to just doing something in Jenkins. I just wrote a little script and just pop it into that whole workflow good enough. And then you've even got, as well, the manual people. This would be laptops, my bash script, maybe someone goes and counsel and does some random things they needed to do, or just deploying even a cloud formation stack in there. And once we found that, we looked at what was it they were using. Well, on the whole cloud formation a lot, we had some use of cloud development kit, and we also had use of Terraform as well. And it could be across these different orchestrations, they could be using cloud formation and cloud development kit, or cloud formation and Terraform even as well. So lots of different solutions using lots of different infrastructure as code amongst our engineers. And they've traditionally had a lot of freedom to pick these different things that into it to service their abilities. So I felt a bit like this, we're chasing a lot of wires, looking where they connected, who was connected to what, finding out where it all went. So we step back, said we need to come up with a few principles here. Otherwise, whatever solve we come up with is not going to be able to solve everything, or at least give the engineers enough of a platform to do their business. First thing was to align with our existing standards. They may be internal standards, industry standards, but whatever those were, we need to make sure we were within guidelines of those. If we were going to develop anything, it should be as open source were possible, as you saw in the initial slides, we are very open source friendly now. So we want to continue with that path as well. Finally, whatever we did, we had to be cloud agnostic. You don't know what's going to happen one, two, three years. Do you stay on AWS? Do you use something else or whatever? So we didn't want to put ourselves in that position where we couldn't move. And infrastructure as code with GitOps, a lot of people, little GitOps, they use Git, but they're pretty much just running a little orchestration engine half the time where they're just running a script or something. So we want to bring that GitOps experience to that infrastructure as code as well. And isolation from the cloud environment, meaning we didn't want anybody touching it with their manual laptop or going directly to their cloud accounts with Jenkins or something like that. So we realized some things were missing. We could certainly align with existing standards and we could already isolate from the cloud environment in multiple ways. But we didn't have anything that open source really met the needs of the engineers being able to pick whatever they wanted. Whatever we were currently doing was definitely not cloud agnostic. And then they weren't using infrastructure as code with GitOps. That was sometimes just foreign for them. So this led us to need to create our own service. So we'd like to introduce our service called Cello. And what is Cello? It is an engine for cloud deployments. It lets us run infrastructure as code with GitOps. It gives us that isolation from the cloud environment. And it is open source. And with that, we are going to run through a little demo. So I would like to hand off to my co-presenter, Amin. Thank you. So before we actually get into the demo, first I want to talk a little bit more about where Cello fits into our overall ecosystem from a developer perspective. So first we are going to walk through a brief hypothetical scenario for a developer that wants to work on a new project. Let's say I'm a developer and I want to make a new Lambda project that into it or something. So the process from my perspective would be I'm going to come to some internal developer portal. This doesn't have to be like anything fancy or anything. We just need some UI so we can ask some questions like what type of thing are you working on? What AWS account do you want to deploy to? The kind of questions you need to know when starting up a new project. So we take that information. We end up sending it to an orchestration layer. The details of that orchestration layer don't matter. But essentially we need to talk to you a bunch of different systems because we need to set up infrastructure for our users, right? Because when I make a new project, I need a place to store my code. I need a way to deploy it and I need that nice integrations that I have continuous integration deployment. So the first thing that orchestration layer will call is GitHub. We need to start with our code. So we'll make a couple repositories for our project. For the sake of consistency with our services, we end up making an application code repo where we'll store our infrastructure as code as well as a deployment repo where we store our manifests. I'll get into some of the more specific details about what I mean by manifests when we get to the actual demo part. But we basically in a similar way to Kubernetes manifest, we have cello manifests. So once I have my GitHub repositories, then we end up talking to cello. I have these repositories where I'm going to be doing some cloud infrastructure. I'm going to be writing cloud infrastructure code. I need a way to deploy it. So we end up calling cello and telling it we've got this new project going on. We're going to be deploying things out of this repository. Cello will do some work, give us the necessary things we need in order to do those deployments, and it will give us back a nice cello project token. That token will allow us to essentially make calls to cello so that we can deploy specifically for this particular project. So we'll take that token and then we'll use that token to talk to Jenkins. Jenkins is our CI CD platform that we use at Intuit. This doesn't necessarily have to be Jenkins. This could be any CI platform. It could be GitHub Actions, Travis, whatever. As long as you have the ability to store secrets really, it could be anything. So we will end up creating a new Jenkins job for our project associated with that GitHub repo that we made earlier and store that cello project token so that that way our Jenkins job will be able to call cello when it needs to. And so we'll make all these different pieces of infrastructure and then we'll go back to a user, either send them an email or give them a message on a UI or something. And we'll say, hey, we've made all this infrastructure for you. You should get started on coding or whatever. Cool, but what does that look like for the actual developer on a day-to-day experience? So the process for actually developing code and actually writing your cloud resources ends up looking a little bit like this. So first things first, I'm a developer working on my project. I'll write up some code. I'll make a PR, get my teammates to approve it, and I'll get that code merged into master. So the moment that code gets into our master main, whatever your default branch is, so the moment that code gets into our default branch, we're going to kick off a Jenkins build. So immediately we'll go through, we'll start our CI pipeline, and it'll do all the things you would normally expect. Let's build our infrastructure as code, make sure it passes all our tests, make sure it passes any security checks or linting rules, anything like that. And then once we've made sure all our code looks good, we'll package it up, and then now is the part where we get back to our actual deployment process. We know our code is good. We know it's all ready to go. So now we're going to deploy it to our various accounts. So this is where Jenkins will actually use that token. We talked about earlier. It'll take that token from, or take that token, and then use it to call the cell APIs and tell it, hey, I need to do some operations. I need to do some stuff. I need to see the difference between what I previously had in my AWS account, for example, and what's now there. So Jenkins will end up calling cello. Cello will perform this operation, and it will end up streaming the logs for the operation back to Jenkins that the user can see, oh, cool. If I go through with this, I will be adding this new Lambda, or I'll be making a new SQS queue or whatever. And then once they see that, they might say, yeah, I want to actually deploy that right now. Press an approve button, and then same thing. Jenkins will end up calling cello. Cello will go through the deployment process, kick off that build, whatever infrastructure as code process that is, whether it be like Terraform apply, or CDK deploy, or just whatever cloud formation, or whatever thing it needs to do depending on the infrastructure as code languages. It's going to use that, it's going to make, run those commands, and stream those logs back to Jenkins, and the developer will be able to see, oh, cool, my resources are deploying, awesome. And so through that whole process, the developer within as little as potentially five minutes, depending on how complicated their infrastructure is, they'll go from, I merged a PR, and I got my code into my default branch to, oh, my resources are all deployed and ready to go, and I can actually start testing against it. I can actually start making API calls to my Lambda, or whatever, whatever they end up needing to do. So that's what the ends, that's how all of this kind of fits together. So now I will get a little bit more into cello itself. So we have a brief demo, kind of showing what cello is, how it actually works under the hood. Basically walking through the process of what it, it's not going to actually, oh, it's a slightly different, oh, it's a slightly different screen size, I guess. That's fine. It's not actually, okay, whatever. So we're basically going to walk through this quick demo. So I have it pre-recorded just because I didn't want to worry about network connectivity or anything like that. But essentially this is what I would be running on my computer. So for a little bit of context here, the idea here is we have a sample CDK application. You don't need to know necessarily CDK in particular. This could be any infrastructure's code language. The thing you really need to know is I write some TypeScript code, it makes a queue and a topic, and we want to get that into our AWS account. And so I have cello running locally in this terminal right below this code. And so what we're going to be doing here is we're kind of going to walk through the API calls that would be happening if I want to actually see how I could different deploy this code to some sample AWS account I had. So now we are going to skip forward a tiny bit. All right, so the first thing that would happen is we need to create a project in cello. So going back to some of those principles we're talking about of separating out our, isolating our environments, grouping, making sure we follow infrastructure's code as a main principle. We want to create a, we want to create this nice little project. That way you can say, we've got this project here. It will be deploying out of this repository. And anything I do, any operations I do related to this repository, we're going to basically be using, we're going to be deploying by reading some information from that repository and deploying to it. In this case, we happen to be using the cello project because we have some example, example infrastructure manifests in that repository, but this could be whatever your repository you want to use. It's just a matter of coincidence that happens to also be the cello project repo. So we would end up, I'm going to skip ahead a tiny bit. So I keep talking about these manifests. I'll talk a little bit more about that. So essentially kind of like how in Kubernetes you have the idea of a manifest kind of describing, oh, these are types of things we need to deploy. We have this idea of cello manifests. So like if I have a couple of different environments and I have a couple of different things I need to deploy and I want to deploy in a certain way, we have a couple of different manifests to say, oh, we have this, we have this CDK application. It's going to be, we're going to be using CDK. We have a couple of args we need to pass whenever we do this deployment, some environment variables. We have, we want to use this particular project. We need to do, we want to do these types of operations. So we'll declare this manifest that has that information. And so these manifests will live in that deployment repo that I was talking about earlier. And so we'll have that set up. We'll call our create project API and now we'll get back this cello project token. So this is the token I was talking about back in that initial onboarding overview I was talking about. Essentially, the idea behind this token is we want to have a token specific to this particular project because we want to make sure if I'm doing the operations I'm really only doing operations specific to my particular project, my particular application. So we make a, so we'll make a token and we'll end up just saving it somewhere so that later when we actually call some of our other APIs later we will actually be able to use it for this project. So then we'll end up saving that forward a bit. So we end up saving that away. And then, so next step is we end up creating a target. So the idea behind targets is you might have a couple of different places you want to deploy to. Maybe you have like a pre-prod environment and like a production environment or like maybe you have different environments or whatever and going back to the idea that separation of concerns we want to make sure we use as little permissions as possible when we do different operations. So the idea here is we will create a target specific to each thing we want to deploy to so that when I do a deployment using my QIL target or something I only have the permissions I really need to do that. So in this case we are creating a target here and we actually say we want to scope it down to this specific AWS role or we want to use these specific permissions. And so we end up creating that target. The names for these projects and targets are just kind of sample ones. You can name them whatever you really wanted to. Probably something more useful to like how you want to actually, you'd probably name it something like the name of your environment or the name of your account or something like that. So anyway, we end up making this target. And so now we've done these initial pre-steps that we do as a one-time operation. Now that we've done that, we can actually get to the actual, the actual main operations we want to do. So the idea here is we're going to take that cello token we decided earlier and we end up using that as our authorization so we can call our main operations API. And so the idea here is we want to do an operation. We know what project we want to, what project we're using. We know what target we want to deploy to and we'll tell it, hey, we want to make a new deployment. We have this manifest here at this in our GitHub repo. Here's the SHA that we actually want to deploy. And so it will, and so we will call this operation's API and cello will do a process of looking up that manifest and actually kicking off an operation. So what do I mean by operation? Well, an operation essentially kicks off in our workflow and the idea of what's happening here is based on the configuration that we had in that manifest, we're going to kick off a new workflow and we're going to actually do whatever that workflow, whatever that manifest tells us to do. So in this case, this API call is actually kicking off a CDK diff, which is essentially telling us, is essentially how CDK lets us see, oh, here's what you previously had and here's what you, here's what your changes would do into your account. So we send the API call over and we get back a workflow name, basically just the name of the workflow that got kicked off by cello. And so now once we have that workflow name, we can actually see some of the different details about the status of that workflow. So we can actually see things like the, we can see like the status of that workflow, we can see did it succeed or not, we can see, we can see the, yeah, so we have an API to get the status of that workflow and we can see things like the logs if we wanted to. So we can see this one already succeeded and we can actually get details for the logs and these logs will kind of show us what's happening. We can actually see, oh, we use that cello token to talk to Vault and get some, get a temporary AWS credential. We can actually see it pulled down our CDK code and we can see, oh, we did some bootstrapping and then there is a giant stack of texts at the bottom saying, oh, these are the resources you, these are the resources we're adding or deleting or whatever. This would obviously look different depending on what infrastructure is code language you're using and what exactly you're trying to deploy, but that's essentially the main idea. This, in practice, you probably wouldn't use, you probably wouldn't want to read it like this. You probably would want to use, we have a streaming API for the logs also so that if you want to just like see it directly in Jenkins or like a CNCI or like something like that, you could use that also. It depends on what you're ultimately trying to do, but that's essentially the main idea. We want, by using cello, we are able to create the project and targets for our specific, for specific use cases and then when we want to do new things like make new code changes, we end up performing these operations and we can actually see how those operations are going. So that's essentially the idea there. That's it for the demo, so now I'll go back to the presentation. All right, so that was our deep dive into cello. Now I'm going to talk a little bit about how we've been using cello add into it. So internally we've been using cello for essentially the past year and in that time, we've been getting a lot of adoption internally. We've created over a thousand projects using cello for various cloud deployments as well as 150 of those, more than 150 of those projects deploying cloud resources all the way up to production. So we've been getting a lot of internal usage but we still have more things we want to do with this. So now I'm going to talk a little bit about what's next. So there are a couple of key features we want to work on. One of them being a better credential provider extraction. So one of the things I briefly, I very vaguely mentioned but didn't go into too much detail about was how do we do that credential exchange between various cloud accounts? Right now we are using Vault. If you haven't heard about it, it is another open source project. I'm not going to get too much into details of it but we want to make a better abstraction for that so you can swap out different credentials providers as needed. Another feature we want to work on is more multi-cloud support. So one of the things we really focused on was when we were designing our principles was we want to be able to support multiple clouds. That being said, internally at Intuit, we obviously have a grand majority of our things on AWS. So we want to do more to actually support other cloud providers like GCP, Azure, whatever. So that's something we want to improve on further. And then finally, potential things like, oh, it's great that we have these APIs that allow us to do these operations but it would be nice if we could make that process a little easier because not everyone wants to use APIs for everything, right? So one of the things we were thinking about for eventual plans would be having a nice user interface. If people could just press a button kind of like in the way like Argo does to roll back this operation or like deploy our things, those kinds of things. So those are some of the features that we were planning on working on next. And now I'm going to pass it back over to Jerome to talk a little bit about wrapping up. So yeah, so thank you, Amin. Yeah, we'd love for you to come out, try Cello out, give us feedback. We have a great quick start, I think. You could set up fairly quick, less than a few minutes. It'll have vault, it'll have all the things it needs and you can try it out with an AWS account together. We have examples in the repo and we have all our documentation up there as well to help you along with that. Thank you, everyone. We're open for questions now. Go ahead, come. Thank you very much for presentation question. What brought you to the conclusion that you need to develop this tool? Why don't you just put the pipeline inside the Jenkins pipeline and it should work perfectly. Right, so our engineers, we have about 6,000 engineers and traditionally they have always picked their own ISE or their own deployment mechanisms. So in order to support them widely, we did want to push them down one homogeneous solution because that has not traditionally worked for us. So we went with a more open solution which allowed also easier path to migration for those, for our engineers. First, thank you for the talking, it's very nice. My question is, in the talking, you create the token for a trailer project, for each project. How do you manage the token? Like, rotate the token, do you have automation in place in Trallow? Yeah, so we do have a couple APIs in place for the token rotation process. So these project tokens are medium long-lived tokens. They're like, I think they expire automatically in a year by default. But we have a couple APIs in place. This kind of goes back to that thing we were talking about, credential provider abstraction, but in the case of Vault, that means we will, we have an ability to clear out and revoke those tokens or make new ones by using some APIs if we need to. And just one follow-up on that is that when we use it, we're not using long-lived tokens, long-lived secrets in the AWS accounts themselves. We're actually getting temporary credentials when we actually do a deployment action. So that token is just for cello use with the integral. Yeah. Yeah. Oh, gosh, sorry. Good question. So you mentioned how different teams have, you know, kind of choose their own adventure, so to speak, as far as their tooling. I see we, I think the API makes sense as far as like the spec, but I'm just kind of curious maybe for the developer experience. Like, do they, did you guys provide, because there's a lot of different API calls. You have to probably manually call and like postman and stuff like that. Could you give a little bit more insight of the experience of an Intuit developer team having to, you know, when they leverage the API, how that generally looks and maybe some metrics, like how long it takes for them to onboard with it and all those kind of things. Sure. I think you've got one thing to say at least, too. All right. So they don't directly interact with the API. We give them tooling around that within Jenkins and we have an onboarding UI that Amin was showing you the UML for. They mostly interact with GitHub and the code they need to create their resources or their application code. So typically they don't have to see that at all. That's mostly our world. Anything you want to add? Yeah, we try our best to abstract a lot of the more nitty-gritty parts away from our developers as much as possible. So in practice, a developer doesn't even necessarily need to know that we're using cello under a hood for this. The idea here is mainly we give them the tools to allow them to do their deployments and as far as they're aware, they have a Jenkins pipeline. They can see what it's doing. So they can see that, like, oh, this Jenkins pipeline is making API calls to something called cello. How much they know where to care about cello is up to them at the end of the day. That's really all I had to add to that. Any other questions? I've got one. So just making sure I understand that cello is abstracting itself as a layer outside of any of the CDK, CFN, or Terraform. Is that correct? Yeah. So who's in charge of writing? Say for instance, first question, you mentioned it was cloud agnostic. So would I assume that it doesn't matter what environment I'm creating it for? AWS, GCP, Azure, cello is intuitive enough to create whatever environment I'm asking for? So essentially, you're just asking about does it matter which cloud provider you're actually using at the end of the day? Precisely. So our long-term plan is to try and support all these different cloud providers. That being said, we need to build out the adapter layers for those different providers. So right now, the one we have in place is AWS. We want to build out more as we go. But right now, that part is a little bit specific to the cello implementation, if that makes sense. So I can follow up after. That's fine. Okay. Well, thank you guys so much for attending and thank you for the session. We can start. Thank you, everyone. Thank you.