 Hi everyone, my name is Nikhil Berthwaugh and I'm giving a talk on Tecton project, more specifically how to build cloud-native CICD pipelines with Tecton. Now I'm going to go into details of what cloud-native CICD is and why to use Tecton. More specifically, my agenda for this talk goes as follows. So I want to start with how the traditional CICD tools are today in the market and what are their limitations. And then we go into details of what cloud-native technology is because you hear the word cloud-native where we want to specifically nail down what exactly do we mean by cloud-native. And once we understand what cloud-native is, we'll go into the benefits and challenges that are particularly encountered when you try to build a CICD pipeline more specifically for cloud-native landscape. And there I'll introduce the Tecton project. And I'm going to go into details of what are the various building blocks for a Tecton project. So let's get started. Let me start with the very traditional CICD tools and traditional CICD pipelines. This is just a random image. Essentially, any pipeline is a directed acyclic graph of tasks, right? You have different kind of tests, you have different build, analysis, deployment, all sorts of tests, but it's essentially a directed acyclic graph of steps. And each one of these tasks or steps or whatever you like to call it, itself consists of a sequence of steps. And there are a whole lot of tools in the market that kind of does it. So it's Jenkins, Bamboo. Jenkins is by far the most popular and open source also. Bamboo CICD, TeamCity, one of my favorites, CircleCI, TraverseCI, etc. So anyhow, so this is how a traditional CICD pipeline would look like. And effectively what it is, is that it would consist of a central server followed by a couple of build agents, right? Depending on CICD tools, this presentation may always be available or sometimes they can go on demand. And the server would poll for changes in multiple source code repositories where the teams are checking in once it detects that there's a new change. It starts the process, it starts the sequence of steps, the directed acyclic graph that we saw in the previous slide and we just build an artifact and we kind of, depending on how you configure it, you can even deploy it or you can just keep it as it is. CICD essentially, you know, continuous integration, continuous delivery and continuous integration, continuous deployment. CICD could stand for both and people often confuse that they are the same thing. There's a slight difference between continuous delivery and continuous deployment. In continuous delivery, you treat each commit as a release candidate. So that's the common part. And then you start your sequence of steps and you build, deploy, etc. you generate an artifact. Up until then, continuous delivery and continuous deployment are the same. The difference is after that, in continuous deployment, you would take this artifact and you would deploy it. Whereas in continuous delivery, you just take the artifact, you keep it, but you don't necessarily have to deploy it. So this is the last step. There's a slight difference between continuous delivery and continuous deployment. Nevertheless, this is how a traditional CICD system would look like. Now, it suffers from two major problems. The first major problem is it's not cloud-native. Now, when we go into details of what cloud-native is and what are the challenges that cloud-native CICD gives, then we understand what is it about it that is not cloud-native. But essentially, cloud-native essentially consists of microservices, devops, containers, Kubernetes, all those things. This isn't really designed for that kind of environment. It doesn't deal with container images. It's mostly build artifacts. So it's not cloud-native. And there are a lot of benefits of cloud-native CICD, which unfortunately a traditional CICD would not have. The second thing is it's hard to scale. Depending on, again, this might be partially true for some tools, but you have these build agents that are always on, whether you use them or not, and then you have to provision more build agents. Some tools, they kind of are more easy going in the sense that they can switch off the build agents if you're operating on the cloud. So that way you have some kind of scalability. But even then they don't really have, there are no easy ways of replicating it across environments, replicating it across geo-regions, et cetera. So there are still limitations with scalability, even in the cases where you can handle the build agents to be switched on and off. But nevertheless, the two main drawbacks of this approach is it's not cloud-native and it's hard to scale. Now, we've talked about being cloud-native and we've used this term a lot. Let's actually define what is it? What do we mean by when we say it's not cloud-native? Now, here the CNC website have actually taken a definition and you can read the definition. I'm not going to read the whole definition. It's a big blob of text. But I'll just highlight certain aspects of this, that it talks about containers, service meshes, microservices, the declarative APIs and all these things, and loosely coupled systems. Essentially, it can be boiled down into four key characteristics. One characteristic, containers, and we'll go into details of containers, but it basically has containers. These are microservices that are implemented in containerized form. There's a containers delivery aspect of it. Could be containers deployment, but at least containers delivery aspect of it, that you're treating every release commit as a release candidate. And then you have the DevOps. So you have the whole operations done by devs, the whole sophisticated machinery, you have the pipeline, et cetera, all the standard DevOps practices, monitoring immutable infrastructure, monitoring API, everything. So you have the whole DevOps system. So these four things put together is what we call as cloud-native. So every time user word cloud-native, you know what that means. Now, we have talked about containers. So let's talk, let's discuss what containers are. Essentially, a container is a binary with all its dependencies, right? And it shares the operating system and runs this isolated process. So there's a whole lot of base systems, any language, any library, any binary, et cetera, et cetera. Now, what is it that is specific about containers? Like why we had an old system where we could just, you know, build an artifact locally and then test it, it works, deploy it. What is it that containers really bring to the table? Well, what containers really bring to the table is it not only takes care, it actually brings reproducibility. Now, when I am building a build artifact locally, it actually is kind of tied down to my local environment. I'm using the operating system that is there on my local machine. I'm using the dependencies that are there on my local machines, right? The environment in which it's operated on is actually operated on my local machine. So you could have a situation where I build an artifact and I test it and everything works, and then I deploy it to the cloud and suddenly things start falling apart. Why? Because the environment changed. So what worked best for my machine or my environment may not work well with another environment. The huge selling point of a container is that container packs the environment with the application. So your application is built inside the Docker. All the dependencies, everything is there. The operating system itself is there. It's a self-contained unit. So I build this on my local machine. I test it. I'm convinced this works. I take this whole container as it is and deploy it to another environment. It will still run pretty much identical in that environment as it's running on my machine because the environment, it's not dependent on the environment. Everything that you need, all the dependencies are actually packed. And that also gives you an advantage that sometimes you want to move from one cloud to another cloud, right? You want to move from, let's say, your local data center to a cloud, on-prem to cloud, or you want to move from one cloud vendor to another cloud. You want to move from Google to Azure or vice versa. Containers is a self-contained unit that you can actually transfer from one cloud to another from without really any implications. And typically, organizations, depending on the size, would employ hundreds to thousands of containers. Now, when we are dealing with thousands of containers, we obviously want some way of managing them. And this is where Kubernetes comes in. Essentially, Kubernetes is a kind of a platform for managing containers. Now, it's not the only one, it's the most popular one. There are some others like, I think HashiCorp has Nomad, which kind of does the same thing. And besides Kubernetes, there was another system called Mesosphere, which was developed at Twitter. I think they spun off and it's open source and I think there's some commercial company behind it also. It solves the same problem. I've never used Mesosphere, so I don't know much about Mesosphere. Kubernetes is by far the most popular ones. It's kind of the de facto standard. And what it is, is basically a platform for building platforms. So it's never an end goal in itself. It's just a great starting point. So it's cloud-agnostic. It abstracts away all the underlying infrastructure, like compute network storage. And what you do with it is that you define a state, your desired state. And what Kubernetes essentially does is it tries to achieve that state. So every time you get distracted from it, it tries to go back. And you would, it's a dynamic orchestrated system. We'll go into details of what dynamic orchestration is. But it's a dynamically orchestrated system because at runtime, the resource fail at scale. So your environment is kind of changing. You could have an outage, you could have machines going down. Kubernetes will detect, okay, your VM went down, your container went down. It will try to restart. It also does a very good resource allocation. So some of the things that Kubernetes is actually good at is scheduling. Decide when the containers run, how they run, their life cycle, their health, the scaling. Load balancing, very important. It has out-of-the-box logging and monitoring, debugging, identity and authorization. And here what happens is Kubernetes is an open source project. Kubernetes.io, you can actually download it, run it on your local system. Now here's what happened is Kubernetes itself is cloud-agnostic. But most of the cloud vendors would basically have Kubernetes in some form of managed service. So Google has GKE, Google Kubernetes Engine. Azure has AKS, Azure Kubernetes Service. Amazon has EKS, Elastic Kubernetes Service. They basically have their own flavors of Kubernetes, which is plain vanilla Kubernetes. On top of it, they might have their vendor-specific extension. Like Google might have identity and authorization. It will use the Google IAM, right? Azure will have Azure Active Directory for their Kubernetes. So there's stuff on top. But yeah, Kubernetes otherwise is a cloud-agnostic. And it is a big, famous project. There's lots and lots of tools, a whole lot of companies. This was actually some old statistics with 3,000 contributors, 82,000 GitHub stars. It's a CNCF project. So Cloud Native Computing Foundation CNCF project. And yeah. So it actually was born out of a Google project called Borg. So Google had a project internally that they were using called Borg. And Kubernetes is actually a re-implementation of Borg. Original Borg was written in C++. Kubernetes is written in Golang. It's a re-implementation with Open, and then it's open-sourced. So now that you've understood what containers is, what Kubernetes is, now we go into details of what is a Cloud Native CI CD. So when we talk about a Cloud Native CI CD, we are talking about microservices are in containers, right? So this is where the containers play a role. And then we talk about dynamic orchestration that Kubernetes achieve. So why would we need dynamic orchestration? We've kind of talked about it, but a lot of reasons for one, your environment at scale, things can fail, right? Your machines can go down, storage systems can go down. So you need some kind of, it increases the ability, some kind of way to cope up with these mechanisms. Second, you also have situations like you have serverless workload. So pay as you go. So if there is a load, it will spin up an instance. But if there is no load, it won't spin up an instance. So that's why you want that dynamism. You also want optimized resource utilization, right? You don't want to pay, you have limited set of resources. So you can specify what kind of resources the containers would need and Kubernetes will try to, basically Kubernetes will try to deploy your containers in the way that it uses the most optimum use of your resources are happening. So optimized resource utilization. So that's what Kubernetes would do. And overall, when we talk about cloud native CI CD, that is what we are talking about. We are talking about containers and Kubernetes. So what are the benefits of cloud native CI CD? Well, we have touched upon it in many ways. Containers are reproducible, right? Because they pack the environment with the code. They are reproducible. All the dependencies are there. They would behave exactly the same way in any environment where they are supported. They're also reusable. You have a base image and you can build layers and layers on top of it. We have seen how dynamic orchestration increases the reliability. Outages happen. Outages happen at scale. And dynamic orchestration basically is a mechanism that can be used to cope up with outages that happen. Then we have also talked about serverless resource utilization. In fact, I'll talk a little bit about Knative very briefly, because there's some connection with Tecton. Knative was basically serverless, a system to manage serverless workloads built on top of Kubernetes. And it was actually from where Tecton was born. But serverless is pay as you go, right? So the resource utilization, when you have service utilization, actually means reduced cost. Because you're only paying for what you need. And then you have conformant APIs. And that gives you the advantage that it's portable. There are a lot of benefits. They are good benefits, but then there are challenges. So what are the challenges with CI CD? Well, to start with the first thing you have to bear in mind is that you're dealing with container images and not binaries. So that's the challenge number one. Second, most of the traditional CI CD tools like Jenkins, they talk about built artifacts. They don't really talk about container images here. The second, you're talking about clusters. So you're talking about many, many environments, you're different clusters. You're also talking about microservices instead of monolith. Now, what is the problem with microservices? Well, the problem with microservices is dependencies. They are small, independent unit with several dependencies. Monolith in some ways was very easy to handle. There are obviously drawbacks of monoliths, and that's why monoliths are not used. But monoliths in some ways was very easy to handle because there was this one big service, right? You don't really need a lot of automation. Everything that is needed is there in the service. If an exception happens, you get a nice stack, exception stack with a monolith service. Microservices would not have all of that. So cloud native CI CD has to deal with microservices. It has to deal with challenges that microservices come with. And last but not the least, you're dealing with FNML environments. So these are short-lived environments. They may or may not be there. And again, traditional CI CD tools don't quite address such kind of scenarios. So this is where we need something new, and this is where Tecton Project comes in. So Tecton is a fully open source. I'll talk about the history, but Tecton is a fully open source project, Tecton.dev. That's the website. And it's essentially a powerful, flexible open source framework that provides standard and shared components for CI CD. Now, here I want to point out something which is very, very fundamental and often a kind of a confused aspect of it. Tecton is not, you could use it as a CI CD tool, but really speaking, it's a set of components. So like Kubernetes, which is a platform for building a platform, you can treat Tecton also as a platform for building CI CD tools. So you could use it in its plain form, but you also can extend Tecton to build your own CI CD tools. So think of it as a layered approach where you have Kubernetes as the bottom layer. On top of it, you have Tecton. You could operate at that layer, or you can have another tool that uses Tecton components, and there is no tool. It's called Jenkins X. Jenkins X is essentially a cloud native version of Jenkins. And it's not so much of it as a separate tool. It actually uses Tecton projects as its components. So it's built on top of Tecton. If you want to, you could use Jenkins as it is. You're still using Tecton, or you can use Tecton as it is, or you can build your own tool and extend Tecton. So Tecton is its open source, but it's very extendable, right? In large organizations, this is a common tool that they have very specific needs, right? So they want to have their own tools. They can't use commercial off the shelf tool. And in those sense, Tecton is the one that actually can be used to build your own tool because you don't want to build your own tool from scratch, right? That's too much of work. So what you want is your foundation is laid and you just build on top of that foundation. And that foundation is Tecton. So what is the vision for Tecton? Well, there are four key characteristics. One, composable. You have Tecton pipelines. You have tasks. You basically have building blocks that you can combine to build. So there's a composability aspects of it. When we cover the details of Tecton building blocks, we cover task steps. You will see how composability plays in. But essentially, for now, we can think of it as you want to build higher-order pipelines or higher-order objects from lower objects. So you should have these modular, compile system that you can compose, building blocks that you can build something big. And it's declarative. So I don't want to describe the exact steps. I want to have it declarative. And this is how what I want. This is how my state of the system is. This is how my pipeline should look like. And Tecton can basically do the job. So for example, you have a workflow. You have tasks and you have dependencies. You just specify, you know, these are the dependencies I have. You don't really have to specify the order in which the tasks have to be executed. Tecton will decide that. All you are specifying is the dependencies that each task has. Then you have reproducibility, right? Reproducible. And containers kind of give you that reproducibility. I build an artifact on one environment. It should be built, it should be the exact same artifact that should be built by another environment. My artifact, my container images, whatever I'm trying to do, should not depend on the environment or how you build it. They should be reproducible across environments, across machines. And above everything else, it is cloud native, right? So we are talking about cloud native. Cloud native environment. So these are the few four key components of Tecton vision. And let's talk about the history. I know I briefly mentioned what Knative is. So now at this point, I could go a little bit into details of what Knative is because Tecton started its life as a Knative component. So what happened was that you had Kubernetes and then people wanted to run serverless workloads on top of Kubernetes. Fantastic idea. And Knative was born. And so you have the Kubernetes, you have the Istio gateway, your service mesh. And on top of it, you have these Knative components. So you had three components. You had build, serving, and events, eventing actually. Serving would handle the traffic. Build would handle all the building of the container images. And events would handle how the events are being consumed. So that's our eventing framework is. And Knative build became so popular because there wasn't any good solution for cloud native CI CD that people started using Knative build as their cloud native CI CD tool. So when it was observed that the build is a very popular project, it was decided that build should be taken out of Knative and should be made a project of its own. And that is how Tecton was essentially originated. It originated as a Knative project, which due to its popularity was taken out and became a standalone project of its own. So Knative website is knative.dev. You can actually see right now you will see serving and eventing as the two main repository. Build has been deprecated because it has moved to Tecton. So Tecton is a vendor neutral project. So it has been given to continuous delivery foundation CDF. They are contributions from Google, Red Hat, CloudBees, IBM or a whole lot of people. It's a very new contributors friendly project. So if you want to contribute, you're welcome to do so. But yeah, Tecton otherwise is a vendor neutral project. And Kubernetes also is a vendor neutral, but it's handled by CnCF, Cloud Native Computing Foundation. And Knative has applied to be a CnCF project. This is fairly recent, like I think a week or two ago, somewhere around, maybe three weeks ago, Knative applied to be a CnCF project. I don't think it has been approved yet. Tecton is handled by CDF, but nevertheless, it's vendor neutral. So it's not controlled by Google or some other person, some other corporation. And the way Tecton is implemented, Tecton is implemented as a set of CRDs or custom resource definitions on top of Kubernetes. So Kubernetes is a platform for building platform, and it has all these resources, and you build CRTs to extend that platform. So Knative itself is built on top of Kubernetes using a set of CRDs, and so is Tecton. And I have a link there, so it actually tells you how to install Tectons on Kubernetes, and you'll see there are kubectl commands there that actually install these custom resources. You can actually go to this website, it talks a lot about how to install, and you can actually see, because it's open source, you can actually see what CRDs you're installing. So yeah, that's how Tecton is actually implemented. All right, moving further, now we go into the details of what Tecton is, what are the building blocks of it. This is just a high level overview, we'll go into each one of them in details, but the most primitive one is step, which is a single command, it is basically a single shell scrub command or something like that, and then you take steps and you put them together and you form what is called as a task. Task is essentially a sequence of steps, right? So task is basically built from steps, and then these tasks combine to what is called as pipeline. Pipeline essentially is consist of a directed acyclic graph of tasks, which is fairly common. Even in the traditional CICD tools, you have directly, pipelines are essentially directed acyclic graphs. So I've used Jenkins a while ago, and essentially Jenkins would have something like stages, and these stages together is pipeline, and then each of these stages or steps or tasks, I think I forgot the term they had, had basically a list of commands that you actually type. So it's pretty analogous there that you have, those commands can be thought of a step, and each one of the stage or task, whatever they call it, I forgot the name, will have set of these steps, and they form together to form a pipeline. And pipeline is a directed acyclic graph, and then you have pipeline resources like source control, on which you run them, and you can run the pipeline, or you can run the tasks. The task itself can be run standalone. It doesn't really need to be run as part of the pipeline. You could run it separately. So now let's go into details of what a step is. So step is a run command in a script in a container, Kubernetes, step, you could see it's all loads and loads of YAML. Here I have an example of a Maven install. I have a second example of a Python 3 command, some script running in a container. So that's a step. And like we talked about, steps are combined to form a task, which is a sequential steps in a task pod, which has inputs, outputs, parameters. You obviously have some kind of way of sharing data because tasks combine to form pipelines. So they have to communicate with each other. So they have workspaces and the results are meant for sharing data. And as we saw in the previous example, tasks can run separately or they can be run as part of pipelines. Either way is fine, which is also true for traditional CI CD pipelines. But here's an example of a task. And this is a Maven task example. You have Maven goal, loads of YAML with some parameters, some steps defined. In this case, there's only one step, but you can have multiple steps. You have some metadata. So this is an example of a task. You have multiple tasks like AWS, ALI, Q-Cuttle, Deploy, et cetera, et cetera. Now we talk about pipeline. So what is a pipeline? Not serial, sorry. Parallel execution of tasks, tasks form directed ASIC graphs. So there's an execution order with tasks. And you have input, outputs, and it has things like retries, conditional task evaluation, workspaces where tasks operate. Usable pipelines can be used across projects. So as an example of a pipeline, right? You see the metadata, you see the parameters, you see all sorts of tasks, et cetera. So why do we want to use Tecton? Now here I'm comparing Tecton with Jenkins, but understand the fact that when I say Jenkins, I'm more specifically referring to traditional CI CD tools. So you could substitute something like TeamCity or Bamboo also with Tecton with Jenkins. It's kind of the same thing. But essentially, Tecton is cloud native. So it's designed for the world of containers and Kubernetes where Jenkins actually is designed for virtual machines. So there are a lot of differences. I'll just quickly glance through them. I won't go into details of each one of them. But you have the CI maintenance engine where Kubernetes takes care of it so there is no ops overhead. And plugins are shared across CI engines, but pipelines are kind of fully isolated. Essentially, because it's built on top of Kubernetes, a lot of Kubernetes handles a lot of things for you. For example, platform manages persistence, whereas admin has to manage that, right? And it's a native Kubernetes resources. So everything is life-cycled as container images, whereas Jenkins, you don't really have anything like that. So there are a lot of advantages, especially operating in the cloud native world that Tecton has over traditional CI CD tools. So we go towards the summary. So I just want to summarize traditional CI CD tools. They are not suitable for cloud native world. We know why. We also looked into the benefits that cloud native CI CD tools present, but there are several challenges associated with it. Tecton is an open-source cloud native CI CD tool. It's built on top of Kubernetes and it's vendor-neutral. So it's actually done by a CDF cloud continuous delivery foundation. So it's not controlled by a vendor like Google, IBM, or anybody else. There are steps, tasks, pipelines, and pipeline runs. These are the building blocks for Tecton. And with that, I am going to stop and questions, if any. Thank you.