 Hello everyone, thanks for joining me in this session, okay so today I will be going to talk on policy driven approach to secure reverse CI CD workflow, okay so all of us know about CI CD right and how what are the complexity we face, how security is much more important in that. So I will be, so in this talk what we going to learn is like how policy will be helpful to manage our CI CD in a secure lane, alright myself Savita, so I work for a red hat and I do contribute mostly on tecton around the tecton, multiple projects of tecton and you can find me over here there, alright so this is the pretty much agenda we have it for today, so basically I will be give a brief about what is cloud native CI CD and how tecton is can be called it as a cloud native CI CD and as tecton is a big org, so I cannot talk, I mean we cannot talk entirely all the projects in a single talk, so I have chosen pipelines as code and tecton chains because these with these two come, I mean with these two I can able to manage my entire CI flow and then I will deploy my workload using a go CD and I will after that I will start use Kiverno for doing my policy stuff in order to make sure that I do follow all the like all the security things and then finally we have a demo where I will be showcasing how I can make use of tecton, Argo and Kiverno policies to make the end to end flow, alright so pretty much all of us know about the CI CD, so I quickly cover like what is cloud native CI CD, so we call it we call CI CD as a cloud native if it follows the principles like deploying that thing as a containers and manages inside the container orchestrators and which obeys the principles of serverless which does the auto scaling based on the request and follows the DevOps pattern, so this is the few differences I have noted down like how cloud native CI CD is different from traditional one, so okay when I when I talked about the print, why we what is what make us to call the CI CD as cloud native and I mentioned about the container serverless and DevOps in that tecton is one of the platform or we can say or the product or project which obeys the cloud native CI CD rules, now before I start I just wanted to know like how many of you heard about tecton before this talk, alright oh pretty much I mean like very less I see, so I can see like I am happy that I got this opportunity to explain everyone what is tecton in a like from a basic onwards, alright okay tecton is open source project it is part of CD foundation it's not yet in the CNCF but I I don't know exactly but there is some discussion going on to move it to the CNCF under the CNCF, so tecton as I mentioned we call it as a CI CD because it it is built for Kubernetes I mean it is built for Kubernetes as it it is native to Kubernetes it it has all the features of Kubernetes like its scales on demand it it handles the security security things very well and it is very flexible as its native to Kubernetes, alright so every project every tool have some basic concepts right like in Kubernetes if I want to write my application and sorry if I want to run my application what all things I should know, one is deployment right and service for containers if I know these four concepts I could able to write my deployment yamal create it and I can write my service yamal and I can exit I mean like I can view that service view my application similarly in tecton we have echo concepts like step task pipeline input output resources task run and pipeline run, okay so I want to go I want to I want to tell these concept in a diagrammetical way so let's suppose the like in Kubernetes we have a containers right which does actions based on whatever image you give whatever the command argument similarly in tecton step is very we can compare step is similar to containers step does the operations whatever commands whatever argument E and V image we give alright so the steps can be like individual operations like I want to clone the code and I want to build the code push the code I mean I want to after the push I want to get the digest to see whether the push is success and then finally I can go and deploy it so these are like steps we can consider a part of operations I want to do individually now what are task so task is a template or I can say static template which actually contains more than one step so it means task task can have one step or it can have a multiple steps and the task can be interconnected each other so that output of one task can be given as an input to other task now the question arises here why can't I as I mentioned task can have one step or multiple step why can't I write one single task which can have all these steps together but there is a there is a concept here called reusability like in Jenkins we have a plug-ins right we can plug in and plug out similarly in tecton we have a task which is kind of a reusable so let's suppose I have a scenario where I want to build Java I want to build go the building process for both the languages are different but the cloning can be same for both the languages so that's why I have divided my task into three different ways so that I can make use of the fetch task for all the languages but task build is only specific to particular languages so this way if we write task such a way that it can be reusable for other purpose as well so that is the one benefit in tecton of task now in the task are just a static template in order to instantiate those object or resource we have a resource called task run now these tasks can be run individually right but again if I if I have a scenario where I want to handle multiple things like in deployment we always try to make sure that we should not run pod alone itself right we always run with the deployment so similarly in tecton it is it is recommended to have a pipeline which has set of task and run that pipeline using a resource called pipeline run so basically what I want to say is that step is a task is a template which can have multiple steps and pipeline is again a template which can have multiple task and in order to run those resources we have a pipeline run and task run okay now as I mentioned like tecton is again a big this arc so it has multiple projects so I will be talking about the pipeline says code today so what is that basically it's a opinionated CI based on tecton so I mean you can whatever you are doing with pipeline says code can be done with tecton different projects like tecton pipelines and tecton triggers but thing is why I'm going for pipelines as code because it has a seamless integration with different source code management like GitLab, Bitbucket and GitHub and some of the benefits what I get is it is version control as I mentioned and it is portability once you write the pipeline run you can just keep in your source code management or the project and that can be used so collaboration means it has I mean it is can be shared within the team and also it has a good collaborations and all then automation yes this is the major benefit like with the pipelines as code in with the pipelines as code what you can as an end user what I can do I can just go and do a pull request or push request on my project so everything will be taken care and the one and after that the status will be reported back to my source code management so that is the one of the major use case here I mean like let's suppose I sent a pull request after all the process it will just reply reply me back with the status okay whether it is a failure or success and where and all the pipeline runs are ran what are the steps or task it has executed and scalability yes of course because it is on top of tecton tecton is on top of Kubernetes scalability is inbuilt and changed changed tracking because we use source code management tools here all right so pipelines right now in the I mean in the current state pipelines as code looks for dot tecton directory in a project so that is the source of truth we can say if your project have dot tecton directory then all the pipeline runs from that folder will be run based on the action or based on the event but we have a we have a scope of removing this dependency so that for a new beginner if they want to try out tecton they can just install on their cluster and just raise a pull request everything will be automated they no need to add the dot tecton directory as well but that is still under discussion all right so I just this is this is our what to say understanding I mean our thoughts like the pipelines as code was a GitHub actions because most of us are very familiar with GitHub actions and all these can be done with GitHub actions but let's suppose I have a project I have a scenario where I want to run my CI CD for GitLab bit bucket GitHub so it is like I need to have different different CI solution for different different source code management but if I if I instead of that if I use pipelines as code right I can I can remove all those dependencies like for GitHub usage person they are very comfortable with GitHub workflow but for same thing when when when their code base moved to GitLab they have to learn the new CI thing but in case of pipelines as code it is common across the all the source code management so that over burden is not there okay so this is the high level overview I mean as an user from the Git platform they will send a pull or push request the request goes to the pipelines as code and does the processing and finally report the status back to pull or push request to the respective Git platform so what are the security best practices which can be utilized or which can be get when we use pipelines as code one of the thing is like it doesn't I mean we have a granular control over the CI CI part where we can control okay who and all have access to what all operations so the pipelines as code policy does that for us and we have a onus file where we can add multiple I mean different different reviewers who can have permissions on that particular project and like token generation I mean like we have a particular permission level for private repo and public repo right so and we have we support multiple comments of retesting your pipe I mean events like whether you want to rerun the request then go for retest and test whether you want to cancel some request we have supported for cancel as well and we have also for okay to test that indicates like okay let's suppose I have a I mean there is a project but I don't have permission to raise PR over there I mean once I can raise but CI won't run for me because that that GitHub doesn't understand my username and all right so the maintainer of that project cannot okay to test to allow my PR to run the CI and all so this kind of things we are supported in pipelines as code today all right tecton chains what is this okay so everywhere we are hearing the word like sls right so basically we want to secure our pipeline run so secure our pipeline run every possible way so in tecton that is possible using the project called chains so chains what it is does it it it actually what it does it's it sign or it does the in to to attestation for your pipeline run or as well as sign the image so these two operations will be done by chains today it is either key based or keyless and underneath it uses cosine and the tecton chain has a controller which continuously watches for the pipeline run and task run and once the those task run pipeline run completed then chains does the signing of the image or sign attestation to the pipeline run all right so ergo cd now am I not the contributor to ergo cd but now have started using the ergo cd as an end user so i have my ci ready now my image is built and it is signed now i want to deploy it deploy it on my cluster in an automated way or like i mean because ergo cd supports in a multiple things right like multi cluster image updater dynamically and all those things so ergo cd like we had a couple of talks today where we got to learn about the ergo cd what it is what are the github's principles along i mean the principles which are follows the github's principle followed only ergo cd or is there any other project so we we got to know about the flux as well today in one of the call so what are those github's principle so basically it is like a declarative it's it it means like i have written some statement and i i am telling the tool either it can be ergo or flux that the statement which i have written should be like that it it should not matter how it how it will do then version version and immutable i mean like as it's a github based everything is controlled over there so we have a shy and all so that's where immutability will come then sorry pulled automatically i mean like we can have a policies like whether it is automatic or manual policy to pull the things and continuously reconcile to make sure that actual state matches with the desired state so this is the high level architecture i refer the standard ergo documentation so basically it majorly have three components api server then repository service and then application controller so api server is kind of a gateway which looks for which is written as a grpc or rest so basically ui cli and api based mechanism will be handled and the request comes to api and it also manages reporting status back to slack email whatever it is configured then application controller is actually making sure that your actual state matches with the desired state and it also helps in managing multicluster deployment and all those things then coming to repository service it it actually helps in caching your git repository data and all those stuff so that the synchronization can be happened very well all right now comes to the actual thing which is called policies all right so anybody have heard about the policies or any one of you like already started using in your day to day work and and how many of you know the value of policies how it makes things easier for you all right so policies this is my understanding that policies is nothing but this set of rules or instructions which help you to like completely block or do some modification or generate things for you right so let's suppose i understand this way policies is like like i am joining as an employee to one company right so there is a rule saying that a person is not allowed into the office until and unless they he or she have the id card so that is kind of a gateway for you or like a validating rule for you so that it will reject i mean they will stop you there right so that is kind of a validation can be done if we want to compare so that is where the policy things will happen so this policy will help us to and do the code quality very well security things then deploying deploying the things before i mean to verify all the images and all signing stuff before deploying actually into the cluster right so basically in a one term if i want to say policies help us to allow avoid to do the bad things in our software or tools now it's our responsibility to control or prevent those things so we have many policy solutions today like one of them is like kivarno opa and six store policy controller so i have tried with the kivarno and six store policy controller and today i will be talking about the kivarno this is one of the policy provided or we can say with the help of kivarno we can handle policies very well and it is native to kubernetes that's where i started investigating some stuff here because tecton is native to kubernetes and kivarno is native to kubernetes so maybe like the that amal syntax and all will help me to achieve most of the policy things with kivarno but again it depends on our use case entirely which policy framework we want to use so it's basically a engine designed for kubernetes so kivarno always does three things like it does the validation it does the mutation and generate right validation is like strictly prohibited mutation is like if you want to do some data changes for your request and generating stuff and it also does the verification of image signing using cosine and into two attestation and it also matches annotations labels namespace and do some stuff what not to do what to do so those things we can control at each level using the admission controllers okay so for my scenario how kivarno is useful for me to do the cicd stuff so as i mentioned i'm using tecton for ci or go for cd so i want to ensure my pipeline run is always should have a namespace in that right and also my pipeline run should obey the security context permissions and i my my pipeline run should only have reference to pipeline which is signed bundle not like any other thing and always i have one more scenario where i want to trigger my pipeline run only if certain labels or annotation matches and from the cd side i want to execute the deployment through argocd if if and only if the images are verified i mean the signing of the image is verified so come that is the major thing i am more interested today so let's quickly see the demo so in the demo part what i am going to do is like i will be having a pipeline and the kivarno does the initial checking for me so once that is success i have a task called fetch which is fetching the code and storing it in the storage then i have a static tools to check my go-wet issues and linter issues and once that is done i'll go and build and push it to the registry now now the question arises like where is signing of the image happening is it inside the build step no i am not doing any of the steps related to the signing of the image that's where i mentioned about the tecton chains so tecton chains what it does once the build task is completed it tries to see that image and sign that image and push it to the registry so this part is called ci so once this is done i have a signed image ready with me so i will ask before the deploying of the image i i want to do image verification and that's where can be done using kivarno so i will be as i am using kivarno to verify my image and if the image verification is not proper it will straight away reject it won't go to the further step but if it is success if the verification is success it will go and deploy the application using argocd okay so i will quickly go and show the demo so actually i have recorded the demo because i have many things to configure so that's where i'll just pause it for a minute and yeah so this is the repo i have so this is the repo i have i have all the instruction and setup so for time constraint i have deployed everything so basically the tecton chains pipelines pipelines as code more controllers then argocd kivarno everything is deployed okay so now let me go back to playing the video the steps which i have followed all right so everything is deployed now the first step is okay so these are the list of policies from the ci side i want to validate so i have kept all the policies in the policies folder so i have couple of things but in the demo i will be showing about the require annotations right so what this policy does is so what i'm what we are doing here is like the pipeline run annotation should not have this max keep runs value more than three right so this is the policy i have written so that whenever pipeline run have any such annotations whose value is greater than sorry whose value is greater than three should fail should reject the request now uh going back right so this is the tecton folder i was talking and here i have kept my yaml pipeline run yaml so before the demo starts i i just wanted to give some highlights on these yaml parts specifically the annotation part so the thing is like if i just write dot tecton directory and pipeline run how does the platform or i can set tecton pipelines as code we'll get to know that pipeline run should triggered for pull request or push request so there should be some mechanism to mention that right so that is where this annotation does so here we mentioned about the event like what type of event we are looking for either it is a pull request or push and which branch actually the event should focus either it is a main branch or any other non-main branch or even we can specify all ref tax star then the another important thing i want to mention about the task right so it is uh pipeline uh so okay as i mentioned like in tecton tasks are usable entity right so similar to docker hub where images are hosted in tecton we have this catalog hub hub hub regis hub hub thing where all the tasks are hosted this is this is upstream so multiple folks contributed so how this task become generic because they have written such a way that it can be used for multiple purpose so that's where like here i have mentioned three tasks so what pipelines as code do is like it will try to fetch this specified task from the hub and even we have the flexibility to specify a particular repository from which task can be fetched as well and this is the annotation which i have uh interested which is like max keep runs uh which indicates that how many pipeline run should be there in your cluster so in in the policy okay let me go back to policy so in the policies i have mentioned it should not exceed more than three right but uh more than more than less than three but it is in this yamal it is five so obviously it should fail according to the policy rules now let's go and see the see it in the action all right so here let me just uh little bit move forward so yeah so it has created the policy okay let me yeah it has created creating the policy then i'm doing the get cluster policy to show that the policy is created okay now the next task is to create some pool request okay so as i mentioned that the here on event is just push but i want my ci to be run for pool request as well so that's where i have changed that uh event and sending the pool request now so you can see that creating the pool request that's it it should work now accordingly but let's see well yeah it doesn't work it is because you can see error in the respective pipeline says code pipeline says code controller it is saying some message that repository match for this thing is not done okay let me show you what is that so when i mentioned like whenever i pool request comes to a particular repository how pipelines has will how pipelines has code get to know that the matching of the like the request coming from the particular repository should match so that the pipeline rail should get triggered so that is where repositories here uh play important role so it actually make sure that this cr whatever url specified in this cr should match with the event coming from that particular project so here in this demo it is missing so that's where the ci did not trigger and it said the message saying that repository did not match so that's where i'm checking whether my cluster has that repository in the particular namespace so it is it was not there so i just went ahead and created using tkn pack now what is tkn pack so in kubernetes we have cube ctl tool right so for argocd it has its own tool so similarly for tecton we have tkn and tecton pipelines as code we have a tkn pack cli tool all right so now repository cr is created so now what i can do i can go back to my pull request now what is the way so that i can rerun the same pull request one is like i can close and reopen the pull request but other way is like i i have this gitops command which i mentioned like i can do test retest test or slash retest to retrigger my ci so now this time the ci should get success uh okay yeah okay all right so it is failing now let me go and check the details the reason for the failure so we should this is accepted reason oh yeah so because in that pipeline run the max key run value is 5 but the policy check started kicking off here saying that i cannot allow this pipeline run because the value is not matching properly from the annotations so that's where this got failed now what i can do i can edit the file in that same pull request to change the max uh annotation value from 5 to like lesser than 3 okay so this is the annotation so i'm just editing that annotation to some number like 2 and saving saving the changes so this time it again retrigger the ci so that uh this time it it obeys the policy rules it allows it allows the pipeline run to go ahead and create it so now you can see that it started running the ci so it will take couple of minutes and we can come here this is a dashboard we have uh from the tecton so i can just show this is how the dashboard look like so in this dashboard the pipeline run is running you can see here this running thing and i can go to the respective pipeline run and see that task what are the tasks which are there in that pipeline run so first it is doing fetching then doing the static check and then it is finally building and pushing the image yeah so it will take time because it has to build it has to push to the registry and all this is my query registry where i'm actually waiting for the image to be updated here okay so let me open up another video about the this thing yeah so after a couple of minute once build and push is successful i should see ci something like this green and if i go to the details page i will see success what in which name is space and what is the pipeline run and what are the stages in that pipeline run everything is mentioned so this is where i mentioned from the pipelines as code presentation in this particular slide saying that once the everything is done the status will be reported back right okay so let me rerun this thing yeah so here everything is success the push is success i should see a new image here and the signing of the image has been done using cosine all right so the next job of mine is to go and deploy using argocd right so now so before deploying argocd let me walk through the policies which i have mentioned right so for argocd we should have a kind called application so where we will mention what is the repository url where actual deployment of service yamal reside and what is the destination i mean the cluster information where we want to deploy our application and what is the namespace and all the information and coming to policy where i will be doing the image verification so let me show that verify image so here i will be specifying the public key the key from which the image has been signed during the build process using tecton chains right so that key i will specify here this is a key based authenticate key based verification but we have an option without key based that is called keyless verification and it always because end of the day argocd creates a deployment and deployments create a pod so image will be there as part of the pod right so that's where i have specified resources as pod all right so let me i mean start this recording thing so yeah i am i just explained these things now right so i am i'm checking whether that policy exists or not so it's not there so i'm creating the policy right so policy has been created okay so now it's time to create the argo application okay so it's time to create the argo application so we have a nice ui from the by the way we have nice ui from the argo side like to create okay let me reduce this yeah so if i go back to application right okay it's timed out okay fine yeah so we have one way we can apply the cr application or else i can go back to ui and create application from there i can specify required information what is the repository url and where what is the destination url all the stuffs i can specify and where my actual deployment resides then the namespace where i want to actually run these things so here you can say i have i'm copying all the things from yaml so that i can show it through ui as well all right so no the image which is signed okay the signed image okay so this is the image name i have the the the the important things of kaivarno is i cannot just give the tag because it it has signed the sha so it while verification it requires sha only it doesn't just take the tag of the image so that's where i have to specify the sha for kaivarno based verification policy verification so let me create it so it will it it will start syncing right so it will it will start syncing with the github application from the github repository to the cluster and i see a error message here saying that the policy verification is failed okay so let's see the reason in order to just to make sure i'm looking into the pod logs as well i mean the i would say it these are the different pods but admission controller is the one which actually has the failure reason what what was what happened so i'm checking the logs here and it is very well mentioned that the invalid signature so it means the public key which i have given in my cluster policy is not the right one from which the image has signed so that's the reason the failure happened now let me copy the right public key which i have locally and update it in the cluster policy so that it should work from there yeah yep so let me just change i mean copy this and update this verify image policy right so after a couple of seconds let me pause for a minute here okay so i'm just editing those lines then i'm just applying it i can do a kubectl edit as well as apply anything is fine all right so let me go back to the yeah now just rechecking whether the content is proper or not so now going back and waiting for this thing to be done so it took a couple of time one i mean like it took a couple of time because it has to re-sync everything the one way is like wait for it other way is like just kill the i mean kill the pod and it should come up so that's where i have done in another recording so after a couple of seconds i just stopped that recording and re-recorded that it is synced okay so this time the verification of the image is proper that's why it went ahead and deployed the application all right so that's all for the demo side and from the tecton argo cd and kyverno based and thanks for listening and this is the qr code where we can get red hat developers blogs documentation everything so if someone wants to learn about different stuff so yeah please take a copy of it all right thank you thank you everyone for keeping so calm and patiently thanks a lot