 Today we want to talk to you about the app project or project in Argo CD and how to use it in your multi-tenant security I'm Luke Phillips a staff software engineer at the New York Times and I'm Serhi Martinenko senior software engineer at the same company and the very same team Here's a brief overview of our agenda for today We're going to set up some background and tell you a bit about the internal developer platform at the New York Times From there we will get into Argo CD and then the app project We will discuss various security and multi-tenancy designs and some of our best practices and summarize it all together So about the times at the times our mission is to build the essential Subscription bundle for every English-speaking curious person who seeks to understand and engage with the world We are a digital first experience leaning into technology to produce comprehensive news coverage News journalism is the times most recognizable product With many others like games wordl obviously, but also crossword and have you tried connections yet? Also cooking audio and two other products that are times properties wire cutter and the athletic Some of the most popular search terms on Google will lead you to the times like wordl This generates a ton of traffic for our site So let me hand it off to Serhi real quick to tell you how as a developer at the times We really want you to focus on building great features into our products Thank you Luke. So I want to start with Speaking about the dream of a lot of infrastructure team and probably application teams would agree on this an internal development platform IDP should consist from the set of tools and technologies Glue together by platform team the platform team should treat the IDP as a product and provide the service to the application teams I worked as a back-end engineer myself and this how I imagine my like dream life day-to-day Development process. I think a lot of application developers would be happy to focus on the things that say no and understand the best to write the code good quality code Cover it with tests and deliver the feature, but in bigger organization in modern life It's a little bit more complicated. You need to think about each of these boxes and Unfortunately, often each of these books is owned by their own team and you as a developer required to find the loose ends to find the newest version of documentation and All the things that are needed like permissions and requests That's why we created the IDP inside the New York Times and we call it the DVSP delivery engineering shared platform it's a set of Capabilities tools and best practices that are available for every developer at the New York Times and its Goal is to provide them with the easy way to follow the golden path of application development We created the self-service tool that Generates a repo templatized source code CI CD pipelines and On board your application to the shared cluster where you can Enjoy the ingress layer and even some of their ability tools since we are on the event That's called argocon. I think it's very important to highlight the important role of Argo CD in this Platform we choose Argo CD as our CD tool because it's open source It follows github's principles and it's just great at doing its job The important thing that I want to highlight here is our Kubernetes runtime We have the shared clusters that are owned and managed by delivery engineering and we as Argo CD Operators are just the regular tenant in this cluster That's why we have a lot of constraints on our architecture decisions that we Need to keep in mind when we build the system and how it works And now I want to pass the mic to Luke to speak about Argo CD Thank you, sir. Hey, thank you for the background of our developer platform given our platform and architecture Choices around container orchestration ID kubernetes and utilizing get ops patterns We went with Argo CD to help facilitate our continuous delivery So when installing Argo CD you might see documentation like this we initially installed this just following these steps And creating a namespace applying the manifest This is done in a cluster where we had full administrative access and this was mostly just used during our proof of concept phases Another screenshot of the documentation how you might add a cluster What we want to point out here is through these docs and instructions through the quick start guides Make many assumptions about the access you have in clusters. You're working with ie full administrative access After reviewing with our security architecture team as well as the rest of our design and developers We had a lot of requirements. We needed to satisfy to run Argo on our multi-tenant kubernetes platform Basically no cluster wide access means that we can't use native tools to install Argo itself We had to limit Argo to certain namespace permissions Means we can't use the Argo CD CLI to add new clusters We had to create a role bindings and roles separately And we couldn't say just out of the box install crd Is we had to create a separate process for those since we had limited administrative access So considering all these requirements how we'd solve this It was a bring-your-own RBAC model decided to separate the pipelines that this that deploy RBAC resource controls from Argo workloads And this allowed us say to install crd's and roles required admin permission separately And we integrated with the bootstrapping of kubernetes clusters to manage some of this cluster resources for Argo Likewise, we took the Argo CD Helm chart from the community It's really fantastic place to start with lots of ways to customize your Argo deploy And then we separated out the results from the chart all the cluster based resources Into separate deployment pipelines from the rest of the Argo workloads itself And likewise customizing the service account that Argo CD uses to connect to different clusters There are many architectural options to choose when going with Argo CD Whether it's instance per cluster standalone management cluster hub and spoke instance prological group or split instances Balancing the pros and cons of the various architectures We enjoyed the experience of one Argo to rule them all sort of the hub and spoke model And we also have various other smaller instances of Argo for our own testing And this is how we kind of chose to build out our platform Now finally, let's look at the component view of Argo CD There exist three phases we like to say the visualize the apply and the retrieve Each phase presents different security challenges within your Argo CD configuration Who can visualize what gets applied to which destination and what is retrieved from which source? Let's go back to Sergei now to introduce the app project Thank you. And let's start talking about the topic of our presentation application project So the app project is a CRD that is installed to your cluster during the regular Argo CD installation process The project focuses on categorizing your applications and putting the safeguards around them You can limit for main things in your app project Restrict what can be deployed, restrict where those things can be deployed What kind of objects might be deployed inside application and set up the RBAC Policies to limit who can perform certain actions inside Argo CD UI or API At this slide, you can see the YAML definition of one of the app projects Even though it was generated for specifically Argo CD Argo con But the YAML is identical to other projects that we have in Production and dev environment. We'll go over each blog and explain what happens there But let's have a step back and see how our project Live in Argo CD ecosystem. So at this slide, you can see the Project XYZ that inherits some properties and limitations from Parent project, global project then Argo CD based on YAML definitions of your project limits the source Posts that are allowed to be used there and the destinations Clusters and namespaces where your application in this project can be deployed You define their bug policies to decide who can perform Certain actions on the applications and the project And if the if your application inside project contains some resources that are not allowed Argo CD would return your error While I was preparing to this presentation, I found a couple interesting discussions Online, what is the difference between application set and app project? They both focuses on categorizing and separating applications. So what is the difference? As you can see application set does not belong to any project It's all proposed is to generate the new applications that can be placed into different projects So your application set can generate the applications that would be Placed into the project for example per environment, dev or prod And now I invite Luke to speak about security concept Thank you, sir. Hey the security aspect of Argo CD app project is critical to evaluate when hardening your Argo CD instances You will face many challenges by attackers, but there are many shields to provide you by the app project Holistically consider the importance of our back how our projects continue Contribute to a secure Argo CD environment limiting the scope and permissions for different teams or applications Clusters scoped to projects sources and destinations limited to projects the read only Argo and managing the default project Consider that there are categories of projects or shields that are appropriate for the right defense Use a dedicated project for the control plane managing Argo CD control plane through GitOps is essential When projects are shared with other applications, there's an inherent risk that these applications Could compromise the Argo CD namespace. This is why a dedicated project becomes pivotal Additionally, it is essential to recognize that our control planes attack surface isn't just limited to our immediate environment It extends to the Git repository housing the control plane resources This is just a glimpse into why securing GitOps repositories is also critical crucial Argo resources are for Argo admins only resources like application set application And that project should have projects themselves dedicated to just these resources We need to ensure that only those truly authorized such as Argo CD administrators Have the power to merge pull requests to these pivotal repositories Here's a couple of examples how we separated out our own repos and how they get loaded in Delete the default project The default project itself in Argo CD might seem benign, but it is a potential Trojan horse by allowing applications from all sources To manage resources across all destinations. It's an open invitation for security breaches It's a nag nagalist to leaving your front door wide open in a bustling chicago city Though designed for learners to quickly grasp Argo CD's functionality Its potential misuse in production scenarios is vast. That's why we advocate for its removal Even if it means directly intervening with tools like coop ctl while removal is extreme Be sure to review the default project if you can't If you can restrict any settings and finally add checks that ensure no application utilizes the default project Block cluster resources in most projects also look at creating tenant type projects Cluster resources like bindings and rolls might sound benign, but that if misuse can Grant undo privileges by blocking these resources at the project level We're essentially setting up our first line of defense against privilege escalation And while exceptions exist, especially for dedicated Argo CD projects assigned to cluster admins The underlying mantra remains tread with caution Likewise consider your allow list or whitelist of resources We enforce restrictions in Kubernetes ourselves for more for more restriction Than just the namespace and we mimic those restrictions in our projects as well Narrow the roles on remote clusters. This is how we connect to our remote clusters in the hub and spoke model It's a collaborative dance though amongst Kubernetes administrators security and developers to hash out the details to find roles and proceed through this design Every cluster has its unique characteristics and our approach must be tailored to respect that as seen in this example Our Argo CD manager service count only gets read only at the cluster level Then our idp will generate Config to manage role bindings on a pertinent or pertinent namespace resources for Argo CD service account again We mimic the pertinent roles via projects Also look at the inheritance of that projects as mentioned before our tenants or apps Have restricted lists of Kubernetes resources available to them rather than repeat this whitelist in every project We utilize the global project properties available in Argo CD Every project is of a tenant type and this property is then applied as shown here There are a variety of other properties available in inheritance The read only Argo CD ensuring that proper Auditability and tracking of all changes to environment or application We enforce that GitOps principles are followed even through the Argo CD or API access our back policies Are designed for every tenant to limit their views to their apps and their actions are read only Looking at the source config in a project utilize glob patterns in your project sources Ideally all your application repositories are behind your organization or follow a certain naming pattern Likewise for destinations again utilize glob patterns for namespaces and cluster names This requires careful planning ahead of time for your platform to create namespaces and clusters to follow certain patterns A newer feature to Projects as clusters and repositories scoped two projects Say a developer has a need to break out of your carefully laid out platform design mentioned earlier in certain naming conventions Perhaps this is adding a way to add experimental or ephemeral clusters into the fold of your Argo CD platform All these patterns can be set all these properties can be set declaratively in Kubernetes secrets While we ourselves have not gone deep into this feature Still evaluating if it kind of holds to the rest of our GitOps concerns and controls But let's look at those GitOps controls Our GitOps controls for example at the new york times We are leveraging two utilities from opa the open policy agent to improve feedback and developer productivity We've used opa conf tests a utility for structured data configuration testing to automate permissions and configuration testing We write declarative policy capturing all of our previously mentioned business and security rules That can then be leveraged by developers locally and runs in our ci processes In this way we enforce compliance in accordance with our security concerns within our GitOps framework Code that isn't compliant does not accidentally get merged into our single source of truth Git repositories Further along in the feedback chain. We also bolster security by using opa gatekeeper a kubernetes and mission controller The opa policies are commonly shared across our workflow app engineers understand what the policy is Operations engineers get peace of mind knowing that configuration data meets a minimum level of compliance Before it is deployed to a live cluster environment. So now let me hand it back over to serhe Thanks, and let's go Um Into the topic of how we Facilitate the multi-tenancy using our project. So we use the GitOps principles to manage the argo cd installation and its structure As luc mentioned we use the open source argo helm Chart to install Argo cd on its own and also there is available another chart argo cd apps that we use to Bootstrap the initial structure of the projects and applications. Then we have two separate reports one for To store the yaml objects of projects for our tenants and another one to store the objects of application and application sets for each of our tenant That's how it looks like from the argo cdui these four applications are created by The The helm chart from argo helm and then it gives us ability to Quickly rebuild the system in case of failure To install it from the scratch. We basically need to run two commands helm install chart for Argo cd and helm install for Up of bootstrapped apps Then we have the up of tenant projects and up of tenant apps Those have the very similar concepts with small differences. So let's have a look at up of up projects. This is a Argo cd application that looks to the Special github repository there. We have the structure of the folders for each environment Application developer runs the self service tool which generates the up project definition and creates a pull request at this moment All the yaml linking and opa policy checks are kicked in and after some approval from engineer from our team Everything is merged back to the main and argo cd creates the new project in the matter of seconds The concept of Up of application sets is pretty much the similar We have the separate repository for that and we have a little bit more relaxed security rules there Um For example, we leverage the code owners feature of github to grant tenants the permission to Self-review the changes to the application sets Speaking about inheritance Luke mentions that Global project that we have is inherent is apparent for every project of our tenants in our Global project we put the limitation on the resources that can be deployed We have just one cluster Level resource that available for tenants and quite a long list of namespace resources We also limit the sources repo so we can deploy only applications from the our github organization And now we'll go to patterns and best practices Thank you, sir. Hey, so to kind of summarize or to bring together a lot of the ideas that we previously discussed um Ideally you want to create groups of projects and design projects around being say argo projects for argo itself tenant type projects or application projects operational projects for say your Cluster admins and also utilize the inheritance and global properties of argo cd Take time to design your infrastructure Use enumerated naming patterns for labels Take advantage of glob patterns to link projects to infrastructure Also take time to design your rback in tenancy common repeatable roles for tenants and apps utilize an oidc for groups and limit destinations for tenants Manage product projects with git ops use After bootstrapping your projects and apps use an app of apps or app of projects synchronization process And use governance ci security Audibility and a various other git repo security tools like say opa um And finally consider the end-to-end developer experience utilize an idp approach to be the front door to your platform Let the idp create update and manage all the project resources in your git ops repo Back to sarah. Yep. Just to summarize everything that we talk about today Couple bullet points. So application project is a must have if you have the multi tenancy argo cd You obviously probably don't want to give the all access to everyone inside the system Using git ops principles for managing argo cd is an awesome thing You have all the benefits of the git ops that application developers has but like for your own system Don't forget the limitations on the application project the sources the destinations Allowed resources and obviously there are bug policies. You probably know the structure of your organization better Leverage the ci pipelines to put some sanity checks on your YAML that you push to the Repository opa policy is a great way to do that if you want to have more information about Our idp dvsp how we call it you can check this talks one would be really soon at multi tenancy Con and the another one is on Thursday If you want to have even more information about new york times You can get this in from these talks online from cube cone in europe this year Thanks everyone feel free to ping us in cncf slug if you have some questions to one chart or ideas to collaborate and If you have question we I think we have a couple of minutes left yes the question is At the moment as far as I know you can't have App projects of app projects. So you can you can't change that down It's so how many app projects do you end up? Is it one per team one per business unit application? We're trying to model one per tenant Is how we do it We're in the way a lot of our awsma multi account tenancies working. We're actually We have different it always some a account. So We're modeling one project per tenant, but it turns into two projects pretend it under the hood And again a lot of it's a lot of that is produced by our idp So yes, it turns into a lot of project yaml But we have automated the process of generating that quick question so Assuming that you have developers that want to push out changes quickly to lower Environments say like a dev a qa or a staging and kind of validate changes quickly What solution have you guys implemented to kind of allow those changes to be Ruled out quickly without having to do a bunch of mr's and stuff like that or pull requests going forward So We have the relaxed rules for pull requests on application Repository so if they want to play with applications that Generator parameters they have the permissions in their own team to review the request for the sub directory We don't we are not involved in this process for the projects It's a bit different story because we want to make sure that the limitations are still in place and that the project Properly tied to the tenant multi account in nws Yeah, just a quick question I wanted to know what the benefits or differences would be between this and like a open source project like capsule that kind of does Multitenancy It sounds very similar, but this one is kind of based more on get ups principle. I don't know if you guys have heard of capsule capsule Yeah, oh, yeah, I'm just I'm trying to understand that multi. Thank you. Yeah Yeah, uh, yeah, it's probably Something you would run in parallel There's a tool like that or some other operator that it will have its rules and if you have argo controlling Infrastructure apps being deployed into that you're going to mimic Say that tenancy in argo projects. Um, so projects might be mimicking. However, else you manage and define your tenancy Quick question Can you tell me maybe how is the best part to add the another cluster? How do you manage the context? What do you want to add a new cluster in argo city? Um, so the process of adding a new cluster in argo city Yes, how do you manage what is the best party to manage like the context um As far as adding a new cluster out of the box, you have the argo cd cli that will Create the kubernetes secrets that argo cd uses to manage that We've kind of reverse engineered that process and our idp will manage that creation Of those resources. We had to like kind of build some tooling that Mashed together our aw we're in eks. We're in aws So tooling that works with eks and aws identities to connect to these new clusters bootstrap them with this new Argo cd service account and then we get those credentials Um, shove it back into shove it into a vault that then makes it back into argo cd