 So today, we will be talking about multi-tenant backstage in a shared Kubernetes cluster. But before that, as we all know, it's been four years since backstage was incubated in CNCF. And it has been one of the fastest-growing projects in terms of adoption and contributors. So yeah, happy birthday. We can have a clap for that. Yep. Cool. So a little bit about me. I'm Devabrata. I'm a developer relations engineer working with the IDP team at Harness. I like to hack around Kubernetes and Kiosmos project on my weekends as open source contributions. Hello. I'm Vikyath. I'm working with Harness as a staff engineer. And we've been building this internal developer portal since a year now. So we've been closely working with backstage for that. OK. Let's get started. Cool. So we know how developer effectiveness is important for developers to have a good night's sleep, rightly encode. But then sometimes, even platform engineers struggle to make systems resilient, have observability in place so that any time you get a bug, you are like, OK, I get the logs right. And sometimes you scold your platform engineer, hey, I'm not getting the right thing. And sometimes platform engineer thinks that you are dumb. OK. That happens, happens with us as well. Well, basically with backstage, we have discovery of your software components like you could see where are your services, where I say about observability. You could use plugins to see all the logs you require. You could do infrastructure provisioning or service onboarding using Capolder and technical documentation using our tech docs. Will it good? We are solving bit of developer toil using backstage. But is that it? Well, no. Managing backstage has never been easy. We have here heard so many adopters talk all these years. And this has been the central talk of it. What's the good point is, at the end of the talk, everyone says how they have managed to scale backstage, use backstage and adopt it. We will be doing the same today, but we'll be building out stuff. Let's build backstage as a service product called Frontstage. So what's the first order of business here when I start with it? Well, backstage has made it quite easier. You do NPX backstage, create app latest, give your app a beautiful name, CD into your app, do Yan Dev. You don't need to learn TypeScript to do that. It is that easy. Well, what do I get out of it? I get this folder structure, which has an app config YAML, which essentially will carry all my secrets, proxies, configs, everything. Catalog Info YAML, when I say you are able to view your services, all your software components in backstage, it doesn't happen by magic. You have to write your own catalog info.yaml. So that comes with it. And pretty much everything, AI React apps would have package.json, packages, app and backend. We'll be talking more about it while we try to build it out. But that's not it. When I remember, I said that your observability, you could view in backstage. Well, that is a third-party integration, and that comes through plugins. We need to install plugins. Well, how do we do that? We have a massive marketplace support, and this massive marketplace has given rise to a number of adapters. Like, let's take a tool. There are very few tools that don't have a plugin today, but you will find hundreds of plugins with every kind of use cases in the backstage marketplace. And from here, you could choose, according to your use cases, install the plugins. Well, how do you do that? All our plugins are NPM packages. And how do you install them? Well, it's sorry. Oh, sorry. OK, yeah. You do Yanard. This is basically for front-end packages. And if it's a front-end plugin, fairly, this is one of my favorite plugin I use to test backstage. Like, it gives you some quotes on whether you should deploy or not, whether motivational quotes for developers, but it's fairly a front-end plugin. It just needs Yanard to have it on your backstage. That's it. But when, in case of back-end packages or back-end plugins, you need to have your integrations. Like, let's say it's a plugin by your Git provider. You need to add your token. Let's say if it's a HANA-CSED plugin where you need to hit an endpoint like using HANA's token, you need to use proxy and mention it right here. Well, and good. We have now installed a back-end plugin as well. Then now we need to add the front-end out. So every plugin usually have a front-end and back-end part. Every complex plugin does. So let's say here in case of HANA-CSED plugin, we have a beautiful CSED page which populates with the table, had some information coming into it. But the table, all the UI things, comes from this front-end route, which is the HANA-CSED page. And you add it in the plugin.ts, yes. For back-end, so you remember when I said about we are hitting an API endpoint and fetching the data. So now we have written a logic to pass that data and put it in those table, and that is done through this back-end route. This back-end route has that logic, and we are importing it here. So yeah, that's how you do it. HANA-CSED content, what? Then if you have been using Backstage for a long time, you know there are three types of plugins. There are card plugins, there are tab plugins, and then there are page plugins. I want all my services to have this CSED thing in there so that for a particular service, I need to view my pipelines. So I go inside packages, apps, source components, catalog, entity-based.tsx. And then there is something called CSED content, which comes pretty much begged in, and you need to add your plugin. That's it. Voila, you have your plugin, and this is how it looks. Cool. So let's discuss what we got till now. We got a UI. We got a plugin, which features through proxy from a third-party provider. We have got all the core features, a DB, powered by SQLite, since this is how we install it using the NPX. And yeah, that's it. But do we want to do it again and again? If like I said, let's build out a Backstage as a service. I mean, this is not enough. And I will let my teammate to go through it. Yeah. So let's take a quick check on the capabilities of front stage so far. I hope I'm audible because I lost a bit of my voice because of the change in weather. I hope that's fine. Please bear with me. So we started with installing Backstage, which has the software catalog and all the other core features, like the software templates and tech docs and everything. Then Deba showed us how to install a plugin with front end, back end routes, and everything. But what about authentication? So most of the SaaS organizations would have their platform or any other internal services that handles the authentication for them. So how do we integrate this into this? So what we can do here is we can write an ARC middleware to intercept the request coming in from the Backstage back end. So we can extract the required headers that are needed and use it to make an internal request to the internal services or the platform service that is going to authenticate this token. So in this example snippet, if you can see, we are extracting the header, the XAPI key, and the front stage account. And this is being passed to the API call that is going to give us back whether the token is valid or not valid. If the token is valid, we can just pass on to the next request handler, which you can see on the line number 18 here. The token is not valid. We can throw an error similar to what you see in the cache block here and just send an unauthorized message with it. So once we have this ARC middleware written, we can plug it in wherever appropriate inside the router here. For example, if you want all the software catalog and the software template request to be authenticated, we can just add it in the router here before we pass on the control to the respective plugins. So with this addition, all these requests will be authenticated now using the headers and using the intelligence services. And all the unauthenticated request will be rejected from here itself. OK, so once we have this, let's say, I open the Discord on the next day and I see that there is a new release published. And definitely, I would want this to be a part of our front stage setup. So Backstage Projects gets frequent updates with new features. This is one of the main reasons not to fork Backstage. And we should build on top of it so that all the important fixes and new features can be leveraged automatically. So let's say we have this new release published now, version 1.23.4. And the next question would be, how do we upgrade to this release? So the first step in upgrading is to take a look at the diff. So Backstage already provides a wonderful tool to check this diff. So in this example, let's say we are at version 1.10 and we want to upgrade to 1.23.4. In this example, we are looking at the diff of the package of JSON here. Similarly, we can take a look at the diff of all the changes that has been updated in this new version. And once we are clear with all the changes that are being introduced, we can proceed with the upgrade. So to upgrade, Backstage CLI provides a command to bump all the Backstage packages and its dependencies to the latest version. So let me show that command. Yes. So with this command, you can update to the latest version. Or if you want to move to a particular release like what we want to do here, you can also use the release flag here to move to 1.23.4. So now that Backstage is upgraded, we have a working Backstage application with all the core features, few plugins, authentication, and a path to upgrade it to the latest version whenever there is a new one. So with this effort, the complete setup is up and running for one tenant. But do we want all the users to follow similar steps and put duplicate effort? That could be a definite no. This is what we are trying to solve with this front stage hybrid SaaS provider with all of you together here. So how can we make this multi-tenant? Yeah. So one of the ways to make it multi-tenant is by doing namespace isolation. Each Backstage app can be deployed in its own namespace. And the app config required for the instance can be mounted as a config map, again the same namespace. And all this will be in a single sharded cluster hosted on the front stage infrastructure. This way, any data related to a particular account will be completely isolated to that namespace only. And each Backstage instance can be customized as per the user needs independent of the other Backstage instances. Let's also come back to the Authenticeptor we wrote earlier. So the Authenticeptor can be added to each one of these instances so that any request for, say, the software catalog or software templates are intercepted and sent to the internal systems for validating the token for a particular request. The key part here is that the account header can be passed in this request so that the internal system can identify each of these instances uniquely and then authenticate the request immediately. Yeah. So another piece of the puzzle that we need to solve here is the database isolation. So Backstage app and its install plugins would create the databases they need automatically. For example, the catalog plugin would create a database Backstage underscore catalog. The challenge here is, do we provide multiple databases for each instance, or should we provide a single shared instance here? So with multiple instances, what happens is it is a big nightmare to maintain it as we scale. But with the single DB instance, so one of the challenges with the single DB instance is that we don't want the data from one namespace or one instance to be interfering with the data from the other instance. For example, let's say user one has imported a catalog ML in his Backstage app. And we don't want the user to, from a different account, to be able to see this because everything is in the same namespace and everything is in the same shared DB instance. So how can we solve this? So this is the Postgres configuration inside the app config.aml. So we can configure each instance of Backstage with the prefix in the database config where we can specify the front stage account identifier, as we've shown in this example, for a particular instance. So what this means is that all the databases created for that particular app will have this prefix in it. So for example, if the account ID here is XYZ, so the catalog DB will be XYZ underscore catalog instead of Backstage underscore catalog, which means that each of the Backstage apps will have their own catalog databases in the same shared Postgres instance. OK, let's take a quick check again on what all we have solved so far. So we started with installing Backstage, which had all the software catalog and all the core features. We added the AuthMiddleBear for authentication. Then we achieved multi-tenancy in a way by deploying multiple instances in single Kubernetes cluster and also making sure of the namespace and data isolation. Yeah, did we miss something here? Yeah, definitely, plugins, right? So Backstage has a great community building open source plugins. We have currently more than hundreds of plugins in the marketplace. And as Deva mentioned, there are a list of steps you need to follow to install a plugin, which involves installing the NPM package, then adding route to your router in the front end and back end, and then making necessary changes in the entity.exe page. And then finally, you have to provide the configs required in the config.aml. It is a lot of effort to configure a bunch of them even for a single instance, right? Is there a way to simplify this? So we can try to solve this by simplifying the user experience here by reducing the number of steps involved in installing a plugin. So for this, we can build a user interface. Let's call it admin UI. And this user, this can list on all the list of plugins that are available in the marketplace. And the user who logs into this UI can then pick the plugin which he or she wants to install and then just provide the configs required to install it. All the other things can be taken internally if we just have a pre-baked plugin with all the code inside the image already. So based on which plugin is enabled, we can just dynamically read the config and enable the route for that plugin when the Backstage app starts. So this UI definitely needs to be backed with the backend service that can do all the heavy lifting. Let's call it frontstage service in our example here. So this can be deployed either to a new cluster or maybe you can use your existing production cluster. The main goal of this frontstage service is do the orchestration between the Backstage instances and what the user is going to configure in the admin UI. The service can take care of storing the plugin, which plugin is enabled for which account and the required configs for that plugin. Just a second. And this information can be stored against each account ID in, let's say, a MongoDB or any other document database here. And yeah, once we have this information on the database and on the frontstage service side, this information can be relayed on to the corresponding Backstage app based on which user is initiating this request and which account the user belongs to. So along with the plugin enablement, we can use this frontstage service to relay the information about permission updates and, let's say, the config updates. For example, GitHub, GitLab, Bitbucket, or any other configuration which the user is going to configure. Yes, so we have plugins configured now and enabled with just few clicks where the user didn't have to make any code changes or install any NPM package. So we are not done here yet because the plugins assume that they have access to all the internal infrastructure, like the Kubernetes, Git, and the Cloud providers. But in reality, this is not the case. Most of the Kubernetes clusters would be behind some private network or most of the Git providers and Cloud providers could also be self-hosted in the customer's infrastructure. So how can the plugins or the integrations get data from these locations? So to understand this problem, let's take a look at a scenario where, first, the user is going to configure an integration. Let's say that is GitHub and the host will be github.com and user can also give a token how to access it. As mentioned earlier, the frontstage service can be used to relay this information to the backstage backend. And when the user tries to register the catalog info.yml from github.com, backstage would be able to fetch this information easily. But what if the user initially configured github.company.com, which is a GitHub Enterprise instance? This is completely in the user, which is in another cluster, and backstage will not be able to directly make a connection to it and have the access to fetch the catalog info.yml from that location. So how do we solve this? So to solve this, we can build a small application. Let's call it delegate for now. And that can be installed in the user infrastructure. And since this is now deployed in the user infrastructure, we are sure that this has access to the GitHub Enterprise instance. Now the question is, how do we delegate the task in hand to this delegate? That is, how do we send this catalog registration request to fetch the ml to this delegate so that it can fetch the required information from the GitHub Enterprise instance? So here we can write a front stage interceptor, is what you can see in this diagram here. It is just an interceptor that is sitting in the backstage back end that is going to intercept all the requests that are coming to the backstage back end. And then it will take a call whether to send it directly to github.com, or if it's a private request, then it is going to route it to the other way. So front stage service can assign this task to the delegate which we deployed previously. And once the delegate responds back with the catalog info.yaml, this can be relayed back to the front stage interceptor again, and which can return back to the original request initiator, which is the software catalog plugin. So with this, we should be able to get the required information, even from a private network. And the same thing is applicable for even other plugins like Kubernetes or PejaDuty whenever they're behind some private networks. Let's take another quick check on what all we have built so far because we have solved multiple issues now. So we started with backstage app installation where we got all the core features like catalog and everything. We added the auth middleware for authentication. We achieved multi-tenancy by deploying it in multiple namespaces and also making sure of the namespace and database isolation. And now we have the plugins also installation being done easily and also have given the plugins access to these private networks. Oops. Yes, so with this in place and functioning, we have a backstage as a service up and running that can handle multiple tenants in a shared Kubernetes cluster. OK. So when we build backstage as a service, we should be taking care of it being enterprise ready. I mean, every SaaS does that, won't we? So first thing is, since we have built a front-stage service and we are telling we are multi-tenant, so everything will be orchestrated to front-stage service, we need to have all the users inside front-stage service and orchestrate it to backstage, where we use the backstage as entity provider to map all the users from front-stage service to backstage. This is the user ingestion part. And why do we need it? We'll be looking into it. So let's say you have a beautiful catalog with all your software components, everything placed up. Tomorrow you wake up, you log in, and you see suddenly one of your software component is missing. And how do you know? So for this, we built a custom processor which audit every event that takes place. Let's say someone ran a software templates or someone deleted a software component or unregistered a catalog entity. Everything comes as an audit event. And since we have mapped the users using entity provider, we now have information who has deleted this, as well as the diff of the previous and the present state. So you could restore those things, and that's how audit trails can be part of it. And in the previous talk, we heard about permission policies. So yeah, we are not getting rid of permission policies here. Rather than we are using permission policies to give your granular back so that you can stop from your software components getting deleted randomly by anyone from your org. So yeah, this is the enhanced airbag implemented through permission policies that is already available in Backstage. Now let's serve it. We have a multi-tenant Backstage as a service, whereby you have a single Backstage app running in various instances, orchestrated through another service, and could be used by various users through names such as isolation and DVI solution. So thank you. That's all from us. Any questions? We are happy to take it.