 Hi everyone, welcome. So this session is going to be all about scaling out backstage. And the focus for this session is going to be the backend. My colleague Hemma had an amazing talk about some accessibility on the front end, but now we're going to focus on the backend. And the goal of this talk is to give an answer to the many questions that we have received on Discord during the years regarding this matter. First of all, let me introduce myself. My name is Vincenzo. I'm a senior engineer at Spotify. I live in Sweden, where I moved nine years ago from Italy. And I'm part of the core team of Backstage working with the core framework and all things open source. And this is our agenda for today. So we're going to start looking at the structure of a new Backstage app. Then we're going to check how communication or plugins communicate together. And then we will transition into the main part of this talk, which is about finding some ways for scaling out your Backstage instance. And then later we'll look at an application of one of these approaches for scaling out Backstage by looking at our own Spotify internal developer portal deployment. And then we will end up with some final recommendations. So in order to start, I would like to start this talk with a story, the story of a person seeing Backstage for the first time, myself three years ago. It took me three months to understand what Backstage was doing, but this is for another time. And I remember the first time I saw a Backstage app, it looked really straightforward to me. It was a JavaScript project. I knew JavaScript, so probably that helped. And the JavaScript project contained a Yarnmon repo with a plugin directory where you could build your own plugins there. So you build your own plugins under this directory, and you could also install an open source plugin directly in your Backstage instance. And once you run this command, Yarn start dash backend, this command basically takes the entire app, take all the plugins, bundle together, and run in a single process. I found this really, really straightforward. Then I started working on Backstage, and I remember I was working on one of our internal plugins. And at a certain point I had to retrieve some data from another plugin. For simplicity, let's imagine that I was working on a to-do plugin implementing a real, a classical to-do app. At a certain point I had to extend the functionality of a to-do app by, for example, showing the author of each to-do. And let's assume that the author of the to-do are ingested into catalog as a user entity. So I basically had to retrieve some data from catalog. I didn't know how to do it, so I basically looked at the source code just to find a code snippet, a cool copy paste. And I found out that there was a very handy catalog client there with a bunch of containing a bunch of methods. And within all this method there was the method I needed, get entity by ref. The code looked legit, so I just copy pasted it and boom, I had my entity and I implemented basically the feature. I've retrieved some data at the first shot. I found this really, really, really good. Then I'm a very curious person. I want to know how this catalog works and actually dive into the code of this get entity by ref method and found something really interesting. This is the code. I've simplified a bit. Does anyone recognize what this is doing? So this is a typical rest request. So my plugin was trying to retrieve some data from catalog using a rest request. I honestly thought this was a bug because it was pointless. I knew the two plugins were running in the same app. So why was my plugin opening like an HTTP connection just to retrieve a bunch of data that was there? So this was really confusing to me. But I'm very fortunate because I work as a spot if any of some colleagues who know about Backstage saw I reached out to the team internally claiming that this was a bug and that we should fix as soon as possible. And Freben answered me. I'm sure he has answered many of your questions on this code. It's awesome. And he told me that the fact that both plugins live in the same process is not always guaranteed. The answer confused me. I mean, what do you mean? I know the plugins run on the same process of my machine. And he said that that's because I was using the default deployment option. But additionally, there are many other ways you could deploy Backstage and there is a way to deploy the plugin separately. I found this was magic. This was mind blowing. This was the missing piece, something that I missed from Backstage. You can deploy the plugin separately. And Backstage knows exactly how to reach out to the plugin. But still, how does it work? I mean, Backstage is very versatile. You can deploy on Kubernetes, so on a Raspberry or on a virtual machine. How could Backstage know where the plugin is deployed? So I reached out to the team and I think I was knowing then so much that Ben jumped into the discussion and he said, hey buddy, look at your code. It's there. Discovery service. Do you remember the code I showed about the catalog? Discovery service. You pass a plugin to the discovery service and discovery service gives you exactly the address of that plugin. This is where the magic was. So I was so happy. Now I knew where the magic was. So I dug into the source code of the discovery service. Do you want to see it? Here it is. So discovery service knows exactly where a plugin is located. But unfortunately, batteries are not included. Just to clarify this example, the default implementation of the discovery service can resolve the address of the plugins only for those plugins who are deployed in the same Backstage app. So if you want to deploy your plugin separately, you need to provide your own implementation of discovery service. And you can do by providing your own implementation, your own class, overriding this get base URL method. But actually, there is another way of doing that, that we have released, that was released less than a year ago. We haven't advertised so much. Probably this is a point for improvement. And basically, you can configure the discovery service directly in your app config. You can list that in case your plugins have an address that doesn't change, you can register them directly in the app config and then the discovery service will resolve them automatically. So remember, if you want to deploy the plugin separately, you need to provide your own implementation of discovery service. What if you don't? Well, if you don't, you can still scale out Backstage. Let me show you how. So this is the first approach. This is a Backstage app. You can see at the bottom, I have some core services containing like logging, configuration and CLI and other stuff. And you cannot touch plugins to the instance. Right now, I have six of them installed. So if you want to scale this, one approach is just take your instance, replicate it, and put a load balancer on front, routing the request. This approach is absolutely valid. Pro and cons of this approach. It's absolutely the easiest way. You don't have to change code. You could just replicate the instance without changing code and it will work. But the downside is that if you have a plugin that requires more resources, for example, requires more memory or requires receive more requests or maybe steal the event loop, that plugin might cause other plugins to starve. So to the point that other plugins might not serve any request. And the other downside is that if you don't provide your own discovery service implementation, plugin can only communicate only with other plugins within the same instance. So in this case, the Kubernetes plugin is trying to receive some data from catalog and it can only reach out to the catalog plugin that reside in same instance. So this was the first approach. Second one, this is particularly used in case you have a plugin that deserve a different treatment. A plugin that consume more resources. In that case, you can move out the plugin outside of the main instance and deploy on its own. With this approach, you can basically scale it differently. You can give more memory, different machine or more instances. How to do this? You can do this by conditionally enabling plugins. So in this snippet, I have enabled Ska folder searching Kubernetes plugin in the main instance, but I've also enabled catalog in a secondary instance. And this is actually, I mean, you can do this with just a bunch of lines of codes. And prior cons of this approach, you can have more control in case you have a greedy plugin. Not so many code changes are needed since you can just conditionally enable plugins. But still, plugins can only communicate with other plugins within the same instance. But now the plugin is a bit bigger. If you go back with the previous example, where Kubernetes plugin was trying to receive some data from catalog, now catalog is not always there. It's not anymore there. So it can communicate. So you remember you need to provide your own discovery service configuration. So this was the second approach. Now let's move to the third one, which is a kind of goes to the extreme of the previous approach. And it consists of deploying each plugin separately from each other. You can do this by creating many different apps, backside apps containing just the core functionality and the code of a single plugin. And you can deploy them separately. Prior cons of this approach, now you have absolutely freedom how you want to scale the plugins. You can do whatever you want. But I don't think that's the biggest advantage. For me, the biggest advantage is that you can set clear ownership on a plugin. Since the plugins now have their own repository, you can have a team owning the plugin end to end, from the code to deploying to the real ability of the plugin. And I think this advantage is massive. But on the downside is that this is hard to maintain. Now you don't have a back set up anymore. You have many back set up that you need to upgrade. And the other downside is that, of course, remember to configure, not a downside, but remember to configure your discovery service. So now we went through all these approaches. Now let's move to an application of one of these approaches by looking at our own Spotify internal developer portal. And when we talk about our own Spotify internal developer portal, it's a bit tricky because of historical reason. So we, as many of you know, we created Backstage as Spotify in 2016. And then later we donated to the CNCF. And Backstage was created in 2016, actually there. But the open source version was released in 2020. And these are basically two different products. The internal Backstage used Java for the services, while the open source version is written in Node. And the internal one was actually very bound to the technology that we use as Spotify, while the open source, you can basically swap any pieces. And of course, during this year, we have developed more than 100 plugins developed between the front and the back end plugins. So our internal instance was massive. But in 2020, we also, after a couple of months, we realized that we had to move to Backstage open source as soon as possible because the speed of Backstage open source was insanely faster than our internal developer portal. So we decided to adopt it and try to deprecate the internal one. But the internal one wasn't all bad. There were a few pillars that we want to keep when migrating to Backstage open source. So the internal one, as I said, contained more than 100 plugins. But those plugins were built in a distributed way. So there wasn't a team owning all the plugins. But instead, many teams as Spotify developed all these plugins. And of course, they had their own, each plugin had their own repository. So any team owned that plugin end-to-end from the code to the plugin to the reliability. And we want to keep this when migrating to the new Backstage. That's why we chose the third approach. So we deployed each plugin separately. And in the picture, you can see our internal Backstage back end legacy, Backstage back end instance at the top. It was a Java service with GraphQL and a bunch of resolvers that basically collect data from the underlying back end services. And at the bottom, you can see all the open source framework, open source plugins deployed separately. And this was very necessary for us to keep the kind of way of working that we were using as Spotify. And now, before we end, I would like to end with just kind of an advice. Because I know how it is. You go to a random conference listening for random guys from a tech company telling you how to do things. Then you go back home to your own team saying, oh, look, this is how Spotify is doing. We should do the same. And then the problem starts. So in order to avoid this, I basically did some research and came up with a question with a binary answer, yes or no. And answering this question will give you a clear direction whether you need to follow the same approach we are using as Spotify. So deploying the plugin separately. And the question is, are you Spotify? For many of you today, the answer will be no. And with Spotify, of course, I mean a company at the same scale of Spotify. The thing is, all these approaches are absolutely valid. They all have strengths and weaknesses. But if you choose the third approach, then you commit to it's going to be hard to maintain if you don't have the right tool for maintaining different repositories. And as Spotify internal, we have tools like Freedshift, which allow us to push changes to different repositories with just a bunch of clicks. So please choose wisely. Remember to configure your discovery service in case you want to deploy the plugin disability. And unfortunately, we didn't have so much time to cover the last bullet point. But if you have service to service enabled between services, you need to make sure that different instance can understand each other. But we didn't have so much time to cover that. So I would like to invite you to Qtcrash on April 24th. It's an online day where we will go through a lot of talk about how to build your own developer portal from the basics to the most advanced topic we didn't have time to cover today. This is Qtcode. And thanks so much. Any questions while we set up the panel? For the next panel, so any questions for... So with the distributed back end plugin model, one of the things that we've seen, we have lots of internal plugins, some of those plugins from different teams incorporate different secrets as well. So one of the benefit we were thinking of with being able to separate the back end is being able to separate the secrets as well. So they're not all being used by the same back end. Any thoughts on that? Are you talking about, for example, authentication secret, this kind of things? Yeah, for back end downstream systems, things like that. So right now it seems like in our single back end backstage instance that we have tons of secrets and the secret sprawl from different teams enabling their plugins. Yeah. I can bring you a clear use case about like when, for example, when we talk about enabling services, service authentication, you know, for example, you need to provide your own key. And in that case, if you're going to distribute or deploy your plugin, you need to share that key in many, many ways. What we did internally in that case was to kind of use GCP service off for providing that key. So basically you don't kind of share secrets, but instead your instance, our cloud instance kind of inject the secret for you so that you don't have shared secrets in that case. That could be like one way for solving that. But of course, I guess like, I mean, every, every, you can have different cases where you probably need to kind of share the same secrets. And it was less about the plugin authentication, more about the downstream systems. But that gives me more to think about maybe an API gateway or service match traffic permissions or something. Thanks. Any other questions? Thank you. My question is how does this scale, if you do this distributed per plugin deployment and whatnot, how does this scale the configuration file, the appconfig.yaml? Can I maintain different configs because there's like the base URL stuff in it and per plugin configuration? What do I do with the config file? Yeah, in that case, you will need to kind of replicate a config, kind of strip out config which don't belong to the plugin that are installed in an instance. So, and actually I don't, I don't think it's a big problem because if you consider that different team owns different plugins, so it's absolutely fine if they don't, if you don't, if you have to replicate a config or kind of provide a custom config for each, for each plugin. But this is the way we are doing. It's like each plugin has its own config that is different from the config that other plugins are doing, are using it. Okay? Hi. One short question. You talked about splitting up back end plugins. What if you have a large number of front end plugins? How do you suggest we should distribute them? Yeah, I don't think we have kind of changed the way you, we work with front end plugins. We have an instance with, I don't know how many plugins, it's like around a hundred and there, all the front, front end plugins are in the single monolith. And that works for us. The only problem we're seeing is that it takes so much time, at a certain point, when you have a lot of plugins, it takes so much time to build, to build the front end. And the way we are solved is like to provide an experimental wide support, which kind of speed up this process. And it's actually kind of working for us. So the way we decide to solve it is keep the monolith with all the front end plugins, but kind of change, improve the tools that we use for building those plugins. But the, right now for the front end, it kind of, that's the approach. We haven't changed. So everything is in a monolith. All the plugins, yeah. Hey, thanks. There was a discussion about this dynamic loading of plugins. How does it work when you have like, multiple instances running? Yeah, unfortunately I can't, I have, there are more people in the room who can answer this question, but I'm unfortunately, I'm not, I can't answer this. So I can go if you want. I don't want to steal the show. So we can talk back there. But you can load plugins as like, through module federation. So you can load plugins from a remote location, from a remote server as a browser asset. So it's, it doesn't matter. It can be even hosted on an HTTP server anywhere else. It's just a JavaScript asset that is loaded to browser and since backstage or client rendered, you don't care where the assets live, right? So you just load it from somewhere. So you can have it distributed per plugin. Each plugin can be living in, on its separate endpoint, on its separate host or whatever. You just maintain some verification that this is the plugin I want to load, this is the version I want to load, and you can load it from wherever. Alright, is there any questions? Otherwise we'll go to the panel. Okay, thank you. Thank you Vince. Thank you. Thank you Vincenzo.