 Hey everybody, welcome to this webinar. You have heard of backstage. Perhaps you have even already set up a POC of your developer portal using backstage. Now the next crucial question is how do I make my developers users? I am your host, Jorge Le Fiesta, author of the Linux Foundation Introduction to Backstage, certified so many years, and I work at rodi.io backstage as a service. And to address the present topic of adoption, I have here with me two experts who have succeeded at widespreading their backstage instance within their organization. First up, we have Kaspar Anishin, a CNCF ambassador in the Nordics, and Lead Thought from Architects at Lunar Bank. Welcome Kaspar. Thank you. How long have you been using backstage? I think we've been using backstage almost since the beginning. So when we first saw the video coming out from Spotify, we more or less instantly started adopting. So that will be almost three years to the date now. Next up, we have Michael Bang, project manager at Relativity, in the team that is in charge of the backstage instance. Welcome Michael. Hey, thank you for having us. Thank you. How long have you been using backstage? We've been using backstage for about a year. We've been seeing it on a lot of, you know, circling the internet, circling developer experience conferences. And, you know, it's one of the things that we wanted to try and tackle ourselves with the large engineering organization. So we've been leveraging Rode for about almost a year now. And it's been great. It's been a great journey. Excellent. Now we know each other, so let's get into the topic. So let's get started with how many developers are using your backstage-based developer portal right now? So yeah, I can go ahead and start. So right now we are approximately 120 full-time developers at Lunar. And I would say all of them uses backstage. I just had a look at our Google Analytics from the past couple of months, and I see that we have around 60 unique users every week. So it's, everybody is sort of getting in there from time to time, depending on, of course, what tasks they have, but we'll come back to that later. Yeah. And then on our side, I think I've seen about 100 unique users week over week with about almost, I think we're approaching like 400 total contributing users in our backstage instance. So we're trying to get more usage out of it, but trying to prove out those use cases where teams are finding valuable and organically kind of get them in there day after day. So like I said, it's a journey. Yeah. Great. That's our really impressive numbers. You've come quite some way across. The next question I have is why do developers go to your developer portal today? Should I go ahead and start out first again, or do you want to? Yeah. I can take this one. So we've kind of seen usage vary across team to team, depending on how they want to use it. And leverage backstage. What I've kind of seen is teams that have been more regularly using it, really like to leverage the tech talks functionality and having their docs live alongside the code, as well as, you know, the workflows that live alongside with it in the code itself. It makes managing it a lot easier, having it everything in one place. And then, of course, the tech talk renders, you know, nicely in backstage and then that content is searchable. We typically have a lot of our documentation stored in Confluence and that information can get updated very quickly. You know, people either copy something, slightly modify it. And then someone, you know, later down the road comes searching for that document which has the same title. All those things can add to cognitive load because they aren't sure which one they should be reading. Even, you know, is this document the right one or is that one the right one? And then you're kind of looking at like when was it last edited. So there's really no good, there was no really great place to really understand where is the formalized documentation for the systems that I'm looking for information for and backstage is a really good way to kind of make that that formalized documentation live. And then another kind of use to this that I've really seen people get excited about when we show them off backstage and generate a lot of excitement is software templates and I can probably get into that afterwards. But we've definitely seen a lot of kind of interest in software templates at our organization. Yeah, I can definitely relate to the documentation in Confluence being like a pain for developers, just having a decoupled from where you work, not having your standard tools available, not being able to write in Markdown or whatever you prefer as a developer has been a really big pain. Before we found backstage, we searched for Markdown documentation pages. We were also in Confluence and didn't really find the good solution until we saw backstage that that was one of the selling points of our adoption is that now we can actually co-locate everything within the repository so that we can provide a really nice developer workflow so that they don't have to go into multiple different places. They can just sit in the good repository and do whatever they need to do. Write that code, write that documentation and everything should be centered around that repository. So that was at least one of the reasons why we adopted backstage in the beginning. We also have a lot of developers visiting our backstage installation due to the fact that we've been building a lot of plugins ourselves. It's actually been like an inner sourcing movement within Lunar. So it's actually one of the two most used plugins is actually something that's not coming out of the centralized platform team. There are more developers that saw a need for creating a plugin that solved a particular problem. One of these problems was they needed a front end for doing reconstitutions. So whenever a new service needed to be spun up and needed to get events from a certain time in order to populate its new database or whatever. So we actually had a squad that built out a plugin for doing reconstitutions to our rabbit installation, which is now being used widely at Lunar. We also have a dead-lettered plugin that basically developers use to handle whenever a message is not or an event is not able to be delivered to some service for some reason. It ends up in this queue that developers can then inspect, figure out what is wrong and recent events if that's what they want to do or just discard them. So getting a UI for some of the things that are really critical to our developers' workflows has been one of the ways we sort of get a lot of developers in there because now they prefer to do it using this UI that's now been built. So that's at least two of the things and maybe you can continue on the scaffolder and the templates because that's also a thing that I can comment on afterwards because we also use that a lot. Yeah, and when we show off the first template that we built, especially to the platform teams that are building the foundations and the tools that other engineering teams use, you can really start seeing the wheels spinning. Like, oh, how could I use this? Ooh, like I have this workflow there. And so like as an example, we had a process that was taking engineering teams about six to eight weeks to build a new microservice without even being able to work like on the actual business logic of their service. This was just a process that kind of involved multiple tickets to different teams, manually creating resources in certain places, getting that repo set up with that boilerplate code and setting up your CICD workflows, security and compliance stuff. All these things took time and each one of them had a chunk of lead time that added up. Whether it was in like, you know, planning sessions or refinements, all these things had a different kind of onboarding document they needed to read, they needed to familiarize themselves with. They needed to interact with a different team if there was questions that were open, they had to conduct a spike, a whole host of things. And then on top of all that there's a number of manual steps you had to go through and then, you know, there's a chance that you miss one of those steps. All of that adds up and you multiply that across, you know, 50, 60, 70 engineering teams that are going through the process. And it was really slowing us down. And so what we essentially did is we looked at that entire process and identified where are all the manual steps, what are each step that these developers have to go through in that process? And what can we automate away? What can we eliminate? What can we abstract away and bundle it into that template? And so now all they have to do is enter in those handful of parameters that we've defined that are important and critical to spinning up that service. And now something that took 6-8 weeks, they can have a microservice kind of endpoint live like that Hello World endpoint live within like 15 minutes, right? And so that process is much more simplified. There's obviously things that we're still working through and we're learning the best way to kind of manage it. But it's really like the first use case that was really eye-opening to the value that the developer portal can provide, especially with like a larger engineering organization. There's a lot of complexities that you have to deal with, compliance regulations, those kinds of things. And so just balancing all that and simplifying it is really a huge benefit to those developers because there's so much complexity that they have to like navigate on a daily basis. And if you can make that a lot simpler for them and give them a lot of that tooling out of the box so they can focus on building out that new feature for the customer and reinvest that time. And whether it's learning opportunities, you know, different, you know, hygiene, code hygiene documentation, making sure that you're having your documentation well thought out. Those kinds of things was usually an afterthought, right? It's usually something that comes after you. You get everything done and you got to move quickly. But now that time can be reinvested in those areas, which is amazing. I think that it will ultimately improve our ability to operate efficiently within the company. So, yeah, that was like the first thing that was really kind of eye-opening to the value that the developer portal can provide. Yeah, I can definitely relate to that as well. We've also been using the scaffold and all the templates for quite a while now. And all the things you mentioned is also what we have been seeing that we also had like a process that required a lot of manual steps. You need to create a Docker registry here. You need to go in and do this and this and this in order to get a database and all those things. Now it's a matter of filling out a form of, I think it's five steps or something. Five fields you need to put in some stuff and get a lot of data using GitHub as well to sort of fetch what teams are available, what domains is out there because we have centralized everything on what domains exist in our microservice infrastructure and all that. You can just select that from the drop-down box and press a button and then you get a repository. You can even build something where you can basically click a box that says, do you want to have your service deployed directly to production? And you can click that box and if you do that and you create the repository, like you also said, like a symbol application will be built. It will be released and you will have an endpoint, an endpoint in our case ready to go. So for developer that means that everything is just set up for them. Everything is ready. Everything is instrumented. The repo is configured using best practices. It's compliant. We enforce review branch protection. We enforce sign commits. All of those things are just set up from the get-go. So developers don't have to think about any of those things. It's just start coding and solve the problem that you need to solve in order to fulfill the business values of whatever you're doing. We also have seen, so we started out very simple, a microservice set up using a template, but now we have 15 or something different templates. So we have docs templates. We have CLI templates. We have microservices templates. We even have, we use a robot framework to automate some tasks in old systems. So we even have a template for how you create a robot to actually press buttons in these old systems. So all of that is configured and is basically being handled for you. So you will get a compliant setup. And now we come to the point where we are able to shut down creation of repositories within GitHub. So everything is controlled through backstage. So developers cannot, don't have the right to create a repository directly in GitHub anymore. That is completely controlled through the process that we enforce using backstage, which is super nice as a financial service institution that we can check those boxes and say we are now compliant in this regard as well. We also use tech insights to sort of provide some features or some insights for developers into are we actually following the right processes, things configured as it's supposed to. So you sort of get the green check mark, everything looks good. I'm good to go. We also have like squad pages. So every team has like a page that sort of provide the most interesting stuff for them. Usually that consists of the GitHub pull request plugin, which is pretty nice. So they get that from the entire squad. So that's what's relevant to them. But also if they have their letter of messages for some of the services or if they're missing to set a domain, for example, on particular services or whatever it might be. You just, you know, surface all the important stuff on the overview page or the team page. So they actually use that in the daily work, which is nice. And the last thing that we sort of seen when you get this software or catalog and everything up and running, we also have a lot of other stakeholders now using backstage for other things. It's not only developers and more. We have a lot of architects. We have the businesses also really using this because now we have the software catalog. That is our asset management catalog. That is where we sort of demonstrate or show auditors that this is our assets, basically our software assets. This is the grid Kelly T that this domain has, which means that if something goes wrong, it's on call that will be triggered. So all of these things are sort of being visualized and also used in backstage to also tell that story to other stakeholders than just developers. So that's that's pretty nice that we now have this central place where we can get all that information and surface it to the different stakeholders that are using the systems. That's certainly when you press it, you've come a long way to be able to stop. Like what you mentioned about developers not being able to create repositories directly on GitHub. I think that's really powerful because then it means that you have already centralized, as you said, the processes and everything that needs to be productive. And it's under your control. That's really great. Yeah, you've already gotten into the question that I was meant to ask. And that's great because it's about how did you widespread backstage across your organization? Because now that you have these features, it is evidence for developers that they get a lot of value to get to your portal. And they have to evolve in the case of Gazpa. They don't have a choice. They have to go to the portal. But in the earlier stages, how did you convince people to start using it? How was the beginning like? Because now it's like really a great picture. And both of you have scaffolded templates that are really calling people in. What was the process for importing entities or all these more tedious work that happens in the beginning? I can maybe start out because we spend a lot of time discussing. We really don't want to put any work on our developers. This is something that we should provide out of the box for them. So at least in the beginning it was like you need to take this GitHub URL. You need to add this thing into the backstage and all that. We didn't want to deal with any of that. Developers shouldn't care about that at all. That is something that should just work out of the box. We spend a lot of time on actually building all the processing and all that. So right now we are kind of lucky because a couple of years before the backstage, we built something that we call shuttle, which is basically a file that lives in all our repositories that amongst others defines ownership, which team this is what is the name of the service. It also defines how does the service look in different environments. So we could fairly easily create a preprocessor that read this file and created all the backstage components based on this file. So we were able to fairly quickly get all of our GitHub repositories in there and then basically starting adding more and more stuff to this file so that we would be able to put on domains or whatever it might be. So we are utilizing our old tooling to instrument and ensure that we get all the data in backstage. So whenever you create a repository, it's in backstage within five minutes and everything is automated. Nothing needs to be added by hand. Which I think was a... If we have forced developers to do a copy, paste, insert, you need to have this backstage component in your repo, we wouldn't have succeeded at all. I think making it easy is important. Yeah, yeah. At the end of the day, we ask developers and team of those teams to do a lot of different things. Especially, I'm assuming with tech insights, right? There's always like, you're playing catch up all the time, whether it's a migration or an upgrade. And that's something that we were trying to balance too. It's like, do we take a heavy handed approach or do we take a more, you know, soft approach where we prove up the value and have people organically come into backstage? And we kind of tried both right now. And I've kind of seen more excitement around, you know, the, like I said, the software templates, the part where we're proving out like the value that it can provide. And that's really getting people excited about it. And so like our first approach was we have a very complex kind of technical ecosystem internally and ownership was something we were trying to juggle. It wasn't entirely clear who owned what. And so our adoption journey really started off with the software catalog. And what we decided to do is couple that with the ability to begin tracking your Dora metrics and the way that you could identify your, you as the owner of a service that was sending, you know, your build and deploy events that were being calculated into a Dora metrics was to register those components into the software catalog. And so that was kind of like our approach to getting people into the software catalog and onboarding their services in there. And we started to see some fidelity at the time to our ecosystem and what teams own, what teams are, what components within relativity are being actively developed. And that was like our first kind of attempt there. And then we started to see some organic usage of like tech docs. People were like, oh, this is interesting. This is cool. Let me check this out. Maybe we'll move our documentation over from, you know, confidence into here. The more formalized documentation and the stuff that we want people to see, not necessarily, you know, like the spike pages and stuff. But yeah, that was like, we kind of seen an extension of a use case that we first kind of started out with. And we're really trying to close the gap on both sides. Like the old stuff that was pre backstage and the new stuff that we want to kind of like start out in backstage. And I think that's similar to kind of like what Casper and his order are doing is that they want every, they are having to guide people through back stages like that, that starting point. And that's what we're trying to do as well. And we're starting to see kind of like a shift in teams thinking about how they can offer more self-service workflows. And backstage can almost like provide that front door to that experience for them. And, you know, integrating those tools in those templates so you get it out of the box. So they don't even have to like think about, you know, catching up the moment that they built a new service. They're already dying because some new tools available or some new versions available. We want to make it easy for them to not even have to think about that and they can get it out the box. And then, you know, I think it's our way of effectively trying to stop the bleeding, so to speak. And over time, those systems that were built before backstage will eventually, you know, make their way in over time because we don't want to say, you know, you need to do this today because we're all moving fast. We're all super busy. We don't want to overburden teams with this work. And, you know, I think that ultimately makes developers happy when they can focus on the stuff that they want to focus on. And over time, they will begin to organically get their stuff in there. So we've kind of tried both, both approaches. And we're still trying to figure it out. But definitely I would say like the approach with the templates, especially if you have a lot of complex workflows is a good route to take to show the value that it can provide. All right. Great. Great discussion today. Thank you so much for coming. I'm sure we are running out of time. But I think you were very excited during your inside time and your experience. Do you guys have any questions or comments before we wrap it up? Maybe if you are around at KubeCon, I have a talk on Friday, I think right after you, maybe. Exactly. So 11.55 Friday at KubeCon, come see me if you are there. Yeah. And I would say if you're thinking about adopting backstage, do it. I do try to have a POC. I think it's like one of the best things that can improve your developers day to day. So if you're thinking about if this has been a discussion with your teams, definitely look into, get it into a hackathon. Get it in front of people that make those decisions and prove out that value. I think you'll have a happier engineering organization overall once you go down the journey. But yeah. Excellent. All right, thank you guys. For everybody going to KubeCon after them, I'll see you there. Thank you. Thank you. Bye.