 Ne vzveč sem tukaj občin, da bi se pošličilo. Ciao, tudi. Mi je Marco Crivelaro, prinsipul in genir v zonu. Čakaj sem spetnja vzveč in spetnja, skupajte s tukaj zonu v 2017. In vzveč je inčen zvokal, tukaj v 2020. Vzveč se počičal, da sem vzveč v zvoku. Remember, the zone is a name. So before we dive deep into the subject of the agenda I would like to tell you about what the zone is, then give a bit of history in how we got to use backstage and then we will look into some technical and cultural learnings and I'm hoping to give you some suggestions on the way. So what is the zone? Beside a funny name. The zone is a global life OTT plafor, je tukaj v 11 krajci, z kvalitvom vratu, ki je Italija Serija, German Bundesliga in Spanišlja Liga, in je tukaj v nekaj 180 krajci, z globa vratu, ki je UFA Women Champions League in Boxing Matchroom. Dazone inginering je pravda organizacija, ki je zazone plafor vratu. Toči se od 7 mili inženečnih začali iz svoj 70 njev, z kodinimi vseh njev. Svečne inženečnih je tudi platform način, da se vse vseh vseh njev poslutijo in težkje njev začali inženečnje inženečnje. Čakaj, zelo se različno izgleda. Nelj na povsem na našem, While implementing the micro-service-based architecture of the zone, we found ourselves multiple times in trouble while trying to answer in questions such as, which team owns a specific service, what's the scope of such a service, where can we find the code. We did a new work to find the documentation, monitoring, so a lot of questions, and we realized that we needed to build a service directory to answer all of these questions. So, one evening, a colleague of mine and I were having a beer in Seattle, and he came with the idea of storing all of this information together where the code lived. And this is how the zone manifest came to be. The zone manifest is a YAML metadata file that every engineering team in the zone is required to store in their GitHub repository. And it contains information such as the owner of the repository, which services are deployed from such a repository, dependencies of these services, links to Rambooks, Dashboard, SLOs, and many more informations. So, we built the foundation that the zone manifest. The SRE team has built an API around it, and we started to build tooling around it, but we never got to the point where we worked on the service directory that we wanted to work on the beginning. So, welcome backstage. And this is really the first learning that we have, and it is that it's very likely that the problem that you have has already been solved by someone else. When we first heard about backstage, listening to a podcast, we were amazed by the fact that it had all the features that we were looking for, if not more. So, it would have been silly for us to not try and use it. So, we started the work of implementing backstage internally in March 2021, and we officially launched it October last year, so we are looking at one year of learnings. So, now let's look at some technical learnings. First of all, how do you start? Well, as for most of the things, the best thing that you can do is start small and then build piece by piece. Backstage has a very large quantity of plugins, which keeps growing. This isn't necessarily a bad thing, but sometimes it could be overwhelming, especially at the beginning. Start with what matters the most and then build from there. When we started implementing backstage at the zone, we made the mistake of trying to implement all possible plugins that we could think of, because we wanted to show all the engineering teams how cool backstage could have been. But that had the result of raising even more questions rather than answering them. So, we made a step back and we focused on the software catalog, then we introduced tech docs and only recently we started looking at the scaffolder. You also want to use what you know and keep it simple. Backstage is pretty flexible and you don't necessarily have to run it in a Kubernetes cluster, although we are at KubeCon, so I shouldn't say that. You can use what you know. For instance, in the zone, we deploy our services in AWS ECS Fargate, so we decided to do the same with backstage and use what we already know. We are building a single container image that serves both the front and the back end via an AWS application load balancer. We are offloading the authentication to the application load balancer, which is connected to Azure Active Directory using OpenID Connect and we store into the database RDS Postgres. So, pretty simple. Also, don't be afraid to get your hands dirty. Backstage has a lot of features. We all know that, but sometimes you need to do something that may not be already available in backstage. To give you an example, we use Ashikov Volt to store our secrets and of course we wanted to use the same for backstage, but we couldn't find anything, so we built a native API that could have allowed us to do that, especially while running on ECS Fargate. So we built a wrapper around backstage, which is started as the container starts, fetches the secrets from Volt, populace the configuration for backstage and then it starts the backstage process. And this has proven to be very effective for us because since we deployed it, we never had to change it again while backstage was evolving. Sometimes, though, getting your hands dirty could be painful. An example for that is how we had to handle the authentication. As I mentioned earlier, we are using AWS load balancer to handle the authentication and as we started to use backstage, there was already an API in place to handle the authentication via the application load balancer, but there was no documentation. So we reached out to the community of maintainers, and we found a solution running a dummy sign-in component on the front-end and handling the identity creation at the back-end side. But as backstage was evolving, as its API was evolving, we had to touch that code multiple times because it was breaking our pipeline, but it eventually got much better and we are now using a native proxy sign-in page to handle that, and as we started to use that code, we never had to touch it again, so great job there. We also love external integrations. We think that they are really powerful. External integrations allow you to consume APIs or other sources that you already have in place. We don't necessarily have to use catalogue info format to populate your catalogue, and we are using those to populate our catalogue. We have an API around the manifest, it's called the manifest API, and we use it to ingest components, an API, into backstage. We also have another internal API called the identity API, and we consume it to produce users and groups. But the power of external integration aren't just that you can ingest what you already have in place. You can build automations around them. An example of the automation that we put in place is how we, for instance, emit a custom tag whenever we identify that an entity has SLO defined. So we are creating that tag, so it's easier to filter the entities that already have an SLO in place. We haven't built a plugin to displace the SLO. And also another automation is how we automate the annotation that enables tech docs on the front end. In the zone, we are publishing our tech docs to an AWS-free bucket, and the engineering teams are already required to maintain a GitHub action workflow to publish the tech docs. And we didn't want to ask them to also remember to go into the manifest and add an annotation there so that the front end could enable the link to tech docs. So we decided to use at the external integration level we are checking if the tech docs exist on the Azure bucket, if it's there, then we emit the annotation, otherwise we don't. So the only thing that the engineering teams needs to remember is they need to use a workflow to publish them. External integrations have allowed us to implement another principle that we wanted to adopt since the very beginning, which is not letting backstage becoming our single source of truth. Backstage is more like an aggregator for us. We ingest entities from GitHub, manifest API that I mentioned earlier, identity API, and other APIs that we will have in the futures. There is no entity that is only owned by backstage. We don't register entities directly in backstage. We configure them to be ingested. And this allows us to rebuild backstage from an empty state, and we will know that it will be repopulated with the data that we had in place before. Much easier to do a disaster recovery. We should also strive to strive and implement real friction onboarding. Automate as much as you can, and when you cannot automate it, provide copy and paste examples to the engineers so that it's easy for them to configure whatever is missing. As we started backstage, as we launched backstage, all the engineering teams already found their services in their software catalog, and for the aspects that were not configured, we were providing them a config example that they could copy and paste. And a good example for that, again, is how we are hinting them how to populate tech docs. If we identify that the tech docs are not available, we provide them a snippet of code that they can copy and paste into their GitHub workflow to publish tech docs. If they don't know what tech docs is, we provide a link to the documentation that we maintain explaining how to populate tech docs and how good they are. You should also strive to stay on top of it. Since it is very easy to get started with backstage, it's also easy to forget to maintain it and let it run. And as with many products, beside losing the latest features you will find a lot of friction when you try to upgrade it. So our recommendation is to try and stay up to date with the monthly release. Since the introduction of the upgrade helper it's much easier to do that, but sometimes you will need to be ready to dig into the changelog or even to the code, especially if you have customized some aspects of backstage. Definitely stay in touch with the community. The community session is a great way of knowing what's coming up and reach out to the community of maintainers and adopters in Discord. They are really helpful there. So we see some technical learnings, quick cultural learnings, and then we all go for a beer. For us backstage is a tool that can... We see backstage as a tool that can influence our engineering culture We think it could become a driver of innovation and creativity, so it must be easy to contribute to it and be contribution internal and to the upstream open source. But it wasn't really easy to contribute to the internal instance of backstage, especially at the beginning. As I mentioned earlier, we use Vault to manage our secrets and not all the engineering teams have access to our vault namespace, so the moment they were cloning in the GitHub repository, they couldn't start backstage. So we have created a couple of scripts, make file targets that would allow them to set up backstage, whether they had access to the secrets or not. So if you are a member of the DX team, you can configure backstage with the access to secrets. What it will do for you, it will create a configuration file that makes use of the identity API and the manifest API using the secrets that you have involved, and then you can run it and populate the catalog as if it were in production. If you're not an engineer in the X team and you don't have access to those secrets, you can configure backstage without access to secrets, and it will populate the catalog... Sorry, it will populate the configuration with pointing backstage to some manifest files in the GitHub repositories, so the only secret that you need is your GitHub token to populate some example of the catalog. As we made it easier to contribute to backstage, we saw a first great contribution coming from the internal engineering community, a contribution that is very helpful for us in the zone, and it's called the TV target plugin. The TV target plugin has been contributed by an engineer working in the living room domain, a domain that we at the X team don't know very well, and it contains information for the packages that can be installed on the TV targets, such as Android, Apple TV and whatnot. It contains information such as the countries in which the application is available, minutes viewed per month for that type of application, how the application is packaged, which features are available, it even allows you to add it feature flags, many additional information that somebody in the DX team would have not come up with because we don't know the domain, so it is key that you enable engineers working on other domains to contribute to backstage because they know what they need, they know what is very... what's important to them and so what would make useful backstage to them. You also need to be prepared to make your voice heard and engage with your internal engineering organization. It's incredible, but even after a year that we launched backstage internally, we still hear from people that they haven't heard that backstage was around. Oh, it would be nice to have a service that tells us who owns a specific service. Well, we have it, it's backstage. Haven't you heard about it? So, yeah, and we are trying to be in touch with them continuously. We are writing, we have an internal blog platform and we are posting there, we post their new features announcements, every time we add a new feature to our internal instance, we are posting there, we are hosting demos for teams, we are shouting out for contribution opportunities so whenever we hear from people or from engineers that they have something that they would work on and that might be done in backstage, we try to hint them to go and contribute to our internal instance of backstage and we also try to attend the townhouse that our company has just to spread the word that backstage is around. And I like to close with a result of the first learning and it is that it's likely that someone else would benefit from the solution that you build. So, give back. And there are multiple ways in which you can contribute back to backstage open source. You can go to GitHub issues and find a good first issue, contribute with bug fixes, documentation as well is a good way to contribute, tutorial. In the zone we have contributed a tutorial to enable AWS application load balancer authentication and also with plugins. In the zone we have created a few plugins. One is the GitHub pull request board that us in DX and other engineering teams in the zone are using during their stand-ups to see what are the pull requests that they have opened, what are the statuses. We have contributed with a similar plugin but for GitHub issues because at the moment we have introduced tech docs and we allowed people to comment on tech docs by opening a GitHub issue, we realized nobody was looking at the GitHub issues so we created this plugin so it's there in backstage. And since it was mentioned earlier soon we will also have a plugin to do neuralic analytic using the analytics API. And also another great way to contribute is to give talks like I'm doing and with that I'd like to thank you for listening to me and it was a great pleasure. Thank you, Marco. Thank you. I can take questions. Yeah. Even at the beer. Again, you make me run. Thank you, Marco. Really interesting, as usual. I know that one of the most recurring questions is tips and lesson learned from real adopters. This is the focus of your presentation and a lot of presentation. Would you mind to share some tips that some suggest what not to do with backstage or lesson learned that you suggest not to repeat or to avoid in your experience, of course. If you have some. Yeah, well, I mentioned trying to get every plugin as much as you can because that is just making you lose the focus so definitely something that you don't want to do not doing in backstage. It's another question. Okay, sorry. So let me change another if I can. I mean, if you would be, because now you have experience, if you would be able to think about one missing feature of backstage, what you would like to see as next or what you would like to see as different. I mean, something different, easier, not done or whatever you'd like to choose for, does own yourself, the developer and the team that you're working on. So I love data and one thing that I would like to see is more data, especially on the back-end side of things. So being able to monitor how back-end is working, latencies and whatever, errors, all of that, if that could become easier, that would be great. And observability as well. We also added, just because we are working on the scaffolder now, did a small contribution that would allow us to see who is creating, who is running the scaffolder, just because it is changing something in our GitHub repository so we need to know that. So all of those kind of details on the back-end level is what I'd love to see. We talked a lot about analytics and data collection and data consumption. I'm curious about analytics and how we solve backstage usage itself. What do we know about users, about their patterns, about their habits? Because we mentioned a little bit, that some teams are using backstage during their stand-ups, during onboarding of new team members. Another extremum is that some people tell that they use backstage for everything. So what kind of groups of users are you identified with what data you're collecting and any thoughts on that? Yeah. As we launched backstage, we were a bit blind on how it was used and then we introduced Google Analytics. So what we see backstage being used is that TV target plug-in is used very frequently because it's something specific to what we do at the zone. The engineers that are using it, now we have even people from the product team that are using it. So probably something that not only the engineers need is something that would make it is what is used most frequently. Then I can see the pull request plug-in because it's a lot of team using it. And then the catalog, it really depends because we have, I think, we are around 2,000 entities right now and you have the granularity that they are browsing. So, yeah. The documentation as well is something that is really valuable, especially as more teams are migrating to tech docs because until backstage, really, until we launched backstage, all the documentation was hosted in Confluence and now teams are slowly but surely migrating into backstage. So that is getting a lot of attention now. I hope that answered the question. OK. Thank you so much. Marko. Thank you.