 Hi everyone and thank you for coming and I hope you had a good lunch. We are from Rody and we're here today to talk to you about an easier way to get to a rich catalogue in backstage. What do we mean by rich catalogue? Well a catalogue that has the information that you're trying to look up, right? So I'm Sam and this is Irma. We're both engineers at Rody. Who's Rody? Well we've been there from the start of Backstage as an open source project. We're still top contributors to the project and we've released about 19 plugins and plenty more scaffolder actions to the open source community. But what does Rody actually do as a company? Well we have a hosted version of Backstage that is easily configurable and has a ton of extra features which is really oriented around making it easier to adopt Backstage and also a lot faster to adopt Backstage. So what's the problem that we're talking about here today? Well I imagine that plenty of you in the audience have either used Backstage regularly or have used it before and I'm sure that pretty much all of you will have seen something like this before when you've come to an entity and you're trying to find out some information but hey there's no annotation for it so we can't find any information for this entity right? So this can apply to all sorts of different plugins and all these plugins rely on some information in the YAML file to actually connect to the third party source of information and it's a very common that that is missing. So this is kind of a frustrating experience right? Now as a good citizen as I'm sure you all are your thought as developers might be well let's try and fix this okay? So you've come to find some information but hey I might as well just fix this while I'm at it. Now what does that journey actually look like in Backstage at the moment? So if we're talking about entities that are sourced from a YAML file in your SEM that's probably going to mean you need to open a pull request to the SEM to fix that YAML right? So what does that look like? Well first of all there's an easy way to open pull requests some of you might actually not know this so take notes and if you go to the about card and you click the pencil icon you'll go straight to a web-based editor for instance in GitHub it looks a bit like this and you can write your YAML annotation straight into there right? Now we know what the key to this YAML this annotation is so let's take an example say pager duty okay? So we want to fix the pager duty plugin and pager duty needs to know what service this entity is connected to so it's going to need to enter a pager duty service ID annotation so let's add that key but then hold on we need to know the value right? So what's the what's the service ID? Well we're going to have to go to pager duty right? So let's log into pager duty hopefully you know the address of your organization's pager duty instance and hopefully you've got access to it right? So we're going to sign in and we're going to go to services in pager duty and hopefully it's going to be easy to find that service for the entity that you're looking up. Now as you're probably aware in large organizations naming conventions aren't always strictly followed and so yeah you're going to have to maybe do some scrolling let's say you find that service so then we open up the service and we're looking for a service ID where is the service ID? Well in this case it's actually not there so we're going to click around some tabs you might go to the settings tab but it's actually not there and so as engineers if you're an engineer you might be familiar with looking at the URL right for an ID so in this case the URL looks like this so I'm going to be like well that looks a little bit like a service ID with that address let's copy and paste that and try that out so we're going to go back to github in this case and we're going to enter that service ID into our yaml then we're going to open up our pull requests we're going to give it a title description open it up and now we need to get someone to review that pull request so we'll have a look through a list of potential reviewers maybe assign some people or we might go into say Slack or some kind of team messaging service and ping a channel and say hey can you have a look at this pull request and review it for me and then we come back to backstage and what we see is still our missing annotation page and we might be seeing that for a few hours we might be seeing that for a few days we might be seeing that for weeks because we're not really sure when that pull request is going to get reviewed and we're not really sure when it's actually going to get merged and the likely hit is that it's probably quite low in your radar right to keep an eye on that pull request and make sure it gets merged and follow it up and so you might not even know when it gets merged so this is the problem that we have at the moment with backstage where you've got a catalog of information and it's often incomplete and the path to actually fixing that information is a fairly slow path that involves a massive context change from what you are actually doing which is trying to just find some information in the first place right you're actually going to have to go to the third party source where you could have got that information from anyway and do a whole bunch of steps in order to fix this and so the likely hit is that you're not going to bother right you're just going to carry on with your day and not going to fix anything and this is a problem right because your catalog is not going to be a rich catalog if your engineers are just navigating away from these problems because they can't be bothered and I don't blame people for not being bothered I can't be bothered most of the time either so this actually is the happy path of this whole scenario because you have a YAML file and you can actually add that information but a lot of entities in the catalog are not actually pulled in from YAML files as you might be aware for instance if you think about groups and users in backstage most most organizations are pulling those in automatically from a third party like github teams or octa or whatever it is and that's running through a processor right so sorry a provider so a provider is taking those that information from the API in github and creating entities in your catalog for you and that's great because it means you can build up a pretty good map of groups and users and they're all connected and that's nice but what happens if you want to actually add some information to those groups so this is like an example of a group that has been pulled in from github teams okay so we've got a title the name of the sorry the name of the group we've got a short description and then we've got the relationships like the users and the relationships so that's kind of okay but hold on a minute if you're coming to backstage to find the owner of some software and you end up in this on this page how do I contact that owner like I know I've got a team name that's all I've got to go off right it's not particularly useful so to fix this we would want to like add links maybe to slack we'd want to add some mark down to a bigger better description of what the team is actually responsible for etc and we would want to add to maybe some contact information like an email address but we can't actually do this in backstage at the moment this is actually not possible on a on a per in per entity basis and that's because it's sourced from a provider and that provider is automated and there isn't a way to individually update these these entities in the catalogue so I say there isn't a way I mean there is kind of a way like you can write a processor and this is true of the yaml files as well you can write a processor that tries to do some mapping from your third-party API so let's go back to the pager duty example we can go to pager duty say give me all the your services and let's try and make an educated guess about the naming convention that you're using for pager duty and try and map those to entities in your catalogue and you might be able to do that okay if your organization has a naming convention that's strictly adhered to that might just about work but the reality is for most organizations and most of these plugins the data does not map cleanly and furthermore you might well be making mistakes when you're actually trying to do automated mappings like this which is potentially even more problematic than not having the information there in the first place because you're going to be showing information for a third-party service which shouldn't be connected to this entity and that's kind of pretty painful right so processes okay might be a potential solution and similarly we could also just instead of doing it behind the scenes in a processor we might want to write a script and open a bunch of PRs to fix this but this is the same problem in terms of the mapping not being reliable and you've also adding in then the PR process right so you've got this like this delay in actually getting the information into the catalogue so what's our solution to this problem yeah so we were thinking about creating something efficient and something sustainable and we had to make easy and a straightforward solution so we were thinking of producing content switching and decreasing time you have to wait for your co-workers to approve any changes for example that you may make so we were also thinking from the point of which are the biggest pain for our customers and for ourselves and we came up with following so SM has mentioned and showed earlier this missing annotation screen is something that we all came across but solving it is energy consuming time consuming and very often you need to go to several places in order to fix it so that's naturally where we started so we worked on improving the pager duty plugin card and we basically wanted to allow users to automatically generate service id from pager duty retrieve through api so they don't have to go through the path that was described earlier and in action it looks like this so I think that we can all agree that this is much faster than going to pager duty obtaining service id and you know like going through all of the different kind of scenarios that you would normally need to go now furthermore furthermore just a second yeah so we were also thinking about the place where you end up most of your time and where you spend majority of your time and catalog seemed like a good place to start so we wanted to improve functionalities that you have there by allowing you to inline edit properties and it also relies on functionality that we worked on where we improved and modified the backstage catalog table so it can kind of yeah that we can expand the feature on it more so what you would be able to do is basically do an inline editing without a need to edit yaml files to create prs to go through the whole process but instead just change it here so as you could see in this video you basically just assign owner or any other property that you may need now of course in catalog you don't have all of the information that component has so there was a need to create something separately and we worked on a concept called entity decorator which is basically an editing form in which you can add or edit properties such as such as tags labels or annotations so things which you usually don't see in catalog so for example set of annotations seen here is generated directly from annotations using components across catalog that you use and enriched with some default ones that we have added for the plugins now the biggest impact we have seen is in decreasing time to change which now is a few seconds compared to day or even days before and that in addition to no context switching so ability to use the centralized approach in which you can change everything just at one place led us to our end goal which was having richer catalog which holds only information and actions so you actually need so how do we actually build this and what does it look like so I'll talk you briefly through the back end and what how you could actually implement something like this relatively easily so to start with we've got an API where we can post something we called fragments so a fragment is just an object with metadata and some spec a spec object and an entity ref that it's associated with right so we can post that and we created a new table in the database where we can store that information so we've got some auditing information and versioning as well there and then we've got our processor so we are talking about processors earlier right so this is just a processor that goes through each entity in your catalog and says hey do we have any do we have a fragment for this to decorate and if it finds one it's going to merge in that that information and the end result is you've got a working plugin within a few seconds so there's nothing particularly revolutionary here right we're just using processors which are a core part of backstage and already do a lot of decoration work on your entities and here instead of the source being a third party API and some automated service you're just using user inputted data right so that's what it looks like now i'm sure some of you in the audience who work on backstage have got some some questions in your mind already about this approach and one of them i imagine might be well we started off with a github's approach right with some yaml in an sem and now we've got this fragment in a database so we've got this hybrid right of data and two sources of information so you might be thinking well when i'm trying to update some information in the catalog how do i know what where to update it so we've got a partial solution to this which is trying to make it more visible to developers where this information is actually coming from so this is the entity inspector which you might be familiar with which shows you details about an entity and as you can see here that we've got some icons for stuff that backstage itself is actually injected these are kind of annotations which the rest of the process relies on we've also got some things that rodies added via this fragments decorator so in this case pager duty and sneak and we've got some things that came from third party source i.e the sem and yeah for for instance the github github annotations you can see that so this kind of helps people figure out where this information is coming from now another potential thing you might be thinking is hey this sounds a lot like vendor lock-in right you're taking my data and you're storing it in a database and how am i going to access that when i want to move well we give you an api and access to all of your entities so you can pull that data and do what you want with it so that hopefully addresses that that problem so we're pretty excited about this feature it feels like it unlocks a lot of new opportunities in backstage to do things in a way which feels much more natural for users right so we're going to give you a little idea about what kind of things we're building at the moment with this new functionality and this new approach so the first thing that we wanted to do is allow people to fix these annotations but on a much larger scale right so you can imagine that it's useful for individuals to be able to fix things quickly on a single entity but what if your organization coming in and saying hey i want to set up page duty for my organization i'm going to have to go and update thousands and thousands of entity yaml files all in one go and that's going to take potentially years right and that that is the experience of many of our customers it does take months if not years to get to this level of completeness in their catalog and so we're building features like this which will allow people to go through a list of entities and basically in a similar way to what we just saw we have a list of page duty services and we can just associate those with a few clicks and save it and that would then just update our catalog and we'll have working page duty plugins for these these entities within a few seconds what else are we doing so i mean this kind of unlocks some cool things like ownership okay so we've at rody we're very we're interested in this problem of richness like helping you get a rich catalog which has working plugins but we're also interested in helping you get a complete catalog right and part of that has been pulling in repositories as a kind to the catalog so that you can see immediately what what repositories you have in your scms and an important part of that is knowing who actually owns that repository right now we've this is our repository table in our catalog where which we automatically generate and we've added this button in the owner field which allows you to in one click just add an owner right and this we're pre-populating this based on figuring out who's contributing to the repository already and which teams have access to that repository and what level of access they have so we're making an educated guess on that in allowing you to just go through and just add ownership to these these entities here's a little clip of it in action so yeah i can just click these and we're going to update our catalog immediately so that's pretty cool and we're quite excited about the future of of this and expanding this into different areas of the product and we feel that it's much it's a it's a much more natural field to like see backstage as a as it kind of a place you can update information immediately right rather than having to go through these convoluted flows or being blocked completely because of where your entity happens to come from so i think i'm going to pause there and see if there's any questions from you guys great test yes we're on hello do you have any plans to add the some featured regarding closing the loop basically top are gating automatically the information back to the catalog yaml file yeah i think way yeah 100 yeah that's that's exactly what we're looking at at the moment is this idea of like raising pull requests so when you create a fragment you automatically raise a pull request to the sem that that entity is originally sourced from and you can have a series of numbered pull requests and you merge them in the right order and then you kind of you're reconciling your your other source but it yeah any plans to open stocks the plugin yeah at the moment we're not we're not going to open source it and we're still kind of building it out and i'm not sure about what that will look like in the future but for the moment unfortunately not but as you saw as i showed you with the back end it's it's relatively straightforward to build this i mean it's just a custom processor and an api and a new table in the database and so yeah any other questions one on the back best oh now it's working thank you for the insights for this plugin one question clarification because you mentioned about the vendor locking so there is no possibility of using the database that we are already having it has to be a database that is hosted somewhere by you no no this that you can anyone can build this just using existing database and we just added a table to the catalogue database and and that's all you need to do really okay thanks any others great all right oh no okay all right well thank you very much for listening um we hope you found that interesting um we're roadie and we're here helping people adopt backstage quicker and more effectively um we're just outside the entrance there so please come and say hi and chat to us um if you've got any interesting stories about your own adoption journeys um we're here for the whole conference uh booth d34 so yeah if you can't make it today see you later um we've got some nice little ducks which you might have already picked up um so yeah you can do some rubber ducking and thank you very much thank you