 Can do it one two three it works. Yeah. Yeah, that's yours work, too Caltered for yes one two three one two three Look where this is going can we continue Well, we just said officially the slot starts at 350 right and if we start ten minutes early and people are actually Unless if anybody waits for us, I mean Shall we go ahead of time? You get more questions as a bonus towards the end It's a good point, too Sounds good. We'll do it All right folks quick question because obviously you can hear by our accent And I'm sure you have guests by our names that we are not from Chicago Any any Austrians in the house Any Austrians now we're the only one over there. Oh, yeah, so what do you do? Okay Have you been to Austria? You should so do we get anybody that is at least a little bit closer has anybody been to Austria before define Austrian, right? Sweden count oh My god, we have a different problem than backstage. You hear this off Awesome, so we are from the lovely country of Austria. It's not Australia. We always say cows not kangaroos Some other facts that people may not know about Austria Red Bull is Austrian Even though I think to be honest, I think Mr. Matt the sheets who passed away like two years ago But he kind of screwed his business partner from Thailand back then and he became really famous with it But still we claim the fame. What else do we have our most what's in a car obviously? Mozart Mozart Beethoven even though is German The winner schnitzel Yeah Here we go All about food so exactly but we're not here to talk about food We're here to talk about backstage and the way we are going to do this We have kind of two parts Wolfgang you will bring us through the kind of the journey From backstage at diner trace and then I in the end kind of wrap it up with a little bit of the observability aspect of backstage because we are an observability company and I do a lot of Advising around observability of platforms From engineers, I want to just bring in my perspective but I Want to push it over to you now because I think you have a great story to tell and a good analogy of our journey towards backstage Awesome. Yeah, thank you and thanks for the internet actually I think this is a historic moment since even though we've been working together for over 10 years It's the first time you're in stage together. It's interesting. Anyway, thanks for for joining us and putting up with our rambling I've two acknowledgements or two to things to acknowledge first if you've read the abstract as another colleague mentioned in the guy called Beirnth, so hi if you watch the recording unfortunately, he couldn't make it here. That's why we have Andy and The second thing to mention is back then when we put together the abstract and came up with the title How am I big backstage improved developer efficiency of a thousand plus engineers. We might have been a bit overly optimistic with the speed of our backstage adoption and have to be honest that we're not quite there yet We're on a pretty good trajectory. I would say But yeah, we haven't yet met that number. So if you're here because of that share below story, unfortunately I I have to disappoint you Anyway, speaking of acknowledges, I didn't want to Go with that. It's a marathon not a sprint thing I wanted to compare it with my wonderful cobit project that did together with my wife Tiling my basement floor, which was also a thing. I expected to be done much sooner than it actually did So that is we've got that going for us um So yeah In this case, let's suit up put on the schwarzenegger gear and get started so how how we started on our Move and on our path towards establishing a developer portal at dino trees um Few years ago dino trees was a company where pretty much all of the devs were working in a few large Monorapos you push code either to the agent part, which is the stuff running on our customer side Or to the server side part, which was a or is a huge monorapo consisting of 260 gradle projects and You didn't have to worry about anything. It just pushed the code in there Versioning was done delivery was done. Um, also hotfixes and so on all of that was already cared for um There are downsides to that over time the project kept growing and growing We had to put in by my effort to keep it fast and maintain it And eventually there was the time when Dino trees decided we want to move towards the dino trees platform and um, that is a New project product where um, we provide apps that are built on top of the platform And the home work for us was I was back then in the team that was a central part of dino trees where we maintained the Internal ci is the infrastructure provided a lot of guidance around how to build and test stuff And the challenge for us was how to keep up with that back then in previous times creating a new project For needed somebody to create the repository create the Jenkins build create some basic build scripts That didn't happen too often so it was fine to do the manually and then we had to figure out how we can keep up with that And for us that year was in about mid 2020 um Let's write something on our own where we can do templating because backstage was Just released. What did we hear earlier march 2020? So we decided not to go with that brand new project that we didn't fully trust at that time But decided to roll with our own and we called it the project initializer It was a tool that you could use to create a template That did all the repository creation templating and so on for you We had a couple of nice features in there We are uh, if a template changes you get updates On your project that you created in the form of a pull request for example We had an approval feature in there. So if somebody wanted to start something somebody had to approve it And that worked. Um, so we were heading Not really towards chaos, but um, it was used quite frequently and people were happy with it But it showed that we were lacking something we were very focused as I said on on the templating on the scaffolding and eventually we realized, okay Let let's stop that We saw the templating is not enough There were so many open questions that we couldn't answer with the tool that we had provided and the question was Do we want to invest further into improving our home ground tooling or do we want to move towards backstage? And since we were at backstage con and not initializer con, I guess you can imagine how that Decision turned out. So we got together in that group of of people running that internal Team we call it engineering productivity. So the guys that are working on on our Jenkins and build infrastructure and so on And we said, um, what do we want to know where do we start again? My tiles in the basement where to start we start at the door So we started at the beginning like a lot of other companies as we also heard today Ownership the big question who is responsible for something in the component or a service who is taking care about it Which resources do we need which apis are used or where is the documentation? Where is it actually deployed? I'm a build screen. Where can I find the self monitoring data? All of that was something we couldn't really answer with the tooling we had So we decided to move towards backstage In our case now we have a screenshot of backstage itself Where we have all of those things covered. We have ownership. We have a link to our self monitoring We have the documentation linked from here Yeah, and the other plugins for CI CD monitoring for the docker images together with the security Vulnerability sensor on we have to then use as well The great thing was that at this time we didn't only adopt it in our own Team and in that group of of engineering productivity folks, but also teams from the In that case the active gates one of the dynatris components Also started to adopt it and on board the the services and the projects in there. So that was pretty cool We also moved on to integrate existing tools The dynatris we have something called dynatris teams Which is a web application that we can use to navigate The organizational structure of dynatris. So you can look for users you can look for teams and of course we didn't want to maintain that structure again in backstage So we built an integration to sync everything in there, which was already quite a cool achievement then that you can see a team from teams What what is owned by this team and and didn't have to worry about getting the users in there getting the right team name in there and so We also use tech overflow for example, so we also integrated that with our with the surgeon in backstage So in that the engineering productivity group, we were quite happy and we thought okay How to grow how to continue with with my tiles and how to how to go on with that And we had a couple of conversations that pretty much went this way with We've got something. It's called backstage. Why don't they use it and the response more often than not what was why What do I get from it? That took us a bit by surprise so Again, we put our heads together and defined a bit of a process for us where we said that that also came together with my Role change towards a product manager. So I said, okay, let's treat it like like a product. Let's look at it a bit more Product oriented identify the stakeholders we want to work with think about the user journey is what are they doing What are they supposed to do? Are we solving the problems that they actually have and then start small and iterate once again, we looked at our Dynatris platform and Try to figure out which part we can contribute most which part of that Platform is the one where we would have the most impact And it was the the top level called the Dynatris apps Where we want to support with self-service solutions and more automation for creating and maintaining apps So for that part of the product there is a Documentation page called developer Dynatris come Where you can see how you can write an app, okay? And we want to automate that so we want to provide a workspace template to train and triggers the ripple creation via A github's approach for github and then we want to move towards an automatic deployment in our development environment Provisioning a workspace in the remote IDE so we use github for that So the idea is to be able to start the app get the repo and immediately connect to the to the workspace in github And also set up the self-monitoring automatically However, there are a few hurdles on that way that I would like to talk about now So on the one hand as we started initially with our initializer There are a lot of existing projects that we need to onboard and that is Manual effort that needs to be done So in our case once again for the apps and the app uses an SDK and the SDK depends on an api There are already a few of that and Now manually creating all the cataloging for the sense that we don't want to do We built some initial automation How we can onboard them and how we can get them into backstage But there is one of the bigger hurdles right now that we need some automation to really Consistently automate not only the SDK onboarding, but also the app onboarding the apps are amp-amp based So there is a lot of info already in the package chase They will also like to use to put it right in the in the backstage catalog The other thing that we saw with onboarding when people are doing it It can be quite difficult To get the format right So what we consistently try to educate people on is to use the entity validator to see If the format is correct and also have the entity checks right Visible next to the component to see that Whatever you put in is complete and fulfills the needs that we have The people that just come in and think why are they already talking because officially the talk started two minutes ago We have a little bit ahead of time here in the conference and we started early. So we have more time for questions And you can also ask what did we miss in the first 10 minutes? Well, we can give you the recording I guess right We'll just tell you again. Exactly. Exactly. Okay. So first hurdle was manual effort for onboarding. The second hurdle is That we have a need for more complex templating workflows So as I said earlier, we had our own approach to templating in our initializer which had couple of features One of them also was to be able to wait for Something to happen in another system. For example, that was where we built our approval on top So you could create the pull request and somebody had to approve it Which is something that we can't do right now in backstage So there is an open GitHub issue where we're also planning to contribute to make To enable gates in templates to make it possible to wait for a PR to be merged for For a user to provide input also for some other resources to be created for example Right now, this is what's limiting us from adopting backstage further since in a lot of cases we depend on Some github's approach So that means that in order to proceed with the templating or with the scaffolding we need to be able to wait for a PR and Yeah, that is something that we will have to invest in And last but not least what we saw is especially during that initial phase when when we started to use backstage There was some some good traction to our teams that were looking into it quite quite early And then we changed something and something didn't work anymore And then we changed something again and people were not really frustrated, but there was a certain degree of feedback like You want us to use it don't break it all the time and we said Yeah, okay, that's actually a nice thing and that is where I'd like to hand over to Andy to talk about Why observability matters in that case Cool. So uh, Wolfgang you basically as you said earlier you changed your role From what you did a couple years ago to then kind of like a product management role where your product is The idp or the back backstage and you want to make sure that this product Is actually running and is adopted and you want to make sure everything works smoothly exactly my PM role i'm responsible for the internal processes that take Code from development to production based on on our platform and Backs to choose a key component of that quick show of hands who in the room here Is responsible for running backstage and also making sure it keeps running and it keeps your users happy All right, are you guys Monitoring and observing how your backstage is running how it is used who is using it Yeah, just a little fewer hands. Hopefully Uh, a couple of tips that I have first of all for us Yeah, I like so la la maybe Maybe we are monitoring means when people start screaming at my door, then I know something is wrong That's an an an approach, but not the best one. So The the test that that's kind of we or I try to figure out in before this talk is how can we make backstage observable? fortunately We have a lot of observability opportunities, especially on kubernetes. I'm sure I don't need to remind you What open telemetry is what prometheus is that you have a lot of logs that are created kubernetes events One of the things that I really enjoy and I learn a lot from This is just a little commercial to is it observable dot i o one of our colleagues henrik His mission is to actually look at all of the different technology in the cloud native space Where that's open telemetry prometheus logs to make things like service meshes to make things like Critical components to our platform like backstage and others observable and then use this data to answer critical questions Like is it available all the time? How many users are using it? Can we optimize the performance and what about all the other depending systems because backstage is not a standalone island It needs other systems and when they are down then we all have a problem So to give you just a couple of tricks and also what we use internally One of the things we do is we're detecting Based on the logs that backstage creates common misconfigurations and misusage right, I think you mentioned earlier that People might get frustrated if all of a sudden it doesn't work anymore. It doesn't work as expected The question is is the system the problem? What a way to use the system? And so what we have here is just like based on Based on rules that you can also apply Whatever log analytic solution you have Assuming you have logs if you get the logs from backstage you pull it into a log analytic solution You can extract exactly these Top problems that we have seen in our organization as it comes to backstage And you can then use this data to say hey Either you have templates that are simply wrong and somebody's using the wrong templates So you need to fix the templates or somebody's using the templates wrong which results into errors and you want to also avoid that Log data can give you this information. By the way, we share the slides later on So you can obviously take a picture of these queries and of these examples But we'll also share it so it will be easier so very Simple figure out if something is wrong in backstage based on configuration The next thing and this is a very simple one I've been working in observability for the last 15 years And we have some very simple means to validate Things like is the service up and running Some simple things like a synthetic monitor So looking at the focus again that raised your hands earlier that are responsible for backstage You need to treat backstage as any other business critical system that you have whether it's your company.com Where you actually make money Backstage is as important because your developers are very critical We heard earlier how costly it is to lose developers if they are getting frustrated And developers with cost a lot of money too So you want to make sure they are productive and they can use the tools Therefore setting up a simple synthetic monitor, which whatever tool you have that tells you is backstage up and running So you don't need to wait for people knocking on the door or Leave in the company because their productivity is not good because of all these tools So a simple synthetic monitor is really easy to set up whatever as many tools out there We obviously use our own product, but you can use whatever you want. Just a good tip Next thing is really use a monitoring. You said you want to know who is using it, right? Because you're talking with people you talk to the our development platform team I mean that for the for the developing damage with apps And you want to then actually know are people actually using it? Where do people From which different engineering locations as you can see here We are we're heavily engineering is heavily based in europe and in austria and in poland So we can actually use really use a monitoring data to figure out if our colleagues are really going to backstage And what they're doing in backstage are they really using the latest features that we've just pushed out through an internal blog post through an internal educational meeting Another tip Sorry, not only for monitoring whether people are actually using it But also to have some metric to to track usage And also to be able to argue why we invest in that area Why we we have to have something like backstage to have a developer portal if nobody is using it Okay Why why don't we put all the effort in and that's why it's really helpful to be able to track the adoption In in in one place. Sorry. No, that's good. That's Then the other thing is From a performance perspective I'm sure you're all frustrated if you try to use a tool and it takes forever To load and it doesn't do the job. So performance Optimization is as equally important to tools like backstage or any other tools you have on your platform Then for your business critic lab So you can leverage Your real user monitoring data You can leverage your built-in browser tools to actually figure out why are things slowing very slow to load And then combine this with what you mentioned earlier open telemetry to figure out what is slow within backstage Maybe there's a problem on backstage itself or the depending components and coming to the depending components Backstage is not an island that is on its own because it needs and depends on many other systems So what you see here is one of the visualizations of a distributed trace So what you should figure out is how do you get your open telemetry data? From backstage to all the other tools it connects to So that you have enough evidence to ac. Oh my god, we just introduced a couple of plugins They all now have dependencies to tools. We've never thought about You actually see where things slow down where things break Because if you are bringing backstage into the organization, you're not only responsible for Backstage that little component, but you're responsible for the end-to-end experience That you provide to your developers and that includes everything from backstage to get to whatever else you are connecting with backstage And I think with this it's a hopefully a couple of observability tips and tricks remember logs you can Figure out misconfiguration Setting up a synthetic monitor to validate that your backstage is up and running And if it's not up and running then you can react proactively and not wait until developers knock on your door Then figure out who is actually adopting it uh, and then Performance optimization is a good thing and making sure that you understand all of the dependencies Every plugin will make your deployment more complex also more powerful, but also more complex And with this hopeful this is good enough great data for a product manager like you To actually that is responsible for something like backstage To then not only build a beautiful product, but also to get the adoption up and with this Finish your project and and see where we go next and what we what we do next. Yes Okay, thank you So we have reached the stage where we can Think about putting Stuff back in Um, what are the next steps? What what do we put in so for us? It will be more automation supporting the onboarding for npm based apps So as I said our apps that we we would like to see in backstage We are working on the prerequisites to make sure that we are actually able to Also cover the various aspects when it comes to sdks and other other libraries that are that are needed Second for us is extending the templating capabilities So the gating I talked about to enable backstage to wait for prs Before proceeding with the templating to allow us to proceed with our github to use cases And the third internal topic is establishing backstage as the cornerstone for developing Internal dynatris apps based on the tool chain that we provide around github and github actions So what are the key takeaways? First as I said it's important to treat the idp like a product not As maybe we did earlier think think about it like something that is working as a side project and Takes us to the second take away build it and they will come does not work at least in our case. We've seen that The the intrinsic drive of people to change the behavior is is not as high as as we expected Third for me was except that it takes time It is a huge change when it comes to adopting and adapting the workflows in the company in the dev team Not to act too intrusive and maybe get adverse reactions It is important to also acknowledge that it is a huge change and that that takes time and the fourth is Apply observability in the same way as you do on your business apps. As I said, it's a product I have to treat it like one and then take it seriously Here are codes for the feedback is up with that we come to the end My basement is done. I can put my bike back in and my lemon tree which has died in the meantime Anyway, that's it from us. Thank you. If you have any questions, we have time left Thank you for gone Do we have any questions? Hi, thanks for a good presentation. How long did this journey take for you? Um After realizing that we don't want to continue with our internal development efforts We started deploying backstage. I think mid last year. So a bit over a year, I would say in the tiles project Sorry How long the tiles project the tile project a lot the tile project the tile project that that started in December 2019 and I think the last thing was done in june. So that was also longer than expected But I realized that Things take time. You have to let it set you have to dry stuff and so it was I underestimate it Any other questions? I have the mic There was a part where you're talking about like your onboarding workflow and things like Once the process changes, right as it evolves, how do you keep? How do you keep things in line for the apps that have already applied the old workflows or the old skeletons? How do you make sure that as you evolve the workflows and onboarding you keep those apps that already applied them Inline with the new changes because that's a big problem that we have that we don't have solutions for That is also one thing we are We are working on right now in our old system, which was used for a lot of these apps already. We have the way to Take updates that are done on the template and pushed it out to ensure the things are in line with what we change on the Templating side Currently our approach is more to move towards renovate to extract things that we want to keep in line to some library that we can update Either for example as a Jenkins shared pipeline library or with our move towards GitHub actions To separate it there and to be able to update this not why are why are actually changing the template But by extracting that from the template and making it a normal version update that you can do through other mechanisms But would be interesting to learn more how you're tackling that maybe we can chat afterwards Any other questions? Yeah, i'm just curious with your With with implementing backstage and dynatrace are you just displaying the dynatrace info in backstage or are you Actively creating some kind of processor to create a dependency graph Or how how do you how do you utilize that within your org? I think that was actually more a question towards you Yeah, i feel if you're talking about how we bring our observability data from our product back in so we actually had Tell us is one of our customers in like a telecom in in canada. They started Like a year or so ago building a plugin For backstage to bring in dynatrace data. They actually now contributed that plugin to dynatrace So we now to go with that plugin and we will extend it to bring in more information like we as you I'm not sure. I guess you know dynatrace. Maybe yeah, so we have our dependency model We have mostly have Our metrics logs and traces so our goal is that we bring exactly the time the relevant observability data Onto the right components in backstage so that you don't have to always Find this data and dynatrace, but you get it and there's a plugin already out there And as i said, it was initially developed by tell us They contributed or they pushed it back to dynatrace and now we're taking over and we have a certain Maybe we'll be interested to have a discussion because I can then bring it back to our product team to see what your thoughts are Any other questions? So thank you so much. Thank you. Thank you