 Hello, everyone. My name is Johannes Deans. And today I'm going to talk about my experience in a real DevOps team with a technical architecture documentation template ARC 42. And I will share a lot of lessons learned, what we saw, what really works in our team. And it's hands on experience. So you should be able to take it home and use it the next day. So let's get it on. First, a little bit of context. I work at Deutsche Bahn, especially a subsidiary of Deutsche Bahn. Deutsche Bahn operates worldwide, but mainly in Germany with over 300,000 employees. And I work at a subsidiary called DB Sustell. And we're the digital partner of the Deutsche Bahn. We're driving the digitalization of the Deutsche Bahn. We also provide a lot of services. And we noticed in 2018, around 2018, that we're not providing the best service we could. We were separating dev and ops. And we said, okay, it's not very good to separate dev and ops. So we are changing this now. We want to develop a lot of services, which are mainly provided by us and sold to our customers. Customers also subsidiaries of Deutsche Bahn. And we want to produce these services and operate it in a DevOps production model. And the DevOps production model in DB Sustell is DevOps. As you mean it, you build it, you run it. So what do we do in my team especially? In my team, we are developing a product. And that's also pretty new. We're normally only a project project delivery for our subsidiary. So the subsidiaries come to us or customers come to us and say, we want something and we build it. And then it's owned by the by the subsidiary. That's not what we want to do in the future. We are going to develop services. And that's what our team does. One of the first teams that that does this thing, we are building the developing the DB content hub. And the DB content hub, it's a hub. So we are hosting a headless CMS with an API on an API gateway platform, API management platform. And it's content as a service. So you buy our product over self, over self service. And then you get a cluster and instance of our product. And you can start creating content and deliver it over an API or over, over front end. And what we do, we have a surprise by software we host in our backend. And we're doing this self hosted in cloud, especially HWS. And we are doing this in a DevOps production model. And as I already said, you build it, you run it. So we're talking to customers, we're implementing the features, we bring it to production and we operate it. And that's pretty special because you see a lot of different flavors, flavors is maybe a wrong word, but flavors of DevOps, where you say, okay, we have a DevOps team that operates everything. So it's ops team. So we have the whole built chain of the software. And that's pretty special, I would say, especially in, in the context of Deutsche Bahn, which is normally a very traditional company corporate corporate environment. Well, that's the first thing. Then we are doing product development and product development is a little bit different from project development. So we don't have a customer that says, here, I have an idea, please implement it. No, we develop a product and have to sell it. So content as a service means we have to win our customers. And that usually implies that you have short iteration cycles to see what works. We are still in the face of see what works. So we have short iteration cycles. We also are doing a lot of mistakes. Yeah, that's that's part of product development. So first, we tried to use a managed cloud, which didn't work because we couldn't scale a scale. Then we switched to HWS. And we also tried OpenShift, for example, but they all HWS is the right thing, but we use the wrong technology there. So we're making a lot of mistakes and have to take some decisions back. And we have to learn and adapt. That's pretty cool. But we also have to deliver on time, because we don't have we don't have infinite amounts of money, which is we are sponsored by Davises Tell. And then we have to create value in a short amount of time. So this has a lot of pressure that lies on us. And what's also special and it's kind of necessary for DevOps to work, we have an agile production model. So we use firstly used scrum, then we use Kanban at the moment. So we are really agile with inspect and adapt and see what works. And we are also very customer orientated, because we have to win customers, we need to know what our customers need. And so also the team, that's special for our team, where we talk to our customers and see what they need, help them integrate our features. So what I will focus on in this presentation is the cost of poor documentation and documentation in the sense of not customer documentation, but in technical architecture documentation. This is where R42 comes in. So we have this fast iteration cycles and product development. And we have this pressure of delivering fast delivering features and a lot of features. So what happens when you deliver fast and have short duration cycles? Normally, if you don't pay attention, you have not, not the best documentation, documentation always comes last. It's even after testing. So testing comes last and then comes documentation. And that's a real problem. And in a talk from Kevin Goldsmith, he said, one employee said, documentation is not agile, because he understand the agile manifesto wrong. So first, I want to make clear why poor documentation costs the team a lot. So the question is, is why do we document in the first place? In a DevOps team, normally the team should be stable, but he had some kind of flukation in there. And every time a new employee comes in, he has to learn the whole system. Our system is a very tiny service, not tiny small service. But it's also a lot of techniques, frameworks, methods, a new employee has to learn. So the onboarding usually takes 25% over a few weeks of one of another team member to onboard this new employee. If we can reduce this percentage by even 10%, we free the old employee, old team member of a lot of work, which we can then use to implement new features or make the better documentation. So every time you document, you should document in the sense of, okay, there's maybe another onboarding or onboarding of another team member. And also, we're in a corporate environment, we are also documenting documentation, because we have to. There's also must haves we have on documentation, for example, an operations manual, and technical architecture documentation, architecture decisions, we have to document these are must haves, not optional. And that's more on the side of user documentation, but also for other teams that want to work with us. Sometimes we document for other teams that they understand how how to use our service, how our service even works. That's our important part, especially if you just lend a front end developer, for example, he has to implement in our, in our backend something to have a nice front end there. Then the documentation should be because you have to onboard it on board the new employer. And finally, and that's more of the more in the funny side of things, I have this thing called developer amnesia. And everything that happened four weeks ago, I don't know about it. So I often document for myself. That's, that's a thing I had to use in this slide. I think it's it's a common thing. It's so fast moving development cycles and you're just forgetting things. In this talk, I want to put the focus on arc 42, the technical architecture documentation template. Normally, I ask, who knows arc 42, that's not possible in the video, of course. And therefore, I assume that only 50% that's normal percentage in Germany who know arc 42, the technical argument, technical architect documentation template. So I'm going to give a little overview of arc 42, not too long, maybe one or two minutes. So arc 42 consists of a structured template, it's a template, a lightweight template. And it says, and it was developed by Gernot Stark and Peter Ruschka. Hands, they had hands on experience year long handles hands on experience in documenting technical architecture. And they found out that this structure really helped them to document everything the same way. And that it was useful. So the first start with introduction goes, what is this system you're building about then the constraints of the system, context and scope. Very interesting is the context and scope for if you don't have a context diagram for your system, you may want to create one tomorrow because you will find out that a lot of systems you are talking to and they are talking to you and you don't know about it. Then you have the solution strategy. And then come the interesting more of the what you think of architecture diagrams, building blocks, deployment, few runtime fuels, then the overarching concepts, for example, logging is an overarching concept. And then come a very favorite, favorite idea of minds architecture decisions, quality scenarios, risks and technical depth, which should be mentioned in the glossary. And we're going into some of these in detail. But you can see they're all they're a little bit more detailed on the right side. So what our introduction goes quality goes, they call the required functions and how the business domain is mapped to the building block view. And the technical view is in the building block view as well. You can break it down the building block view into deployment view. And later on the runtime view, where you're calling functions. So this looks like a very heavy weight approach. But I assure you it's not. So what are the requirements of technical architecture documentation in a DevOps team? I would say Arc 42 is perfect for DevOps teams. And I have a few arguments to make there because if you're working a DevOps team, what you want is to move fast to grasp the architecture, the technical architecture of this of the system very, very fast. And it's perfect if you use Arc 42 in your system. If a new employee comes in, he knows he already knows the structure where to find what. So if you always use Arc 42, it's very easy to get in the system. Then as I already said, it's lightweight and lightweight means there's no obligation to fill out every single chapter. If you don't need a runtime view, if you don't need a deployment view, don't fill it out. If you don't have architecture decisions yet. Yeah, there are no architecture decisions. It's okay. So create them on demand. Then not specifically important for Arc 42, but for every documentation, just document what you want to read. If you want to create a runtime view because you need it, yeah, okay, you want to read it, then create it, document only what you want to read. That's important, but not specifically to Arc 42. And there's also something I learned in the last few weeks. If you have to create an operations manual, often the technical architecture gets duplicated there. And yeah, you can reuse the technical architecture in from Arc 42, especially the deployment view in the operations manual or a modified deployment view. So you don't have duplicate of application of work. So just to make clear what we are seeing in our DevOps team, what we really need in our in our technical architecture documentation, we are very hands on everybody. There's we are cross functional, T shaped, as someone might say, that's okay, but we also we all know our system inside out. So we found out that we need introduction, goals, constraints, context and scope, and the solution solution strategy very, very much. We created them in our Arc 42. But we don't need all these views. The building block few we created it, because of as an exercise, I created it. But it's not very necessary because our system is small, we know how it's built. And it's, in my view, very easy to grasp. So you don't have to create these three views. What we then added was concepts. For example, as a DevOps team, we knew we need observability, monitoring and alerting. So we added a concept view in the last few months. And we are documenting architecture decisions and quality, quality is a little bit of a step child. Unfortunately, we have to work on that. And we also have a risk and technical depth. What we didn't have was a glossary. And I recommend to everyone of you this the first learning you should, you should take home a glossary is very, very much needed. Because I was sitting in a customer integration workshop. And our product owner was talking to the customer and he always said, yeah, we have this content asset, we have this content assets. And you don't know and the customer didn't also know there are also content types, which is basically the content data model of the content. And the technical technical term for content asset, technical implementation of the content asset is a content led. So I was sitting there and thinking about what is he talking about content asset, content asset, is this a content led? And the customer thought, oh, yeah, content asset, content as it's a content type. So there were three people talking about the same thing, but having different things in mind. And this is very much mitigated by using a glossary. We say, okay, here's a glossary, we use this term in our team. And then you can give it to the customer says, okay, we use this team. And it means that that's very important. The first thing you should do is creating a glossary of terms you use. And since we made that clear, and we created the glossary, it's, it is better, we're all always talking about the same things and meaning the same thing. So a lot of misunderstanding is mitigated through that. So the first thing thing you should take home. Then if you're using arc 42, there's a confluence template, a wiki template. And normally a wiki in a corporate environment is easy to get. Plus, it's also this way. Then we say, okay, we create a wiki and put our arc 42 documentation there, a wiki is born. Yeah. And what happens there? Everybody writes in this wiki because everyone has rights, right permissions. Yeah. And what happens? The technical architecture, architecture documentation sometimes gets filled with tutorials. And the tutorials are mixed with, with technical architecture documentation, and information ideas and everything else. So okay, there you have a tutorial, but it's very domain specific for specific customer. And technical architecture documentation should be free of tutorials. That's a point I want to make. Because we experienced it and it didn't work out. So we had all different kinds of information in the tutorial. And it should be somewhere else. For example, if you have an API repository in your GitLab, then put it there, put it in a readme in the repo, you can already read you can reference it or include it on in the right user documentation, for example, no tutorial and technical architecture documentation. Then what we, we experienced in the year we used a wiki was the wiki is a problem child. Because it has a lot of problems. You might notice saying a wiki is the place where information goes to die. Yeah, that's okay. Somebody puts his thoughts, his heart, his ideas in the wiki. And it's in a very unstructured way. And then the information just goes to die there. Because everybody can brain dump. Everybody can brain dump his ideas into the wiki on a place where it doesn't fit. And this usually, usually leads to developer pros and developer pros has everything and diagrams, ideas, comments, tutorials, examples, everything you can mention is in this Enwickler pros and this Enwickler pros is very hard to read. So and then you add a comment function and everybody knows comments get outdated and nobody deletes them. So it's just not working out for technical architecture documentation. And also you have this, this mixture of domain specific ideas, information and technical documentation. And this is very hard. And we learned it the hard way because we had a big customer and all our arc 42 documentation was, was around this domain of the customer. And we had to separate it because the technical architecture documentation of our system is generic. It's content as a service. And we have to head it head to separate it. And we are still finding some references of this first big customer. Yeah. And that's a very sad state. Normally wikis are in. And I would say every we all have this dream of creating this this cool wiki as a book. And it's a nice castle and it just doesn't work out. And the main thing I think or I saw in our team was and I see it in a lot of wikis is that the quality is bad of the documentation, but you have to ask the question, why is the documentation and I think the main reason is because everybody body can brain dump in this wiki. So you have no review, like in code, for example, normally you should review your code. And then a lot of problems get spotted before it enters the master branch, for example. So the quality is bad because there's no review. And there's another problem. If you're moving in a DevOps environment and have very fast release cycles and you want to version documentation, this is very hard in wikis, or for example, in work, but it's not versionable per se. So I really would say wiki wiki is not the right tool for DevOps team. And I want to stress this, this information wiki is not the right tool. So what is the right tool? Then you ask. Yeah, we figured out that the switch to docs as code is the right thing. Why? We are using code all day and code is basically text. Why not write our documentation also as text and version it like we version our code. This led us to, okay, create a git repository and put ask it in there as documentation. You go from VCWIC, VCWIC editing to yeah, we have code, we can version it, we can review it, we can do everything. So the advantages are clear. We use the same tools. We use the same tools for documentation as we use of for code. And then it becomes much easier to be enthusiastic about documentation because you don't have to switch in an ugly format like for example wiki or docs, docs like word or another processor. You just have to write text and that's much easier. And it's easier because if you use for example ASCII doc, there are a lot of great plugins for vicious studio code. There's a very, very great plugin for IntelliJ. I use it daily from Alexander Schwarz. You should check that out. Or if you're a more purist, you can use the bash and WIM for example. So if you treat your documentation as code, you don't need a special program to edit your code, your docs. There are also more advantages, which I already scratched a little bit. If you have documentation as Word for example, or as a lot of Excel sheets out here in a corporate environment, I know, then you're going from the paradigm of wisiwik to wisiwim. And wisiwik has in my experience annoying problems. They are not as big, but they are very annoying when you create documentation, which makes you dislike creating documentation. For example, if you use wisiwik, then I often have a heading. And if I use this heading and say enter, then sometimes it stays a heading and creates a new line, which I don't understand. And sometimes it's standard text afterwards. And this is not deterministic. It's just annoying. If you write docs as code in, for example, ASCII doc, yeah, it's just text. And it's wisiwim. What you see is what you mean. And what you see is what you mean. It just gets rendered and it looks good. That's cool. One of my main points for docs as code is you get the same workflow you use for code like you use for docs. And I want to stress this. In our team, we experienced much higher quality of our documentation when we started protecting our main branch and that could only be pushed to the documentation repository over a feature branch. So if we have an architecture decision, we just create a feature branch, which is HDR042, for example. And then we create this HDR. Normally, I'm creating the HDR because I'm enthusiastic about HDRs. And then all of our team members can quality check that HDR at their own ideas. And they get reviewed and we have a discussion in our GitLab and normally what gets merged into the main branch is a lot of higher quality. So you always see the AsciiDoc format here. So you might ask why not use Markdown. It's much easier than AsciiDoc. Yeah, AsciiDoc is a more complete standard or more complete specification. If you want to want to put it precisely than Markdown. Markdown has a little specification and it lacks, for example, tables. And these tables are a big problem because if you have a lot of different Markdown flamers, just use GitHub and Stack Overflow, they're completely different flavors and they're not compatible. So if you're not, if they're not compatible, then you have a really big problem because you can't switch from one system to another. That's not a problem with AsciiDoc. Everybody implements the same specification, the same standard. So if you have Pandoc or DocToolChain or your plugins, it always looks the same. So that's a big plus for AsciiDoc. Then another big plus for AsciiDoc is the possibility to use includes. So you can structure your documentation modulary, modular. You can structure your documentation modular. Let me stress that. Modular. So if you have a readme and in a repository and you want to include in the user documentation, yeah, just create a user documentation overarching AsciiDoc and include that file. Don't create it new the documentation, just include it. And you can structure everything modular and it's pretty cool. And then you say, okay, you're in a corporate environment and you have to deliver, for example, Doc, PDF or PPT, yeah, no problem. We have tools for that. So AsciiDoc can, with the right tools, you can create every format from AsciiDoc. That's very powerful. So how does documentation really look like in our team? Firstly, we started with a single repository for documentation because it was easy. Put everything in this repository. In the last few months, we wrote an operations manual and we found out, okay, the deployment should be in the in the repositories. We have a lot of Docker containers at the moment. Four for one cluster deployment. And then we said, okay, how do we create the deployment? Yeah, this should be in the repository of these. The documentation for creating it should be in the repositories. So we just moved the documentation for the deployment into the repositories. And when we create, create a user documentation or create a technical documentation, then we just check this out as a sub module and use, for example, DocToolChain or you can use any tool but DocToolChain is developed by a colleague of mine and it's very Ralf de Müller and it's very sufficient in doing this. And we create everything from these documentation, for example Word or PDF and you can even style it. So it looks like you created it from Word. It's very cool. And what we also use in our team is a thing called MicroSight. We render from our central documentation repository on MicroSight and HTML static site with a static site generator. And I will go into detail a little bit more because I think as I already showed DocSys code and Arc42, it's more or more or less a laugh story. You have these same tools and you can use the technical architecture documentation. And there's also a template for Arc42 in ASCII.Doc in the right repository which is linked in the last slide. I have links to this repository where you can just check out Arc42 in ASCII.Doc. So I think it's a laugh story. And we create this MicroSight. And this MicroSight has a lot of things in it. Don't worry, it's in German, but architecture is architecture. And you see it's a static HTML. This is a screenshot of my browser. And there you have all the chapters of Arc42 in there. So you have one to 13. And everything that's not filled out, you see it. The screenshot is a little bit older. We don't have the deployment view, chapter 5. And you can also put in this MicroSight besides the Arc42 user tutorials or some deep knowledge or some infos or FAQs into this MicroSight. So you can a central place for your architecture documentation or for systems documentation, not only Arc42. And this is very powerful. And there is one thing we have, I don't know if every corporation has it, but we have an internal search engine called DB-Suche from a DB-search. And an HTML site is very easy to index. We have the search button on the right side. And we have a sitemap xml and we just point the search bot. The search has a web crawler and just across our website. It's easy searchable. It's very accessible. And everybody without knowledge can browse this because it's just an HTML page. So it's accessible and everybody can use it. So now you say, yeah, your quality of your documentation is high. But don't you have outdated documentation? Yeah, I just made a year end clean out of our documentation. We have outdated documentation. I don't want to lie about that. So how do we deal with that? Our technical documentation moves fast. We change a lot of things. We switch from cloud formation to terraform, which made technical decisions obsolete. And the rework of our architecture was needed and we didn't rewrite everything. We just missed some outdated documentation because of moving fast. So we have this two-hour slot every Thursday where we just go over the documentation and see how the documentation works. We review a documentation that's older than one year in this time slot and say, okay, do we need it? Can we disregard it? And the best thing about it, if you version it like code, then you can just recall it from Git history. There's no problem in it. So deleting documentation is very, very easy. Then I have a few more minutes. Sometimes I skip this section, but let's talk about architecture decisions. In our team, and I have a whole other talk about it, if you want to hear, just give me a call. I give it in a meetup. I have two talks about it actually. So what we found most valuable in ARK 42 were chapter 9 architecture decisions. And architecture decisions are normally decisions which affect your architecture, but in a way which are very costly to change or not even possible to change. And we use these architecture decisions for these decisions, but also, for example, tool decisions. A lot about automation and DevOps is about tooling. What tools do we need? And don't use too much tools, but one tool. So we are very, very diligent about choosing our tools. And it's very sad that it's on the ninth chapter. I would put it a little bit up there, for example, or after four. But I see it makes sense to put them there, but we are emphasized architecture decisions because we had a lot of discussions about technical decisions and we want to distribute the knowledge of our decisions. We did to even, it's even mandatory in our corporation to make them transparent decisions. So we do that. So let's talk about the format we use for architecture decisions. And there's a very popular format of architect decisions called HDR architecture decision records. It's not from us. It's from Peter Nygaard. I learned from Peter Nygaard. Actually, it comes from the book Software Architecture and Practice that originated from. And we use the template from Peter Nygaard. I also put in the links in the last slide. You can see a lot of different HDR templates because I think this template works very well in our context. Let's see how an actual HDR is structured. We have this title and this title always starts with HADR minus and then the number 003. I don't think we get more than 999 HDRs to change the format. And then you have a noun phrase and this noun phrase is staging with cluster environment for our CMS. And then after that there comes the decision. The decision is usually one or two sentences sentences long or short and they and states what we are doing now. So do we staging or do we cluster? No, we use two environments and we set it up as clusters. So everybody knows what we decided there. And there are a few status it can have, status. Most is accepted sometimes and even in the last few months we had this a lot. We said okay, it's deprecated or we say there's a new one and link to the new one. The status is not set up. You can use whatever status you want. Then you have the context and the context in a corporate environment is sometimes funny, sometimes strange. So put everything that comes that leads to this decision in there. Even if it's say, oh it's meetup-driven development, your lead architect says yeah, we have to do this because I saw this in a meetup or in a conference and it's cool but it doesn't make sense. So put it there or you have some data or some service you can't access because the corporation says it's not for you, not for use for you but it's perfect then put everything that makes your decision transparent there. Then you have the consequences. The consequences are usually short-term. So what have you do now? I have to do now to set this decision or to implement this decision now. Then you have the alternatives and I want to stress this. This is a very important thing. It's very short here because, okay, I had to fit it on one slide. So normally you have a lot of alternatives. If you don't have an alternative then it's a constraint. It's chapter two. But the alternatives are very interesting in the long term when you say okay, I have to deprecate a decision. Then you can say what alternatives did I evaluate so you don't have to evaluate the next so you're saving time. So now you have the basic architecture decisions we use and how do we document them. We document them in our main documentation template repository and just put it there as a template and then you can copy it. It's very easy. Just copy the file, enter what you know and create a feature branch and then we render it as a block view in our microsite and we even use tagging. So we have a course system and we have an API system which one is hosted on HWS and one is hosted on OpenShift and then we created a text API and course system and then you have an historic few of your HDRs and can see what HDRs did you have and you can also search it so you can see what turns you took while developing your system which is very cool. So to sum it up you learned a lot about DOCSIS code and documentation. Maybe the same as ARC 42 but you saw that ARC 42 is very suited for documentation documentation of technical architecture. You saw a lot of mistakes for example don't use a wiki in a DevOps team. I don't think it makes sense. You should have the same tooling everywhere and then the DOCSIS code approach comes in and if you use the DOCSIS code approach with ARC 42 then I think and I'm very sure that you will be happy in your DevOps team with documentation.