 Good morning everyone Let me quickly introduce us on the stage today. My name is Andrea Stolzendalger And I've been with right up for six years now. I work in partner enablement So I do technical trainings with partners on cloud and infrastructure products Originally my colleague Marcus Koch should have been here because he's the SAP specialist who actually did a lot of work of What you're gonna hear today with me on stage is Thomas Ludo from SVA one of our partners and he actually did Most of the work that's behind those slides. So he's gonna talk to you For the next 15 minutes or 40 minutes So just a quick question We're gonna talk about Ansible automation for SAP HANA installation Who of you is familiar with the SAP products and what they do? Just a little bit. Who of you knows HANA? Not all of you. They're just for a very brief and quick introduction SAP German manufacturer. They create a lot of business applications for Critical enterprise resource planning and management purposes just as an example If you go into your local BMW dealer and order yourself a new series 3 BMW What will happen on the IT side is that in the back end the ERP system from SAP is going out and plans the whole Production of the car in its workflow. That means it will say okay first Wednesday of May this car is going to be manufactured in Munich factory 3 room 25 or whatever So to get it done I go out to the partner system of Brose which is another manufacturer and say give me the cabling for that car Two hours ahead before the production to that location. So all the logistics and everything is done So this is a very complex piece of software. You might have heard of the term R3 Which is the whole package? R3 Netweaver was the full name of it which has been in the market for a couple of years Now in the past SAP was based on just a sequel database like DB2 or mostly Oracle And it was running on Windows. It was running on AIX It was running on HP UX wherever you could think about it Now they're going the next step and SAP HANA basically is the new database underneath all that ERP CRM system and HANA there's two stories about the name one says because then the boss of SAP Is his name is Hassel and they say it's Hassel's new architecture. That's why they call it HANA But in the real time they they call it a high availability and performance analytics database So the basic thing is HANA will be the basic and the only accepted database underneath the next architecture which is then S4 and In addition to that the new versions of SAP will only run Linux So they quit support for Windows for AIX for all the other stuff And there will be two Linux distributions supported one of them Susan the other one as well HANA the problem with HANA is it's very fast because it's an in-memory database So everything is stored in memory and that makes it a little bit complicated To get it installed and running Because they use the memory differently than other applications do so they need a lot of modifications to the operating system And the system settings before they actually can run the database correctly and SAP since it's such a Enterprise critical application the manufacturer requires you to go to a verification process For your food installations And if you don't follow all the steps that they require This work this environment does not get supported and that also goes deep down They look at the nick and the network cards and the chips of your network cards that you have in your server If this is not a Service and a supported version of a network card. You cannot run this environment in production Supporting so you have to be very careful when you install an SAP environment, especially with HANA. That makes it quite complicated and if you do that just Standard-wise with Standard installation it will take time to set up a very big HANA environment And this is why Thomas kicked in and said I'm gonna automate this With multiple tries until he came up with the with the answerable idea. Yes, that's right. Pass it over now to Thomas Yeah, thanks. Yeah, my name is Thomas. I'm from as far as fate SVA and I'm working with linux since 20 years and at the moment I Automate enterprise environments and that's the point. I'm here to show you the way I've developed the automation process For SAP HANA deployment with SAP HANA with Ansible and one thing of SAP HANA database is that is one thing it's real Not typical in the linux environment is that the SAP HANA database isn't Isn't published as an RPM package. So SAP is using a binary to install the complete SAP HANA database and This is on your command But the point is you had to do a lot of work before the installation of SAP HANA database and this Configuration you had to done is saved in SAP nodes It's a documentation and it's a lot of It's a lot of documentation you have to do by your own by by hand To configure the system until the state you can install the SAP HANA database in this case Yeah Yeah, you had to Use the sub nodes and you had to enter every command or every configuration which is described in the sub nodes if you don't Configure this as described in the sub nodes SAP would say later this system is not supported and you can't Run you have you could have performance issues and you can't you don't get Certificates that you can run SAP HANA on a productive environment and so it's very important that you can use every point of the sub nodes of the documentation, but there's a huge problem because the documentation is so split it that you have to search a lot of sub nodes and Use every sub node in the environment to use Yeah, to configure the system and you don't have the complete overview. What if you're done and when you are doing it manually You lost a lot a lot of information or you can lost some sub nodes and in some cases some customers were coming to me and ask When I've completely finished the system is this sub node already included and I saw the sub node for the first time Because it was very special for this environment And so I had to build up or do a lot of manual stuff in after the configuration and installation of the system Here's the example, okay for For one sub node. So he has described in the sub node that you What kernel version is a minimal requirement for an rail for seven four for example and The sub nodes Described very districtly what you need and what you have to configure on the whole environment. It's a good documentation It's described what you have to do, but Overview about all sub nodes is very complicated and you lost the overview about every sub node You have to integrate in your system in this environment. So this is only as an example and Here you have an overview what sub nodes are required for installation for a subana database So one sub node is describing which operating systems are supported. So Could be two versions of subana of this subana database version one and two and When you use one version for example one you have another Supported operating systems as when you are using version two and that described in this sub node For example, then you have a guideline sub node where you where it's described how you to configure the whole whole system on the operating system and When you are using another architecture like IBM power you have to use additional subana node for configuration system or optimize the system for the power environment and so on and so on and one point is The optimization of the network stack. So it's really exactly describe what you have to do that's environment is so optimized that subana is running on it perfectly or nearly perfectly for the use case and Yeah, and one interesting point is when you are using all the sub nodes and you have a virtual environment You can throw it away and use Some other sub nodes or some other documentation Because for a virtualization environment you have to do other settings in this case and another configuration For the subana database or for the installation for subana database Because it's very performance optimized Performance. Yeah, it had to be very performance optimized the whole environment So that's a short overview about about the subana database and This is the process how it is how a database is built up from the beginning to the end and One point is in the whole process Some guys had to build up the whole hardware environment. So that's not one point of this Presentation, but that's very difficult. It's very complicated because you have a huge Service which had to be configured completely right you have to do the cable management and so on and You have to Yeah, configure the whole whole hardware environment and one point is where I started is the linux configuration so the linux is Yeah configured after sub nodes I mentioned before I had to be configured after sub nodes I mentioned before and you have to install this Linux in system you have to configure it and normally you give then the system to the SAP guys and they will configure and install the subana environment and that's some point We are a lot of information We're lost because in sub in some sub nodes are described you have to configure the network correctly and then this stuff is done we're doing are done by the by the colleagues from the SAP team and Sometimes I saw this before or it's very often happens They would configure like a network bonding and only put one interface in it because they don't know what exactly this is and in the sub node is only was described you have to use bonding interfaces and When I saw the systems later for the validation I thought oh my god what had he done or what had done here before and to fix it later and it was doubled work because they don't spoke with me or we don't spoke with each other and so a lot of information were lost and every time the work had Be doubled and done again and again, and I and that's what was one point. I said, okay that that couldn't be The future of my work and Then I started with an automation process and the last point is the maintenance so you can when you have a validation validate system a validated system you can give it to the Yeah maintenance or to the operating guys, which will manage the whole system So as described before The whole process was done a lot of times Bimanual work it was four years ago when we got a project where we had to start to build up an whole environment with 47 subhana instances with servers with everyone has two terabyte of memory and We built it up in our building center because we had to deliver the complete system to the customer at this point and I saw very quick when we don't automate this environment with 47 servers It's nearly impossible to deliver it to the to the customer in Nearly three weeks because it was a time this was a fixed time and It when we don't do it. Yeah, automatically. We don't it's nearly impossible to deliver the system on time so I spoke with the SAP guys and Told some a we had we had to automate because it's very important and after one hour. They said, okay, that's right because The work they had to do was a lot of work for this environment because every server was configured individually So I started with some configuration I used puppet in the first point of view or in the first configuration to deliver one automatic system and Configured everything as it was needed for SAP in the first version and It worked because I spoke with the SAP guys They told me you had to configure this this this and this and in the end of the deployment everything worked fine the special thing was the half of the data center 24 servers wasn't available at the end of the three days before delivery because the guys from the hardware part were missing some yeah network cables and power cables so the typical thing and The servers were available three day before delivery and we deployed with a puppet configuration and delivered installed 24 systems within 20 Within sorry within four hours because everything was preconfigured the public configuration was Configuring the whole system and we had some additional configuration for an automatic Installation method kickstart Which had completely installed the systems as defined so For hours for this mesh for this service was very quick because with two terabyte of memory the booting process by itself is Is going up to 50 and 20 minutes because of the memory check and so it was very Interesting because everything worked fine and work perfectly and the validation was fine for the last 24 servers too because we have Yeah configured it was a configuration management before and it worked very fine on the other side We only had three weeks time and it was a very quick and dirty solution. I will show it Here we had built up An exit file where we input every configuration in it and it is was a long long line for every server and We saved here every parameter for example the sub SID. It's a parameter which is used in SAP HANA for the deployment process for every environment and it's it was a very Huge exit fiber every configuration was saved then we converted it into CVS and then from CVS to the to the puppet variables and it was very very dirty and nobody knows what I have done and It was very complicated in this but at the end it worked very good and every system was delivered as expected so In this case I saw this is very this is a very cool solution It's quick and dirty and it was quick and dirty at this point, but it worked and we have Collected three points of the whole installation process Linux configuration the SAP HANA installation and configuration and the validation of the system because when the configuration management was going on on the Configuration and on the systems, you know exactly every system was configured as expected. I tell everyone It is working Nothing or no server or every server is working at the end of the environment and it's very important because You have when you have in staging environment. I will mentioned it later You can test in your in your test environment if everything is right and not and if it's not right You know it's not right and you don't forget one parameter on one system And that's the thing you can validate the systems because everyone is right as configured on the right side and At this point When I saw the advantages of the configuration management with SAP HANA installation types, I Yeah, I developed a solution which is much more understandable in this way And will be or I could be used in a better way and other people's too because It's very important to have except in this in this in this high performance environment that every server is configured on the right is right configured in this case and One point is yeah, it's an automation for the whole environment and you Instead of three steps. You only had one steps for the configuration management deployment and everything is working fine then so then I Told it before I started with puppet and it was a very huge environment we have to Head build up in the last time well in this time where I've developed the first version and I really fast saw okay puppet is a very complex environment because you need a server in every Every environment and on every customer side you had to install an administration server we have few put on puppet when the customer don't had puppet and Then I thought okay, it would be better when we will use Ansible in this case and It was a very big advantage for Ansible that you have a very fast very fast starting process with Ansible so You can introduce more people into the into the automation process and my first thinking was will I Will I maintain two projects the puppet project and the Ansible project that was one thinking I give up later because it's a it's a huge of work you have to do with with one configuration management in this case and later with the testing tool and So the thinking was when I will manage to Repositories for configuration management. I will use an easy way to convert puppet configuration. I've done with To Ansible and I use some tools which will convert the puppet configuration to Ansible I searched one or two days, but in the end I thought okay, it's not a perfect tool who can do this and I Don't find anything which can convert automatically the complete environment So I do it by myself and then I configured the whole puppet configuration in the first way completely by myself in within three days to have an first Ansible playbook which completely done the work of the of the puppet modules and Here, too. It was nearly ununderstandable for others because it was a direct migration from puppet which were included with with bash scripts a lot of bash scripts and with a lot of configuration tools and I had done it in the best way I have known it would work and But it was only wowed down and The processes were the configuration were done, but I never know which configuration I had made were mentioned in the sub nodes later and the customers were asking, okay, which which sub node is is Which sub node is will be configured with this is the end of the playbook and if this sub node is Configured with this end of the But at the end it worked and it was fine for the first step So here's some some basic example How the configuration was done with puppets? So In the first step when you will start a stop configuration services Services from Linux here stop Pneuma D. It's a Thing it named in some sub node to stop the service to disable the service In puppet you have to enter three lines of code or four Resist so it's a service you define a service you enable this service and you ensure that the service is stopped And so while I've done it in three days It was very easy to configure it in an Ansible 2 because it's From the configuration part. It's nearly the same. You're disabled the service and you Configures the sale of service to be stopped. So that was the way Why it was possible to convert the whole puppet environment within three days? the same with installation of packages and When you're doing it one you convert it one to one it's Yeah, it's very easy, but as I mentioned before unreadable for others we're un-understandable for others and It's a sample for for Ansible 2 so Then I worked with Marcus and he has mentioned before Marcus Koch worked very strong together and built up and Way to Understand the module it's much more better. So we used ready to use roads from redhead which were published as Linux system roads from redhead this Yeah, role definitions from redhead which will be I don't know is are they already So they were published from redhead in the future and or it already already Published at the moment and this are worlds which you can download or which are Delivered with redhead by itself and you can use it Like a network configuration and something like that and Yeah, we then in the next step we we We we we use the sub-nodes to see if every step in the sub-node described in sub-node is handled by the Ansible playbooks and We use some some first style guides to have a better understanding because we worked with two people then on the project that we know what exactly the other was done and where it was done and how it was done and We reused the scripts snippets I've mentioned before which I wrote we reused This Scripts to instance as Hannah and stunts. It was historical Why I've done this is very because in my mind was already okay, perhaps in the future I can do or maintain the puppet and the Ansible modules but the structure wasn't much more better in this case and It was no way to for proper testing in the whole environment. So We had the problem when Marcus was changing something and I would to deploy it I had to build up in a whole test environment with some small instance with a test environment It's a minimum requirement with 32 gigabyte of memory to have an installation for the partner and then I have to install the operating system and configure the network configuration and then to play up the Starts Ansible playbooks and it would be a long time to configure it because Marcus was testing on his own environment I was testing on my environment and Was there very difficult when he had done some? Basic changes that I would test it on our time or in time because I had to build up the whole system in this clean up the operating system or in Salah new and so operating system in environment which has enough space and This was something a lot of time was lost because When I done the changes Marcus had the same problems because he couldn't test the whole environment and so this was the first version with which was working but was yeah nearly impossible to maintain with more than one people so we built up a process overview what exactly is needed with the SAP HANA installation type and And the first point is we need to in basic environment set ups something like Confirm the subscription management something like third RP third party software repositories with which could be included in the whole environment to have To have to to split up the Ansible playbooks or roles In this way to get a better understanding what we are doing and don't mix The whole environment in the whole process because the playbooks I mentioned before were mixed with everything We thought it could be a thing of SAP HANA part That's one thing I described in the basic or a setup because we have done some network configuration in the playbooks in the roles we have done the NTP configuration in the in the original nodes and We enabled some some repositories in the nodes And that was the problem. It was nearly ununderstandable for us by a safe by ourselves because You had done a lot of work which don't had to do with the SAP HANA deployment and Don't had to do with the SAP HANA sub node. It wasn't described in sub nodes because of the missing test environment so We split it in two parts basic environment set up basic setup and Sahana installation and configuration part And at this point we focused on the HANA installation and configuration part because it was a main and main important part of it and That was a part which the sub nodes were described exactly what to do and what had we had to done and Yeah Yeah, we split it in Yeah, two points the HANA installation and configuration guide one part was OS The operating system pre-configuration. So here's described everything you need for Configure a clear rail system for the for operating a SAP HANA System so the sub nodes are describing the network configuration for example what services you had to start and so on and We split it in two parts because One one part of sub nodes were describing and part way when you are Need when you will Install or prepare the system for a sub environment what you had to done and the second one What was what you had to done when you are using a SAP HANA pre-configuration for the system? And so we split it this and with two roads and this is the two roads will which will be supported in the future with the Linux system roads and Which will be described the sub nodes use one example for a sub node How to configure the whole environment at the moment? I'm developing the other three parts the HANA deployment Which will install the HANA completely from to until to the end and I'm covering a ring at the moment This is SAP HANA system replications set up that's on the configuration where you can build up to SAP HANA Environments you install to SAP HANA environments and they will interconnect to each other foreign cluster configuration and the last part is I'm developing at the moment as a pacemaker cluster set up that it will automatically switch from one side to the other and Yeah, we'll be used for an automatic Switch for SAP HANA environment It's much more complicated as it sounds. It's the first point because with SAP HANA You have the possibility to build up scale out environments. So very huge environments of environments for example for Scale out environments and you have every scale out environment Doubled into into data centers for example and there's a lot of scouring things one the cluster for example can switch From one data center to the other so it's not possible Okay, the cluster had to change now to the other data center and everything is working fine because the system replication for example should had to be started Had to be started again and started in a new way and it's complicated scouring thing in the pacemaker configuration So To have a better overview what exactly What exactly? What subnotes exactly needed for the SAP HANA pre-configure role we started with and process overview which Subnote is used in on which configuration state and That was something it bring us a lot of Yeah, it was possible for us to have a better overview what exactly we are doing and to describe to our customers What exactly? What subnotes will be configured in the whole Ansible playbook? or Ansible role and We used the subnotes Yeah, as a process overview to show it to the customer and to ourselves When which subnotes should be used? and The first point is the operating systems. It's a playbook a role. It's a configuration in Ansible where we checking if if Operating system is compatible with the SAP HANA version, which should be installed and This is one interesting interesting feet Yeah role in this part because this SAP HANA pre-configure part is based on the subnotes and it could be used by by by pure Linux guys because the whole system will be configured without knowing any Configuration you only need the SAP HANA version you will need for for the deployment and the whole system will be configured Without knowing something about SAP HANA because it's really the base and the pre-configure Ration for a complete HANA system and This is some point Which is very interesting From this point of view then the next the next process part is The repositories I mentioned before we change it in the in the first step of the whole process overview But we are checking if the repository as a fight from repository available At this point and then you see as I mentioned before to the power Planning as a configuration for a power system that you have do Another stuff for the configuration of our power system or as another sub node is using and this is Configuration in the puppet roads which check if the system is a power system and use this sub role configuration for this part And then we have different ways with which will be used if it's a real seven system or a real six system And what had to be done you see in this way for a real six system you have three different nodes for six Okay, it's missing here. It's for six seven six six and six five You have three different sub nodes what you have to do when you have a six five system or six six system and so on and For us Seven system you have a sub node how to configure this configuration and You have to a sub node where you what you have to done when you have applications or sub applications Which were compiled with GCC GCC seven you have to do Some you had to create some links for example that everything is working fine and you have to To use this sub node to And when you're finished with sub nodes from sub directly We have included a redhead recommendation part in the Ansible configuration and this are the parts which will be Done the system perfectly for running for sub partner because not everything is described in the sub nodes So when you're not doing a basic Installation for for redhead they are missing it could be possible that you are missing some packages and This packages will described in the redhead recommendations That it that this packages will be installed and some fine-tuning from the operating system will be done in the redhead recommendation and it's Yeah, it's based on a documentation from redhead by itself and we don't find this configurations in the sub nodes and Then the process is finished and you have a complete pre-configured system where the sub guys can start with the sub partner installation for example and That was a very interest And that is one point we started with service a process overview and used this with a combination of Better style guides we found and one style guide we are using at this point is run from the affinity sys group Which described exactly how you had to build up the Ansible roles and you have to The distinction Between the installation of packages for example and the configuration of the whole system You have clear rules for those for the variables for the text and where you have to put informations and you have some guidelines for the documentation for the documentation by the Ansible play a role and by the developing to the Further role and this helped us a lot of this helped us so much and we Could the work together the first time in the whole project and understand what the other one was doing and with this With this clear structure we it was possible to include much more people in the whole project to deliver code and Give us some information about this and extends the project One point is yeah, I mentioned it before the more granular role definition. So we have configured every sub node as a own file I will show it in the next slide and That was was a very good thing because when a customer was asking is this not sub node configured with your Ansible configuration We can show look in this file and say yes, you can See the configuration for the sub node is configured with this configuration file in Ansible and the customers were happy then and main point was Which helped us to recognize the whole development process to build up and test environment in a cloud to build up a test environment by itself and So we can see if some changes were made as the whole installation process will be done or will be finished Without arrows and is working fine. And if something is forgotten We're not not mentioned we're not thinking about it So with the variable files here's example we put example files To the whole deployment process is example file for the configuration of our yeah for the Preparation of our whole systems the password is here in clear, but when you are using a word, it's it's not a Big problem and the other part is this password is only for the installation process You can change it later, and you have fun complete installation complete installation or preparation of your environment we are using in this example two roads the subpan is Subbase settings the subpan a pre-configure settings and the subpan a host agent the subpan a host agent is for example Used for the huge scale out environments for subana because you had to install subana host agent to configure the subana scale-out environment and The next one is the directory structure So we used the mode most important thing as a subnotes as I mentioned before I use this Configuration files and write down every sub node which should be configured in this in the subnotes and this directory part was helping us to work To work much more clear this with a complete system role And you see here the underrated six and underrated seven the recommendations I've mentioned before and under subnotes subnotes and the one interesting point is the variable configuration So we have built it so clear that you have the whole configuration in the in the main configuration file for the Ansible play role and We only change the variables for changes for example for minimal Minimal requirement package minute packages mini versions you need and this will be described in the vast configuration part of Ansible and And here's some some example for the sub node which is describing to install a tune D service to configure the tune D service and In the next slide, it's described how it is done with Ansible so you can very good See the difference see what you have done in the end of the play in the Ansible configuration What you have is described in the subana nodes For this example and here you see the configuration for the tune D Demon which is enabled and to apply the tune D profile and for example when you're using a VMB environment that you had to use another another profile Instead of the normal hardware environment as I mentioned the one of the first slides Yes, it's a different you see here the configuration the one side and on the Ansible configuration part and the same is an example for the network configuration It's a little It's more explained what what we have done here or deeper in the system So on the right side for the network configuration SAP describe what exactly parameters for the system You are using and on the left side. We are configure it with Ansible playbooks in this way so The test pipeline I mentioned before was a very important thing because I started some Month before with Terraform it's our infrastructure as code software where if can build up an whole environment in a cloud A private cloud public cloud wherever To use it To to use to build up test environments It was very good because you can build up the environment you can destroy the environment and it's very very interesting for for for development pipelines and You have every time a fresh install the system so you don't have a install which was misconfigured or you don't know what exactly had we done or was done on the system and You can start your Ansible playbooks every time to this to to fresh systems to see if if everything is working with a normal environment and Yeah, it's a moment we are we are planning to integrate Jenkins with our test development pipeline to see automatically what we have done or what is done with with the whole system and For us it was very important to use the documentation for the automation we have learned in this whole project because Without a documentation later. You don't know what exactly you have done in the Ansible playbooks Ansible roads Or are you can't mention why you have done this in this world's because In the first steps of the first customer Questions were really why you are doing this and you are you doing are you configuring this after the subnotes after this? documentation and then the first versions we don't couldn't say we couldn't say We had to look after it and we look in every configuration file which was deployed and see if the subnotes were already already yeah configured on the system and Yeah, I built up everything from scratch was very interesting because you know it's a very clear insulation and you don't have misconfigured something We never touched touching at the moment systems when we deploying it manually to do some changes from the systems and very important as a version management because It helped us to to have the actually version of the whole environment and one point was the staging of Of the environment so when a user dev core and test environment You can test everything before the before you deploy the whole environment To deploy the ends of the roads and see if everything is working and this help me as a customer example to build up a very huge subpar environment on a solution center a test center for example to you to clone a complete environment which wasn't built up by automatic and the playbooks But I got the data from the customer and it was possible for me in the cloud environment to build up exactly the environment with 36 nodes With a smaller environment which each had only 32 gigabyte of memory And help the customer to test his cluster environment and see what exactly the whole environment is doing and that was very easy Automation process with the endable playbooks and with the configuration of this environment and helped us because I only had I had to fill up some some variables in the conflict in the public in the Endable Playbook and the configuration was finished and exactly clones environment Yeah, and at the moment I'm building up. I mentioned before and complete SAP HANA scale out system Scale out system the system replication and the pacemaker configuration and using this endable playbooks and it helped me a lot of much to speed up my my work because It was possible within one hour to build up a complete environment and with the cluster configuration you can when you have Configured too much on the system and touch to use too much to system for the test and something we're broken You don't think oh, how could I repair it? You only for a way the whole environment and install it new and with this environment you are finished with one hour for example And one point is we have I've switched it With public roads and custom roads the public roads are roads We are publishing at the moment and get up and the custom roads are roads which are Directly for the customer environment like host configuration network configuration local whole local disk configuration satellite connections this are things which are Used by the yeah, which change from every from customer to customer In this case and yeah, that was a short overview about the standard are standardizing The subhan complex IP to environments with subhana for this example any questions So the first question is I saw your overview was in BPM So you the actual flow of how you're deploying the thing and I saw in the actual files You will be basically just put them separately It seemed to me as the steps are were different files depending on whatever Do you basically just do ifs in there and how do you map them like we're yeah the question was how we I Said it so it's required. Okay so We included the configuration in the in the main configuration file and it's included likes the BPM and overview So we have worked with if crash it was if Statements and include the files After the world so when you're using it it's based on the BPM and overview Okay, and the second question is how is the configuration looking now? I would assume puppet also has some sort of inheritance and kind of structuring of the configuration I don't know puppet, but I would assume so I don't understand How can it would have become better the configuration right from the huge excel to your current setup? How would it look now? So you had previously said a huge problem of puppet was you had this huge Excel file where we configure everything right a bunch of variables now I would assume puppet has a way of managing configurations. Yeah, right. I don't know but that's right the configuration with puppet That was the first version with the with the excel file and we have Plated up later to the to the public configuration Directly because it was only a sample for the first for the first part of it In the back The question was if we are using modules for the deployment or if we are just used commands and Here it's The system row definition. We don't use modules at this point We only use the definition which are described in sub nodes and we are using some Some direct shell scripts which are used to start the deployment process in this way Thank you. And the second questions. How did you ensure that the roles I'd I'd important could you rerun the whole setup? Did you need to you need to destroy it before you rerun the setup? Yeah in this case It's not possible to change any so when you are installed and soprano system we are only checking if the system is installed and Is if the system is installed the installer didn't will not install it a second time so we only had to check for the installation process and it It Will watch over if the installation was already done with the configuration of some subana parameters you can get with effect But then it could happen there's some administrator changes something that's no longer compliant with the sub nodes and and you run into troubles in the end when you want to Ensure that the system is set up in the right How it should be Regarding the SAP nodes You know what I mean How could you ensure that nobody makes manual changes? Yeah, the point I was described before it was only the installation so it's only checking if the installation was done and all other parts likes is Pre-configuration of the system like configuration of the network part is doing with the Ansible Playbooks so there are some parts which only checks subana installation Because it's only the installation process. It's not the management of the subana environment Okay, so we don't have time. Yeah Then very fast Just wanted to make a comment to your question which it just had so I'm working actually with the team at SAP Waldorf on our Efforts targeting SAP and your question is pointing into a different project. We are working on which are the inside rules where we also have SAP a specific rules Which went test for the content of the SAP nodes So if you want to make sure that the systems remain consistent with force SAP nodes the inside would be able to do that for you and Then from a redhead point of view we use insights for those who are familiar with the inside service That would check your systems and they would check them with the SAP nodes Once they're all in the insights database and once you have that if your system would violate what we have in the insights database The inside service would give you an Ansible playbook to correct that but this this is the Installation in the maintenance something else Okay Yeah, thank