 Okay, so let's get started because we don't have a lot of time Especially after the try-ons we have we have done so welcome everybody to this after the day of the open-stack Summit where myself actually Even myself Will give you some insight in our ideas this travel with with on the open-stack journey If you have more than three or four slides then it's always good to have an agenda So we start with a short introduction so that you know who is talking to you and also what is the business of our materials We are coming from because this is also important for for the later slides Then actually I will give some insights why we actually did look at open-stack Why we actually have selected open-stack as a technology to be used and then most of the presentation Actually we will spend on our journey, but what was the challenges we were facing and also later on then e-figure some technical Details on the actual setup you're using at the moment, and then we wrap up with a one-slide one-slide summary So while they're still getting prepared I start with myself So my name is Ute Seidle and originally I'm a teacher for mathematics and physics and I spent some additional years at the University and back then actually I got Infected by the Linux and the the open-source virus and luckily I never recovered from from that one and before I joined Amadeus more almost 10 years ago I was working in different areas as a consultant as a trainer and spend some additional years also at HPC environment supporting a big compute clusters for German car manufacturers like Audi or BMW and inside Amadeus I also have I had different roles and responsibilities and since our last three organization I belong to these CTO office, which we internally call the Infrastructure and architectural governance unit and with that one I hand over to Yves to introduce himself Good you may actually ask yourself why Amadeus and VMware sharing the stage at the open-stack summit and on the first part Why VMware and and actually for some reason I Get to this one later So maybe I introduce Amadeus a little bit first so normally I did ask always okay Who knows Amadeus and we had just a chat in front of the room about that one So for the people coming from the US if you have heard about world-span or saber Then Amadeus is kind of the European counterpart of it If you're coming from China or Asian or travel sky Amadeus is kind of the European counterpart and actually compared to open-stack We are quite old company Amadeus was founded by European Airlines in the late 80s and according to those founder airlines We actually have distributed our main functions So we have our headquarters in Madrid in Spain because Air Iberia was involved the main development site is in Sofia and T-police close to Nice in France because the French France was involved and The data center so the heart of the service We do provide is in Germany because the German Lufthansa was part of the game as well And this is actually where I work. So our main data center is located close to close to Munich Now we started like the same person travels skies as a classical a global distribution service provider computer reservations for travels by by air but since then we have evolved a lot and if you look at a picture you you can see that we Cover many aspects of traveling today. So it's not limited to traveling by airplane also by cars if it's a rental car booking hotels going by ship even up to the extent to Travel insurances and also getting closer than to to the airport. So for some Airports and airlines we do provide so-called departure control services Which actually makes sure that you can actually get your porting pass that you actually get the tax for for your luggage And for us actually this was a big game changer when we went into this business and we went even further Now providing as well scheduling of the runways for some pilot airports there as well So since we started in the late 80s, we have changed a lot We gained a lot of more more business and we have been quite successful You're not the first one in the market, but we are now more since quite some time We are leader in the market now as I mentioned our main data center is in central Europe Close to Munich and we do have operational sites across the world the main sites are on the east coast and the yes And in Australia to provide the kind of Follow the Sun principle to cover global operations But we also have people are ramping up in Bangalore in London somewhere else in in Germany and in other areas as well And now I get to the point which I by accident wanted to get on earlier Why VMA and Amadeus who coming from totally different businesses are actually teaming up here at the OpenStack Summit and yeah Well, the story is quite easy We have a relationship with VMA or VMA has relationship with us for for many many years So we started quite early with our compute virtualization With some GSX servers and we're having running Windows guests on them just to better utilize the capacity and Sustain as you can see on on the second part of the slide the world has changed a lot on our side So we're talking about 1000 plus your six hosts today 6000 plus guests running and both numbers are actually growing to the natural growth to the migrations from micro from mainframe to the distributed world and also also new business and because of this long Lasting and helpful relationship. We actually came to a commercial agreement Which is a kind of all you can eat a contract for us Which makes it much easy for us to as Amadeus to use VMware products So some of them are mentioned here for Software defined compute for software defined networking and for software defined storage and of course here for software defined Infrastructure so VMware integrated OpenStack, which is the distribution we do use for our Installations which we cover on later on So as I mentioned we are World leader and in the travel industry and we have been successful for many years. We are still success We are still successful. So the question is why should we change things never change a winning system? So why should we look at things like cloud type things infrastructure as a service now The truth is quite quite easy and I skip over this slide to save some time Our customers have changed as well Sometimes because they wanted to change but also sometimes because they needed to change because the customers of our customers have changed The entire T world has changed So if you really look in the crystal ball and if you look what's going on there what other companies are facing We still can continue to do the traditional business with our traditional customers But our new customers we want to attract there are new business lines We want to explore in order to do so we need to change things And then looking what other companies are doing facing the same Challenges are coming from the same background then looking at infrastructure as a service seem to be a quite a good thing for us And if you boil this down a little bit to what we need actually is so our workloads Which which traditional was served out of the data center in Central Europe? So inside our own walls behind our fences We need to make this work those workloads moveable and even more Cloneable or replicatable to a large large extent being movable within our premises So maybe from the main main data center to the disaster recovery site But also going outside our premises off-premise maybe to a hosted cloud or even to a public cloud And by doing this we didn't want to give up on the freedom Selecting actually the hardware we need to run this keep in mind We still have a large data center and we will have this for quite some time So we don't want to give up or we don't want to introduce Artificial lock-in to buy a particular hardware or to buy a particular software if you want to join the open stack as the open stack train Now Open stack. Yes does provide infrastructure as a service, but there's a big difference It's it is open source. So now looking at open source technology is not really new for Amadeus and actually even before Linux was born internally The applications we have developed the source code was available to everybody inside inside Amadeus We did have kind of this open source mindset already many many years ago And we are not early adopters But we are not light up that does either of Linux and other open source technologies like Apache or Chaper So we are quite used to how to interact with a particular community How does it work when you have our community versions and enterprise versions and the second aspect which we are looking for where? APIs and and standardized API's and such kind of things are also not new to us either because that Market we are in the industry. We are in we have to work with competition We have to work with so many different customers the standardized API's for something we were used to anyway So this is something which is not new to us, but actually it's a requirement what we want to have Now getting back to the question open stack why I would counter the question why not I think in these days There are not many options left if you want to have a cloud type or infrastructure as a service implementation On-site so in on-premise If you want to have open source a few years ago I think this question might be more Applicable, but in these days does not much left and the second point I want to make is that when you want to go this way or when you want to select a particular product, please make sure that you Stick only to the technical aspects of it and I will touch on on that one for most of the parts of my slides during the presentation there's a huge non-technical aspect or even at least a semi-technical aspect of choosing a solution as well and actually this one I have seen quite often will determine if your project if your journey is successful or not Another point which was important for us by selecting open stack as the technology for infrastructure as a service was that okay We have to think beyond if you have infrastructure as a service That's good That's cool for the infrastructure team because then it's easy for them But doesn't provide any business not good enough for us because we provide really applications or even services on top of that One so what we need is first of all easy way to integrate existing platforms existing applications as well But also a possibility maybe to explore new paths on the on the platform layer or on the software layer So kind of doing our two two things up with one set of software Like application packaging and here I think about Linux containers or going even one level up And of course integration and interfacing with existing environment because we cannot simply throw away the data center We cannot simply Build a new one some some bells and start from scratch there We still have to do a traditional business to serve our traditional customers And we have the new stuff on on top and it would be good if we can integrate this as much as possible So the overall journey I think there are four Phrases of it Before you start and this is what we what we did as well take a deep breath sit down and breathe Think through the situation. Where are you coming from? What's a current situation? But also where do you want to go and then based on that make sure that you get the right start to it because later on It's much more complicated to fix your your path or your travel when you have screwed up at the start And while you're traveling make sure that you stay focused that you all that you don't lose the momentum That you attract as much as possible To support you on on your journey and as I said already Don't lose the focus always keep in mind where do you want to go and what's the ultimate goal on that one? now for the moment we Remove two of the three main parts of infrastructure as a service or infrastructure So we forget about storage we forget about Network we just look at compute and the quotes here not so long time ago You would have heard inside Amadeus and this will give you an idea how complicated already Well, it was to introduce software defined compute a car of Virtualization in our data center. So this makes clear. Okay. There's a lot of work to be done Non-technical work to be done if you really want to be successful only on the STC part But also then on the software defined infrastructure part Part of selection for a specific workload You will see later in slides that since I've been trying to address But trying to schedule certain work builds on certain And I guess that's what you need to do enterprises to bridge the gap on which bridge from like the hardware Source to Kind of like in compromise way To say okay, you will go around on this hardware that has the characteristics that you Yeah Thanks So some of the things listed here we did on purpose because we know we had to do this some of them We did more or less by accident, but later on with hindsight it turned out this was something something good So we did actually sit down and looked around. Okay. Where are we coming from? What is the existing environment also? What is the existing mindset of the people going back to the quotes on on the previous slide? So became quite clear and I had just a chat with somebody else who's facing similar challenges Do why why are you doing this? This is important. It's not good enough This is just something for the infrastructure unit or for a particular department or division in your organization Actually, it goes to the extent that you have to show it will add business value It will help your company to be successful even more successful in the future and this will also help lay down to get the the power or the The support from the from the senior management or even from from the sea level management Which you will need especially for for the latest point which says secure the resources if you can and this is not easy especially if you are coming from a IT landscape, which is large and which is already running in production Try to fence out bits and pieces that you can start as small as possible and this helps a lot It may not look cool at the beginning if you start with just two or three notes Not with five hundred or five thousands, but it makes it much more easier to identify Okay, what are the dependencies who are the key people which I do need who are which are the processes which have to change Which have to be adapted but also work around blocking stones Which you which you may face and you can extend later on so you have separated your scalability Problems or challenges from the let's say mindset or non technical problems As mentioned already identified the key people and it's not only inside the infrastructure unit And so this is the same on our side They are the core, but you need also the people in the lion organization They are your champions to the spread the word to support you and as I said at the end you have to provide a service You have to provide a business and unless infrastructure is your business You need the layers on top as well or need support by those people on top as well Ideally you great win-win situations and this should be a no-pray now if you cannot easily come up with those with such situations I think there's something wrong anyway on that one. Yeah, I did List some quotes related to compute Virtualization and actually this is the angle where we are coming from so we have acquired a huge a compute virtualization environment We have some highly skilled people there We also got started educating the people preaching them now It's not bad and hypermises are secure as well and things like that So those this was the angle where we are coming from so we started from from that part of the Of the infrastructure not from the network part and not from the not from the storage part Yeah, and then it happened a customer came along a new customer with some let's say interesting Requirements and we could have sat down saying okay Let's see how we can build this the source provide this service with existing modules It was clear we had to change bits and pieces, but actually we took the decision. Okay, this is good This is exactly our pilot where we start from scratch where we build a new stack based on open stack Which will be our future stack and we started really green field with with that one We did select the right people which we needed there based on the experience based on the mindset and based on other soft skills And it was not that we said okay this particular team or this particular team So we didn't care about the line organization actually picked the people we need we picked the experts and we pulled them together Not really not to virtual teams to to real teams So there were rooms provided for them and they were sitting there and then working there And of course in order to do so you need resources and luckily on our side This was initiative which was fully supported by the by the highest management level you can get inside Amadeus So they did Take care that we actually get the rooms that they get it get the equipment that they get pulled out of their Current assignments that they actually can concentrate on that one And then this is something Amadeus specific in the past. We are used to the habit do it your own Nobody else can do it better than us But actually here we changed this also lots we teamed up with quite a few companies PM They're being one of them because we were quite new in the open stack business So it is really wise to look okay. Who has much more experience either as a provider or as a customer Or some something else. So we did team up also with other companies Where we had relationships also healthy relationships like like redhead or or mirantes So based on that there was quite a bit of known stuff We knew the VMware part and we knew okay This is the way we want to go but there was a lot of new stuff and a lot of unknown stuff as well and here My advice is don't be afraid if things change out of the sudden if you didn't plan it that way That's that's okay. That's that's part of the of the adventure So one of the new things we needed was new mindset And this goes a little bit back to the quotes you have seen on on the previous slide So we needed people really think differently who don't think okay I have done my work now for 20 years to accept it the same way and it was successful It is still successful. Why should I change it? You need people really with a different mindset Also different minds of regarding how we approach failures that I might do is because our business is so critical We are quite afraid of failure something fails that it can impact our customers However with open-stake or going this way you have to change this a little bit You have actually to embrace failures and and think okay your failures will always happen And you have to be confident by experience or by the skill set of the people that you can handle that one And then you can decide if you want to handle Availability on the infrastructure level on your platform level on the software level on both or mixture of all of them And also change the way how you work So at our side we are used to maintenance windows of a few minutes within three months or six months So our approach for change is always was to pull them up to stack them up have a big change And then we then we throw it in and we can manage that one But it doesn't work with with open-stake if you have a release cycle of six months for open-stake release You cannot pull up the changes for six months to do so and of course and for the sake of the password I meant I will mention deaf ops as I said earlier So the main development side was and still is in France and the data center where corporations is working is in Germany So even geographical we were quite quite separated But it doesn't work that way and to end responsibility means that also the developer understands if I do this particular change This could be the impact to the customer or just the impact to my colleague who actually has to run the code Or maybe I experience this much better if I run the code my run the code myself Being in the business for so many years and we still have mainframe to give you an insight here We have a lot of platforms so-called platform one or platform two applications If you can avoid don't try to migrate them to put them on on the new stack You may get it working, but it's much more challengeable is much more complicated and it doesn't make the things the things easier So we took the decision. Okay in order to do so we start from scratch We rewrite the application. We don't want to use the existing application and make it kind of working No, we really do some development work and and start from scratch Of course, it's not that black and white you cannot out of the sudden do the same business With a total new application so at some areas and we actually had to step back and to say okay This one we would like to have to have it this way But actually we cannot be fully stateless We have to cover state to a certain extent and we have to make some some commitments there Sometimes to cut corners sometimes to do workarounds to do things which you would not like to do But we have to do it in order to to progress But that's also something good with that one because you have noticed 2.5 or 2.8 Platform applications and they're actually a good bridge between the traditional business and the new business So we didn't go from zero to 101 step It needs a little bit more steps, but that's okay because it gives us also more time More time to learn and last but not least and at least here you need support from the upper management level so our old organization We had a lot of silos so there was application management and actually there was several application management units and even Inside infrastructure and inside the infrastructure era We had a storage department and there was a backup team and then there was a network division and things like that now or Our executive vice president he took the old organization actually turned it by by 90 degrees And he created what we know called so-called a service management units and they are responsible end-to-end Starting from the operating system up to the application providing the service that this one is running and providing the service to the Customer that he or she actually does expect and as part of the of the same re-organization a so-called Infrastructure unit was created. So this was already separated out to a certain extent and Their task is not only to provide infrastructure based on the hardware we have in-house But also to provide services may be coming from hosting providers or from public cloud So we already kind of created this this the same layout you have on open stacks You have an infrastructure part and all they provide is kind of API type things to the services on top Then this is a material specific if you are if you are in an environment Which is this for a while and was quite diverse and with a lot of walls and silos So we have a huge tools landscape and of course we have at least two Ways to approach a particular thing to automate it Which is okay, but it really doesn't make your your open-stack journey easier So we pull tools and automation together in one department one unit to make sure those things get harmonized and consolidated Because automation is something you do you do expect as part of open stack it makes the life much more complicated if you want to also Introduce automation while you introduce open stack and this is actually what what happened so as part of the of our open-stack journey We did some new things and and tried to change things which are done in particular way in the past To do it in a newer better smarter way And last but not least of course we did create particular dev ops teams One at the moment which actually will then run the environment for the for the customer so now some more details or facts which actually will then Never hand over to to Eve to provide even more technical details So at the moment we have we ever in the great open-stack version one which is ice-hole space So we are one of the customers who contribute to the large ice-hole space installation out there But we are looking forward to upgrade to 2.0 which will in kilo base the plan is to do this in in November So this year so ideally before we go into the Christmas freeze Everything is done. We are quite settled on the software defined compute side with these here six We have also NSX Used in in this particular environment six dot one But we are we are also looking forward to upgrade to six to two because it will provide some additional features helping for this project But also for interfacing project with is related to improve how we provide disasters recovery services These are we kind of got started, but we are not as advanced as we could be but that's okay Despite the fact we had the support from upper management We had we have only limited resources and we still have to serve also the traditional Business with the same people. So there was no Urgent needs to tackle this one straight from the beginning. So we kind of have some I think for the control for the control brain. I think we use visa on And and if it touch on that one later later on we will expand it and it can wait That's that's okay We have three installations one in Central Europe in our main data center and actually we have two other installations in the US East Coast and West Coast The one in Central Europe actually was the first one and lessons learned from that installation We actually have then used also to avoid the same mistakes when we built the US locations the Central Europe one is also used for us internally Testing things before we hand them over also getting used to things Spreading the world helping the people educate the people and also bringing on new projects Services which are running on different infrastructures the service implementations at the moment to pull them back to run on an open stack Yeah, and last but not least and parallel to those technical things. We are also working with teams like process management Security offers to get them on board especially to avoid statements like the hyper virus was not secure and If I stick to the security topic for us, it's also something told you in the past all the data were behind Defenses inside our walls, but now we have things outside our walls even outside the continent Which is something totally new for Amadeus we just acquire changes Change approaches different setups different technologies Which also have to work together with our existing technologies because also the environments in the U.S. They are part of our PCI DSS audit security audits because we do deal with with credit card data Training I cannot cannot stop mentioning that one train the people spread the word They need to need to have a foundation to start because only then they can Work with on and of course we try to expand physically expand the installation base We have especially in our main data center Yes, and with that one actually I'm now shutting down and tending over to Eve to give you some more technical insights Before I start What we do is we deploy multiple lower compute Notes are basically on VMS on management VMS and each of the normal compute is actually Interfacing to the v-sphere API or the v-center API is talking to a cluster object. Yeah So no one compute for us is not one single hypervisor is multiple hypervisor species is fixed up to 64 So it's a large pool of capacity we're interfacing with Glantz is also talking to the recent API to upload images onto a v-sphere data store Cinder creates VNB case on v-sphere data store to map them to instances So it's it's very like you you always have the center in between the open-state drivers and And the hydrographs and the same for for neutron at neutron side We have a blogging that speaks to the m-sx and builds those virtual networks that spend as over the networks between Hypervisors, so we have really a complete virtual networking technology Okay, so you heard about the requirements to have specific hardware for specific needs and that Rover requirement together with an h.a. requirement to have To have specific hardware pools within each rack So there are multiple racks and in each of those racks We have hypervisor pools for compute and we have hypervisors for database clusters Actually two different database clusters and also one for data analytics now since the unit is a rack You can't like put multiple of those big boxes in there Supposed to do database stuff and have a lot of local SSD capacity So it's it's it's a single hypervisor and with the logic that we have at the moment They are single No, we're compute notes and is that we we map them to a single availability zone Which is not ideal because what we are want to be able to do is to move workloads around And so for for the next round that will happen November will we use it that and likely will use Combination of host and DRS groups in these here There are DRS groups, which is basically just a collection of hypervisors within a cluster And we can address them on through either image metadata or through failures In that case it would be much metadata. So we take like a database instance We'll attack the image and say run only on those specific host groups and then we can we can Take that down to a single availability zone Similar cluster So those are like the challenges that we had to address to get to this point where we can select specific hardware for specific workers specific workload types and Along with that is is a storage consideration. So there was a tendency to use a lot a lot of local disk capacity instead of share storage and We decided then to go for shared storage more than local capacity because of multiple Issues and those issues are not v-sphere specific. You will encounter them also in a In a KDN based environment and the first one is spawning times So if you have shared storage, you have shared image cache and so your spawning times will just be much faster You only spawn once and not every time not with every hyperlibrator Also, there is a lot of network traffic that is caused by this spawning process if you don't have shared storage And now like in the v-sphere world, you want to be able to move workloads around DRS does a good job in Automatically rescheduling based on load and moving things from one hypervisor to the other and without shared storage It's simply not possible and also for administration purposes if you want to evacuate a hypervisor Shared storage is just helpful. I mean there is there is what VMware calls advanced v-motion You can also move data around between storage, but that's just again causing network traffic and it's lower So shared storage is a good idea Today, it's it's NFS based But likely it's going to be visa and so what you could compare to you are to set distributed storage system That is using the local disk capacity Okay networking we do is neutron and NSX so the deaf environments the production environments etc etc they all take the same Network definitions so no matter if you look in test that on production You'll find the same subjects the same routing tiers etc etc so it's a completely Virtualized networking biome and the network environment is completely configured Within the heat templates that also hold all the information of the application logic and the instances for the application On a physical perspective, it's a layer 3 network. It's an ECMP layer 3 fabric And since we're using overlays, we can build their two networks across those racks Without any problem, and there is an infrastructure utility rack that holds the gateways that go to the outside world the management connections that go back to the audience data center and the internet access and As you said we have a management cluster where the actual open-stack control pane runs So in the IO Open-stack is is based on VMs that run on this management cluster and that management cluster is also across racks and is using visa Okay, so couple of other tools that I used as I said all the application environments are defined in heat those heat templates are all held in an internal gate and All the images and packages are held in the architecture So with that combination, you basically have your whole definition of what the application the networks the whole environment looks like in a centralized version controlled Repository and and then you you pull from it and you deploy your application using using heat and The images also Now the application comes up or the environment comes up at the first thing that comes up is a puppet master And that puppet master is actually then like the focal point of configuration management that deploys the the application decoys things like the couch base cluster And the base application it also configures a local DNS server in inside of the environment And the other instances that come up Auto-registering that DNS records into also this public master What else yeah, there is a little interesting thing which is puppet runs then ansible to deploy a little shift and actually They try to go up the whole application runs in a passing line And that is pulled on open shift and open shift then uses containers To run a lot of the code. So most of the code is ran in containers and sometimes the database and other Monolithic functions are running on the end base that's basically it for the other tools and Yeah, we are at the end. So I take over again to to sum up Yeah, on high level what was what made it successful on our side or what helped us a lot is that we kept the size small and that we State focus that we did select the right application. Actually, we didn't pick any existing application We did create new applications to To bring them on the open stack at journey and selecting that the right right key people And if you look just at the infrastructure part, but we have built in our main data center So we are I think in Clovis. We are more than 900 people But only four or five people actually did work on on this particular project from infrastructure point of view So if you have the right people the subject matter experts the gurus the champions this this helps a lot and of course there was a Good selling of divine win win situations. So actually this particular project. There was Interest from both sides from global operations and from development to go that way. So there was an A natural joint interest to to go so to go that way and to create this to create this new kind of culture Like like DevOps mentioned before The point I wanted to make is with the second item. Yes, it is technology opens open stack is technology But it's really people first It's up to the people who make the thing successful or make it fail and people on all levels Technical managers technicians are people managers or even sea level sea level people If you're not starting your own business if you cannot really start full-time Greenfield It's okay to fence it out and ignore the prone field at the beginning But don't forget it at a certain point you have to look back. Okay, what you do with your existing environment Will it just die out will you migrate it will you run it in parallel? It's okay to leave it beside at the beginning But later on you have to you have to look at that one and this is one of the challenges We have in the midterm future once you have successfully Gain a lot of experience with this set up that how to expand and actually how to migrate existing services We have them and last but not least. Yeah, don't do it yourself a team up as mentioned So we have teamed up with the MMR to a lot of the infrastructure part But we have also teamed up for instance with the threat head We are co-developing the threat head on the open shift side for the orchestration Because this is the community way of doing this of influencing things to make sure that in open shift version 3 is the functionality We do need and we have teamed up with other companies as well for the for the same reason Yes, and with that one and looking at my my watch. I think we have still two and a half minutes left I think it's a lot for your for your attention Also for staying awake and staying in the room as much as possible I think we have time for two or three questions right now and if there are more questions We are still around the entire day today at the party and tomorrow and we also I think easy to find on the internet If you're Google for us Because of OpenStack So and it's kind of more concurrent if you to use But honestly like the complexity of the stack is so big That you lose time at so many places that that might not even be the biggest challenge Especially as we're using lean clones also on the B-Sphere side, which speeds up some things compared to other technologies Yes The practical challenges we had on different layers Okay, you got me Yeah, yes and no So did the person running the project? I didn't have a chance to get briefed before before I got here Actually, there has to be a schedule. I don't know if Arthur discussed it with you in the detail already Oh And they're actually copies of each other they are So Right and of course we start with the one in our main data center first because this is only for internal test purposes So we can we can break it here I think this goes mainly back to really the support from from DVM where people as I said only a few people Did work on that one inside inside Amadeus, but we got huge support out of the M Yeah, I have to say kind of expected because we are used to getting this kind of support But was really a lot of support of dedicated people like if for instance you really made his hand dirty To to make this happen. I think the cluster was running up and running in in June already Would you like something like that really within a couple of months? We made it running and we couldn't have done this on our own We wouldn't have had the skill set and and even if are all the Spreading the word to getting the bind of the people no chance Yeah It's a self written application are providing some services for for hotel customer This is Amadeus specific so the majority of our application or this our core services is provided by application We do develop on our own you you can't buy them You will find them somewhere else which sometimes makes it easier because we can easily change the code or adapt the application But you're also on your own on on that level. Yeah Oh Yeah, long-term of course absolutely we have to so especially in the e-commerce environment you're thinking about To actually that's a pilot just just started But we are there's so many things to to be considered We didn't only change the way how we're going to provide the infrastructure We also changed the way how to want to provide the platform the platform there as well And and the project will focus on on that one first and then the other one should should be easier long-term Yes, everybody should run on on on that layer midterm We started with something from the e-commerce area, which is different to the core business to certain extent But we just got started here I guess actually we are over time but nobody else claiming it. Yeah, okay. Thanks a lot