 Can you hear me? I'm Salvatore Dario Minone and I'm a software developer at Amadeus. Amadeus is we are going to speak a little bit more in the next slide and I live in France, but I am Italian and I Spent the five years of my life working on the cloud for Amadeus But I also all the experience I spend some time working for Amadeus business unit and I have a previous life in ETA Entronic design automation and finance here in Milan, which is also the place where I'd be Graduated in computer science engineer So for me being here is like being back at home. Thanks Diane for the invitation. Thanks Tanya for the invitation Thank you really I appreciate it. Okay in the next 20-30 minutes of Time together. I'm going to speak a little bit about Amadeus and what is Amadeus more or less and when everything about clouds started This is a community Meetup so we're going to speak more particularly about the Amadeus on the community Sure, we have a lot of things that can be said But I would prefer to focus on the community at least today in the future or at least on in the cloud in the web You can find other presentation that we gave in the past that you will do in the future We're going to speak about the community and we are going to speak also What we miss it in the slide if we could come back We are going to explain what have missed some mistakes that we made and Some things that okay probably me we missed something But at the end of the story For cultural reason for business reason for budget reason we couldn't do differently and Simply the model that openshift and Kubernetes in general expose Didn't fit that well with us, but it's the way it is And in the last slide when they keep on running I Will explain what we are working on and what is really What we are really focused on for the next month or a year Okay, what is Amadeus? I don't know if you know what we are we are a Pretty big big company. We are more or less 19,000 all over the world and We are an AT travel Actors one of the major IT travel actors I cannot say the major because I don't know very well the market of the brave competitors I but I know that we are one of the major and if you look at Wikipedia asking them what is the travel AT what is the business there are two or three things that you can learn for example There is a DCS the partial control system. It's the software that handle the The fact that the plane needs to go needs to leave the airport and land in another airport And there is also a lot of software that is handling all this. We do that. We have any DCS in inside the Amadeus There is also a reservation system like the computer eyes at the reservation system as well there are some brave competitors that do that we do also at well and there is Also the GDS the global distribution system that is the system that distributes the ticket between the travel agency the plane and so on We do that as well so we cover all all the all the picture in the travel IT and To do that we need a lot of stuff On the slide you can find the digit of all this you are present in more than 190 countries in Italy as well here in Milan and We had the at least in 2018 billion of passenger boarded and 64 million of total booking process it which is our pretty big values And all these come at the price the price is the complexity. We have more more than 100,000 software load per year, which is pretty big because in any other reason there are 360 days and a lot of software load per day Here the software load is it should be considered in a Enlarged way there's not only software load, but it has to change configuration network configuration all that stuff DB migration and so on but this is the the machine is pretty complex and One of the things that should be considered when you think at Amadeus is the fact that in the pre Gaffa Gaffam age The data center that we have been nerding in Germany was the biggest in Europe now There are other actors Google and so on that came to Europe but we for a lot of time we add and we have still have The data center a physical data center where people pull the cables and so on and this is also This is playing also some interest we have in the hybrid cloud today because we do both not only we are Trying to target in the public cloud but we are also doing the cloud on premises for our customers and This is playing up a special position Okay, Amadeus to be honest is a 30 years old company. So I'm not going to do the story of all this and Just trying to keep it short But everything for us at least in for the cloud started in 2014 Which is more or less they hear where Kubernetes and OpenShift started together at least for Kubernetes It was the summer of 2014 and we signed a project with a strategic project for Intercontinental hotel group in US When they ask it has to supply the new infrastructure for for their reservation system and This Will has been done in the cloud and we were looking for a partner for for the cloud We spoke with multiple people with a lot of Brave corporates but finally we met Red Hat and And After a travel that I did with other colleague to Riley in US it all ended up with a win a signature and At the end of 2014 we signed under Red Hat a part of the partnership engineering with Red Hat company and This fact changed a lot of things Inside Amadeus and some things also for Red Hat side Personally, I did my first forecast to Kubernetes in 2014 it was the December Just before Christmas, and I always remember that because it's been an emotion and In summer of 2015 now, I'm sure about the dates to be honest probably was July Kubernetes and June OpenShift or something like that We Start to the we entered in the in the real release of OpenShift 3.0 and One year later as we managed to win under the Red Hat Cloud innovator price, which is always something that is still Considered an added value for us in Amadeus people Keep to recall me this fact And for the future I'm going to speak about the for the next from the in the neck in the next slide Okay, let's start to speak about what the Amadeus and the community Amadeus is an extremely strong The culture the corporate culture in Amadeus is extremely strong a lot of people in Amadeus that are start to be Hold now all season I would say Or always work it only for Amadeus are plenty of people with 25 years of experience That's they always in Amadeus and this is create a very very strong culture in for a corporate so But This create also an isolation the fact that When I moved to Amadeus, I I had the other experience as well at the beginning. I was understanding nothing because in the industry For example a demon is something that is Specific software but a demon inside Amadeus is not the unix demon that you most of you that I net D something like that is something completely different and this is and this is special and The first things that the community Gave to Amadeus has been the unification with the industry and now today in Amadeus It's not completely finished but is ongoing When people speak about the service now it start to be an open shift to our Kubernetes service and this is a really an added value believe me because for example the culture unification in tapas across regions before open shift Moving it moving from a team to another team. It was like changing a job inside Amadeus and this is Something that okay, we can cope with but the reality is art So people spend a lot of time trying to understand just the jargon between teams division and so on and this has been a real added value for for us In the Amadeus also Since the culture is so strong before open shift we never consider it something like Canary testing something like probably not not canary testing but replication replication the replication controller rollback Those kind of features that camps automatically with open shift adopting or funding I don't know. I know you're You're skilled, but these are you basic use case in Kubernetes and open shift and now people In the Amadeus always speaks about adopting the pod which is crazy for all these years all these years in in the Distributed computing never people talk about this and now it happens and this is thank you To the adoption of open shift And more than that now we start to rethink about internal practices now it happens Often that people have asking for probe liveness probe readiness probe and we spent a lot of time Just to work around this problem the fact that we didn't have a probe Has been a problem has been a problem for a lot of time and now every application starts to Do you have a probe if you not have a probe you cannot load and so on this is another added value and we fundamentally Managed to rethink our internal best practices again This was comes from the fact that the company is a well-established Practices and now is open Thanks to open shift the communities adoption Somehow we also gave something to the community one of my colleague Alped the one redacted engineer Paul Paul Mori, which is who is a friend To to design the secrets the actual the kubernetes the kubernetes secrets, which are Fundamentally the open shift kubernetes secrets Since we're there since the beginning we also did some backfixing adding some tests and so on just simple grant job and workforce and We push it for example to have observability Observability the initial use case was a bing the label of the pod Exposed in a in the logs say okay. I have this pod that is going to run on Under these labels for example We just need a way to connect the pod information the pod metadata Inside the application. That's the reason for For downward API for example, I personally helped by always Paul Mori Coded downward API or at least I contributed significantly and since we're there since the beginning again We spent some time on deploying we push it the first version of live your chorus now has been deprecated with nice and we had several contributions to a shift ansible for You or the open shift ansible github the But the contribution that we have That we are pretty proud of is To have pushed the requirement for CRDs the original use case was having like a dagger workflow for Incuburnities natively incuburnities for several reasons which are totally understandable We couldn't have that and at the time a brand numbers now that Microsoft pushed the TPR third-party resources If people in the room and Pretty young to cover it is probably you will never heard about their party resources. That's fine But before customer resources, there were third-party resources, but third-party resources. We are so bagged that fundamentally readout folks The guy at the moment is no more at readout, but life is like that Jordan Liggett explained to me Kindly that there were nowhere to there were no way to build the really a Workflow a DAG a graph of job using TPR. So this push that the certain the customer resources use cases which has been implemented by Another Red Haters David is that's my great for a lot of for his contribution that has been huge and I know you know Custom resources simply changed completely the The scenario of Kubernetes and open shift at the point that today the idea is that everything is going to be a CRD is and this created the operators the operators and the creating a lot of stuff and We are behind all this and we ask it to one you'll be one of the open shift leader to Clayton Coleman and which is Who helped a lot in this process and We are very proud of this and that's all this is our small contribution, but It's just a way to to state what we did in the last five years Okay The reason that For CRD is that at the beginning we were trying to create a workflow control with the workflow operator And it's called controller because it comes from an area from a from an age Where the word operator did not exist. It was just a controller because Okay, now everyone is speaking about operators, but it's just an extension of Kubernetes controller It's just is an extension of her ship controller a Kubernetes controller And at the time that and the and now he's published an open source on the github on Amadeus github And we then we start to implement all this other stuff like a radio operator like cannery radio operator Obviously it handles a class already slasher using the CRD's Cannery and all the development the the load the load the through cannery and has been presented at the cube gun In Barcelona this year And we have all the operator as well that which are not open sourced For obvious reason because it just handle the internal technology We have some for example for our ESB the software bus Is more an ingress controller than an operator bet anyway We are doing a lot of stuff on on using CRD's and operators Okay, this is the therapy slide if we have What we have changed if we have known before This fact come from the fact that we are there at the beginning and the beginning was tough Today is still tough at the beginning was even more tough the provisioning infrastructure for VMs we had an internal knowledge of puppets and We started the provisioning infrastructure using puppets then using Terraform and We are still struggling today for with this We have some legacy again comes from the fact that we have a long history in Kubernetes and OpenShift and Is there I don't know if we had to do something different But at least we had to share our use cases explain why why we had to do to do some strange things and Instead we did not do that It has been an error not the implementation as well not the implementation really But just the fact that we had to speak more to the community and explain the problem and so on and for that OpenShift 4 for us is bringing the hope to forget all this or at least to Simply slowly the commission the actual cluster that we have with this legacy technology and we are still waiting for for that another things that I'm sorry to say that I know that Updating through Ansible can be a Tough experience. We have a lot of discussion with Your colleague in US with that Again, we have to share more on this. We should share probably more on this today and OpenShift 4 always always keep to bring the hope for a better future I would say our appropriate manifest files is The perception in the community at least at the beginning was that the manifest file The blueprint or whatever you want to call the YAML the JSON was hard to understand So we have the same reaction and we decided to create our own manifest files Which has been probably one of the worst error that we did and we are still struggling with that We are just the commissioning and the point is that Helm was already there Helm I met the first time I met Helm. He was at the first cube gun It was not like the matter Helm that we have today with the tiller the stuff, but But we decided to to bet on ourselves and We have this legacy today. OpenShift template was there as well but we took an Alternative part we and the final outcome is that it was wrong Again sharing our use cases Explaining more at least I don't know if it could change something, but at least At least may mitigate the problem somehow following the This the previous slide We we've we started the production with OpenShift 3.0 at the time Prometheus was not there a Ocular for example for logging was not where as well and we start to implement in-house technology and we are just the commissioning at the moment we Struggling on struggling we we have some bandwidth that is taken to do that And a cultural problem that we had is the fact that internally in Amadeus We we need to do that in production everything must be fullback Everything everything Insert a line in a DB. You have you need a way to come back to the previous one It's like a rolling update, but not that rolling. Let's say a transactional way This this is not so loud as a practice in general But is a well-established and the culture inside Amadeus is really strong about of this for example in it for the ATCD back up until Nine months ago I don't know if the bug has been fixed the gap has been fixed I know if it's a bag there's well nowhere if you move to another version of ATCD to come back to the previous one Which is considered not so well acceptable for us, but we cope with it This is a cultural problem that we have some friction that we have And Another point that I would like to state is the fact that working with open source for a corporate that is like us Is something that we have to learn is something that they need time Is interesting for developers But he's even more interesting for pms that set the roadmaps and so on with the budget and so on it can be Tough sometimes can be the developer in that case. It was me or their colleagues Can be exposed to some friction because okay, how can it can be that you did this and he's not still Released it's the way it is. It's the open source it's people should start to enter in this mentality and is like that and The final point again open sheets for four brings hopes Is the cloud provider APIs? For a lot of time people keep to think keep to expose this metaphor saying, okay, the cloud provider is the new platform hardware platform And Kubernetes or open-shift is the new operating system Okay, this is a way to see the things But does not work exactly like that in the sense that the cloud provider In previous presentation are all different and the concept most of the time are different. So Just wrapping all these behind an API most of the time is not enough and We learned this in the other way I'm not saying that there's no never work in this metaphor I'm just saying that it should be taken with Carefully with with some attention Okay, and again so late In the next future In my team in my group we are Seriously looking at the Federation. We are seriously looking at a cluster registry which unlikely has been fold some disgrace or today but We have plenty of cluster all over the world and we start to have We start to need a yellow pages for the cluster. We were looking at the Federation Six and I know that there are cluster registry probably now we are we are deploying a cluster registry internally I spoke with all the people in the cloud in the in the multi cluster in the multi cluster C Down this difficulty inside but anyway, this is the way This is the things we are working on and we are Start to working we started already to work on open shift for on AWS and on GC as well We have two teams that work on this We have an internal pilot project pretty big I cannot say a lot of things about it, but it's That is going to be done on open shift for any AWS Since we were used to build our cluster inside at home We are seriously looking at open shift for on bare metal is going to be probably starting in one month or two months something like that by for 2020 You will learn about you will learn more about Amadeus and open shift for on bare metal. Sure and Some of the cloud that we are internally And luckily not only on the red art open stack bolts on other other red art other open stack flavors. We have some Project on open shift for on open stacks as well We are so it looks the operator lifecycle manager seriously, I have two developers that working on this and Ensure the matches to replace or at least to extend our internal technology Um, we are waiting also the red art roadmap. That's why I was taking the picture But anyway, we have a weekly Contact point with people in us So we are we are well Most of the things I don't care. I cannot say all and this is just my final slide just to tell you Thanks you. Thank you. Really. I appreciate all this. That is my image. Amadeus is hiring Like every cloud company We need we need the ends digits and some Finger sorry and so on Don't hesitate if you are Looking for for new opportunities a new challenge. There are some that are really challenging and On the right column There are the people I would like to thanks personally that is not a corporate stuff That is my because are all the people that tell me in this adventure in the last five years always Obviously Clayton Paul Dan Derek David ease and and so on really Thank you and thanks again Diane and Tanya to invite me here. All right. Well, thank you very much