 Hello everybody pleasure to be here. My name is Gabriel. I'm an SSP and designer at Red Hat just So we can know who here is a redheader can raise your hand, please Okay Yeah, a lot of customers. I think thanks for joining us who here is from Latin America. I know some Brazilian people Okay, okay. Okay. Yeah Thank you. It's a pleasure to be here. We are going to talk a little bit more about real-time payments and OpenShift Here with us today. It's Henrique from Banco Original, a great institution from Brazil And yeah, I think we can start with the agenda, right Henrique? Yeah, so hi everyone like Gabriel said, I'm Henrique. So today we're gonna talk about We're gonna talk about some real-time payment processing using Red Hat OpenShift at Banco Original So we'll cover a lot of overview today for all the Latin American and all the Americans too of the Brazilian bank scenario and a lot about how Our challenges were and how we solve them. So my name is Henrique. I'm Brazilian and Yes, I love soccer if anyone's wondering. It's a stereotype, but it's a pretty reliable one I worked mainly as a software engineer, but also in architecture More recently at DevOps as a manager and now I'm a tech manager Yes, I do love soccer too. Actually, I found out yesterday that He could play soccer every week and he didn't invited me like until now Yeah, but he lives really far away from me. So but I'm still waiting for it. Yeah, right Yeah, we have to As I said, I'm a solution specialist in design at Red Hat Work with a lot of FFSI companies and also previously I was a an architect at Open Innovation Labs So I love all the things about digital transformation and real cases And Hickey, I think to start we should tell all the people about Banco Original, right and The journey of being like the first digital bank in Brazil, right? What can you tell us about it? Yeah, as we are all in an interactive session Just to know who here is familiar with the digital banking or digital banks with your hand, please So we have some some are not so just an overview digital bank is a real bank with no agencies or branches Everything is done online. You open your account online and with an app and but it's a real bank So we Banco Original are the first digital bank in Brazil So we started with a lot of challenges even loss Laws have to be changed because you had to go to an agency or a branch to open your account the manager They needed to see you so we started all digital with facial recognition Remember it this is 2016. So it wasn't a thing Here I imagine in Brazil and so today we are we have a large set of products in 2019 We we change our strategies for for three years. We operated with external partners now everything since 2019 We internalize all the development department and all new products were developed We we absorb all kind of know-how and today we have 10 million opening accounts And as I said a large set of products So yes as Eric said and I think that we can frame it as a mobile technology rising during 2016-2015 or since the since the release of the iPhone right and the Banco original was the first Digital answer to these new kind of necessity that the society in Brazil Had to financial services because it was the first bank that didn't have any branch like to so the customers to to receive customers and They actually had like a pretty complete and solid application, right to establish communication with their customers So and we are talking about 2016. So this is prior to the to the pandemic and They when the pandemic rise started you guys were pretty ready for the situation that also followed some new services that were pushed by the central bank of Brazil in 2019 one of them being the open banking Architecture and the the other one called picks that is the real-time payments in Brazil And they so so just an overview picks as Gabriel said is the real-time payment system It basically allows anyone to transfer money between accounts within 10 seconds so the idea of our central bank was like to include as many people as possible in the banking system and It brought a lot of new challenges but a lot of new opportunities to the banking market and It has a sum of requirement for participants bank original is a participant It has to be fast. So as I said transactions must be completed within 10 seconds It has to be available. So it's a 24-7 system and Is that it has to be secure as I said everyone needs needed to use so it's country-wide So it has to be secure with some some rules and and with help the help of technology Also, it has to be integrated. So Every bank is a participant. We have a lot of non banks but financial institutions or even fintechs that Can be part of it. So it had to be integrated It had to be versatile it it opened a lot of opportunities to the market and it launches on a lot of new features Every month or every three every quarter So every bank that is a participant has to be versatile to adapt to those features and It has to be cheap because it's free for users. So every individual that has a bank account can use banks for free and just How do we use it? So if I like I need to pay a coffee for Gabriel So I really type his key and the the amount that I owe him and my my bank app communicates with my saying and he receives the money within 10 seconds a Key is mainly an identifier to his account. It's pretty much either his Social security number his phone number or his email So it's pretty easy. I don't need his bank account number or anything like that In the background is a little more complex Here just to illustrate that not only banks are participants. So we have Direct participants and indirect participants that communicates with the addressing database that is the SPI a Direct participant is mainly a bank a regular regulated bank But quickly now is a direct participant and indirect participants are any other company or tech company that wants to be part and use picks It can use this through the direct participant communication with our central bank It's pretty pretty big today. So I Think I can say almost a hundred percent of people in Brazil use it So we have today. This is from last month. So it's I think it's I'm sure that these numbers are Bigger yeah bigger right now. So we have 500 million keys registered 146 million users Brazil has I think 220 million people today. So it's mainly everyone that that uses a bank and we had a average 80 million in transactions and transactions Realizing transactions Daily average game in 2023. So just to summarize it like none of the Brazilians here They would have any cash Realizing they all because nobody's using cash anymore in Brazil. Am I right or right? Right? Yeah so What is picks is doing is kind of like changing the way that we do payments through our day-to-day activities So and also like with credit card and also open banking APIs And also these changing the society also brought a lot of challenges to the companies to answer with that as in As in Hickey said like You don't have only Direct participants, but you also have indirect participants that can use the APIs from the direct part Participants to participate in the ecosystem So you have a lot of companies that are included in the global ecosystem of the central bank of Brazil And also that raises like a lot of the numbers of about the transactions and also the amount And also availability was a really different thing because you had to be 24-7 right and then in 10 seconds every transaction What can you tell us a little bit more about like how Banco original? Technologically face those challenges right because it wasn't like from the day to the night that you solved all the problem Yeah, we have kind of a journey, but before that. Oh everyone that's standing up There are a lot of places here. You're free to come here. Okay, you're comfortable So like I said, we have a journey we came we started with Legacies we had legacy systems that wasn't developed by us. We have a ITH department actually only validating stuff and products and Today we have a hundred percent of managed microservices at a distributed environment and everything Everything is managed with OpenShift So how do we start from here to there? It wasn't a short period of time so we had I think three years, I think three years of a long journey to To get from here to there and we have some steps in the middle. I think Everyone of these steps were really important for maturity and to to get where we are today and really to To implement PICS because it was a big challenge. So after our legacy systems We started internalizing our our our development department to have our know-how and We started developing our standalone applications and new products but With all the evolution in the ecosystem in in technology We saw a great opportunity in using this a distributed architecture developing microservices. So we have a lot of work of Migrating a lot of our standalone applications to microservices But microservices weren't manageable at all So we have a lot of VMs with a lot of Java processes there Each one in one port And a lot of batch applications. So we needed to manage we needed To limit memory to limit use of resources. So we started using Docker with all of that in containers But even that we didn't have all that managed. So we have to like have all kind of alerts for containers We have a lot of manual work That's when we started we started Searching in the market for solutions. So we can have all that managed and we started using OpenShift To be to get all those containers managed Oh, it's actually what it was actually and introduced a new culture culture because we use only Docker containers now We have a managed service like based on Kubernetes. How do we spread that word? Inside the company and he can actually like if you can talk just a little bit because when we met It was actually during a DevOps event, right that I was doing like a webinar, right? kind of like a talk because it was live right and also a Since that day I saw that you guys from the architecture team was trying to bring everybody together Like not only for the operational side, but also from development And you actually had like an idea to create like a TV series of webinars and do kind of seasons, right? So it's not only about like having the best to but also trying to Make everybody like to feel included and also to develop the the right culture to to so these Systems that can foster in the right way. Yeah, so with all those new challenges that picks brought us because I didn't mention But we have a really short period of time to develop everything of picks. They announced. I think in late 2019 that we we had to have that in production in November 2020 so we we had to run not only us but every bank and when we implemented and started using OpenShift mainly the infrastructure team Take care of that the developers and the architecture team and the other security team weren't involved I think at all but We saw the opportunity and the the need to have everyone involved because everyone needed to know how important that is and how OpenShift and the other tools that I'll mention Could help developers in their day-by-day are the architecture team defining patterns and the security team Guarantee all the security in our environment So that's when we started involving everyone to spread that idea of a new culture for everyone to develop and know all the benefits that they could have on their on their job Yeah, yeah, so Talking about picks that's a big picture of our architecture overview So we have a hybrid environment So we have a lot of things on on-premises and with picks we started using cloud mainly AWS We have OpenShift Installed on our on-premises environment and it manages all All this is an overview only for picks so on the on-premises side we have over 70 components connecting with the cloud side that has over a hundred components because like I said about opportunities we started Selling our services Picks as a service to other institutions indirect participants and even to other banks that Didn't quite had the time to develop at all. Like I said, it was a very short period of time to develop picks and OpenShift controls mainly everything on on our on-premises side we Don't have OpenShift controlling helping us manage the cloud side yet But it we have in a in a we doing a proof of concept We're doing a lot of tests to have that up and running pretty shortly and We faced when we started using OpenShift some challenges So we had a lot a thousand of micro services. We have to worry about deployment. How will we deploy? Using a pattern how we would deploy that safe and How would that environment be secure? So we started worrying about security. How will we we are banks? So everyone everything has to be secure I don't know if anyone may maybe someone here works at a bank. It has to be secure. We have some regulations to follow So it's it's a a mainly thing that we have to worry about and reliability So it's a lot of products. So they have to be reliable I can't have like if a client needs to withdraw some money the services It's not working. So it has to everything has to be reliable We had hundreds of developers as I was in the architecture team We have to worry about How do we define patterns to? Developers worry only about developing products. So we have to worry about abstraction They don't have to worry about infrastructure or security. Of course security in developing security in cold but not security of the environment itself Their services needed to be easy to deploy that they had to have easy to deploy the services To not to worry about all the devos pipeline. They have only to worry about developing and Like I said, they had to have reliability to develop to know that what they're coding today Will be in production and working as they and the team expected And lots of new products being released not only peaks But a lot of new products in the banking market was being released So you guys know how the product teams are they they needed they need things they Passes the requirements and we need to deliver. So We have because they they worry about time to market And we have to worry about to develop their products to be secure and to be available So all the products need to be a need to be available the withdrawal example is a very good example. So The withdrawal had to be available if someone needed to to withdraw 10 10 he eyes Like at 2 a.m. So yeah, and I know what you guys are thinking, right? You look at this and you say containers because that's not possible without containers, right? And if you think about containers, you're thinking about Kubernetes, right? And so you got you guys actually had a discussion about Kubernetes, right? Was the first step to think about that. Yeah, so we were using containers here and We had to give the next step and I think is the most common path of most tech companies So let's start using Kubernetes to manage all that It can manage containers. You can use pause everything is as a resource so but as a company and Mainly as a bank. We had some concerns how the configuration of that environment would be How how much time we needed to to get the configuration? Up and running the way we needed how would we have security of our applications and even image security? So I think everyone here knows about Like the lock for j problem. So that was a concern for us The knowledge and the learning curve for our teams developing team developers team the even our team the architecture team the infrastructure team because we were on premises environment everyone managing VMs and stuff and this the Yeah, the security team the developers team at the infrastructure team and How about support so we're a bank. We have to be available and reliable We're regulated so we needed support if something happens happen and Then you guys found out that we have something called open shift. Yeah, right open shift plus And it answered most of our questions. So we saw that we have easy configuration And so with just a few steps effort had that in our environment. We we could configure Everything as we needed as a bank and our regulated institution We had security image security using ACS because of our our applications I'll get more of that later We have a we had a no low knowledge curve So with that we could get our developers worrying only about developing our security team Worrying only about security our and us architecture team worry about the architecture of all of it and our Infrastructure team not only worry about VMs and all the manual work and stuff, but now managing a Pretty distributed environment pretty easily and mainly we had Red Hat support So that was I think the main point for us if something happened We had a support on all of those topics So like he said he was given a workshop and a lot of others that occurred that we had your help to Spread all the knowledge about open shift and Kubernetes and distributed architectures With easy configuration you guys always helped it up a lot What what you said that really strikes me is that you said let Developers should focus on the value strings, right? So if we talk about team topology is like you have the platform Then you have the string-aligned teams that they've lever value direct to the customer So you try to separate like complexity from those things because they they already have complexity Dealing with the needs of the users So you had like a platform as a service that you could use So you could have like something more templated already that for them to establish and created the applications on the top of it, right? Yeah, and even they didn't have to worry about the pipelines and it was very easy for the DevOps team to The team that has to worry about the pipelines We we don't use yet the open shift pipelines, but we use tecton with Argo CD Interacting with open shift and it made it pretty easily for us to deploy and use deploy strategy Like canary release or a blue-green release or even a simple rollout or stop start So yeah was pretty easy for for the DevOps team to and Some stats. So how are we today only on the big scenario? So we have 10 k pod about 10 k pods up and running We process 1 million picks transactions a day. We have a 99.9 uptime and we can process Until now I'm pretty sure that we can do a lot more of that 130 transactions per second picks. So it's That was a peak but our environment was were pretty Healthy and nothing alarm then at race didn't blood, you know, so didn't bled and Yeah, actually like what strikes me that you also said that you talked like about a lot of developers You guys actually have you have no the number. Yeah, hundreds, right? I think like About 400 100 so a lot of things and you are the architect guy So you need to establish patterns through the company, right? Because you you're thinking about the long-term journey So what can you tell us like about how this platform the service helped you to also? Spread the knowledge and also to make it easier to the other teams to use it. Yeah So like you said, we have to worry about a lot of people using our products Our pipelines our panners developing stuff. So it has to be simple yeah, because new developers come in coming every day we teams grow up teams split in two or more teams so and applications and products keep growing and pad defining and implementing panners could be a headache if you have a Messed up environment because you have to worry about. Oh, this is this will work on this scenario, but you know the picks guys They have a different environment. So it wouldn't work for them the Like transaction guys had a different environment. They don't use like Kubernetes. They use I Don't know no JS wait Something else and so it made us it made pretty easily for us to define all those part patterns and Gave us confidence that it will work on Every team. Yeah, so here we are also talking about and I know that you had a lot of of work because you lead it some some of the initiatives around CI CD and you guys are used like Source stream as you guys use a tecton as you said Argo CD Also, like a lot of observability, right and monitoring resources because as you said like it needs to be 24 7 So you guys use it a lot of like the probes and health checks that are already present in the platform. Yeah, so We had the dinosaurs, but dinosaurs only help us help us a lot on but with alerts and stuff but We define it up patterns actually for that probes and health checks So we have a reliable environment. Like I said, it was a requirement for pigs but actually a requirement for a bank and a requirement for an IT department a development department everything has to be reliable the product guys need needs to know that the product They thought about they discover they they had to work a lot of discovery on interviews And everything has has to work up on What on the way they they thought of yeah, and uh just and you came up to the next one So also, we know that pigs is still going on. We still have like some new products that are being released Do you think that? Having that approach like also not only like to to implement it the two but also to help the teams to understand it Helped you guys to reduce the time to market to release new features Yeah, of course because like now developers can only worry about developing and Security only about secure security and infrastructure about infrastructure. So you keep them safe off like yeah Yeah, they didn't have to worry about anything else But like I said, we Well help does a lot was not not only OpenShift. We use Those three platforms that help us a lot again, we were we are a bank and ACS helped us a lot with images can in and Degreting the pipeline right like to my point integration and every every feature mainly Right has OpenShift platform I think there's no need to say anymore about that because yeah, it's very clear and we We started we starting using ACM more because on our hybrid environment It's becoming a thing to to worry about and We have multi cluster management on the on-premises side We have multi we have some clusters on the cloud side We have a lot of clusters and everything needed to be managed because on the cloud side We we can we have a lot of products that A lot of partners and other companies other banks other financial institutions use a lot of APIs So we have to have that managed to and it's going pretty well. We have some good results We're at the beginning. We're making a lot of proof of concepts now, but we're having pretty good results Also one thing that I remember is a it needs to be scalable right because like the So we we had in Brazil like the five first days in the month We used to be like a little bit rushed for for banks because but now it picks like you can have It can happen at any time right so you spoke a little bit about probes You guys also use the the horizontal horizontal pod outscaler to the to provide that scalability that those 70 applications that you said needed right yeah, so those things helped us a lot And new products are being released so pigs open a lot of new opportunities new products using pigs on the Backstage are being released and we have a lot of adoption of our customers. So we HPA and Probe and help say backstage On the the background Yeah, but I spoke a little bit about Yeah, we also use backstage But I'm I'm in the background and and with that pigs keeps evolving So that's those are some features that are helping us to To keep in keep keep the pace with basing in pigs So those are some highlights I got from the pigs agenda of our central bank So they're implementing international pigs. So at some point we will we will Will we will we can transfer the eyes to dollars and any other country they're starting working with direct debit and Guarantee assets with pigs and the payment API. I think The payment PI is the really important one everything is part of but it's a the main one because it could The idea of that is to distribute the pig pigs and payments for anyone Today I can you I can't pay all my bills using pigs I can pay anyone or anything any retail, but the payment API will Bring a lot of new opportunities and and a lot of new features for the bank environment for more people for more companies and It'll be a lot of there will be a lot of new things. So and more So you we spoke a lot about like delivering code to production faster scalability scalability With all these new products that need to be released in the next few years What is the biggest challenge that you want to to to succeed now? And what you see in the years for come to to open shift and bank original? I guess it's keep growing and had our have our Technologically speaking had our cloud environment all managed because we have products there that other partners use and more customers use We cannot we can achieve much more customers and a lot of new products. We could build and Have a revenue for the company because you know, that's what companies are for but I think having all that manage and Everything interacting and everything working is our challenges for the next multi-managed way. Yeah multi-managed So that's it. We have some time for questions Yeah, we have some time for questions. Yes, sir You said you were running multiple open shift clusters on prim Sorry, sorry I'll slow down. You said you were running multiple open shift clusters on prim. Yeah, how many are you running on prim? You know me asking we have Three today. So because we have multiple data centers So we have one when initiate data center Yeah, and it's that 10,000 pods total across all three clusters Oh, yeah, but only four picks. So we have more but with to all of the other products. Cool. Thank you You Cody then Hey, and you mentioned your monitoring strategy that you use the kind of the built-in features that came out A box of open shift in terms of external monitoring or a probe or let's say a black box Do you guys use anything? External to the cluster itself to monitor your applications. We use we have some metrics Collective systems that run within like sidecars and in applications as libraries to that collect metrics and send to an external metric Collector that we have and we can produce a lot of new monitoring on that So not not only inside open shift, but we collect the metrics of the applications and set it externally So we can build a lot of a lot of new dashboards and when it are here kind of a mix between Dynatrace Prometheus Grafana. Yeah, right? Yeah So so don't address is the external sauce to collect the the metrics that say so if the cluster was to fall down that was your What about the cluster itself I would say but about Specific about the cluster would be Prometheus and the specific about pigs applications would be Dynatrace and Prometheus together, right? Dynatrace and those systems and libraries we we built to collect all those metrics and Because develop we built though those libraries because to help Developers collect the metrics. They think they needed to they can create right metrics like product metrics or system metrics to collect that and everything is is managed externally a Lot of questions. Can I go after? Yeah Hi, and I have a question about Backups and when you are using what bounce closer manager, how do you manage backups for containers and on the different clusters Do you have any solution or it's not necessary in your case you do you mean on backups of our Like the diversion of the microservices that's that's applied So we use that based on on image on our image raster's registry I think today we are using ECR a double SCR So every deploy that generates that image is Generates a new version at ECR So if we lose anything on the cluster side, we can recover just by deploying the image again Just one are you talking about like data or the container itself? Both okay, because for data would say there are Another platforms right in the with data storage and everything but about containers They do have like quay and also ECR that have the images as well. Yes You yes, and I go back to you I Would like to know like how is the performance, you know, when you are running on the bare metal now, we are like, you know we we have an SLO of Max 500 milliseconds for all applications and for PICS applications We have a lot that on 200 90 seconds so that that's valid off on the young premises and on the cloud side So on our pipeline, we have a performance in test step that tests all of that and And validates all all our SLOs. So if microservices Doesn't achieve the SLO it isn't deployed. So it I can to answer a question. I can say We And we started using performance testing only with open ship So I don't have the data before that because it wasn't a thing that we worry about because we had Fewer products and fewer users, but with PICS are We have a lot of users and a lot of products we started worrying about so today is for pigs 200 Milliseconds, and I don't have the previous data for you. Thank you Yeah, regardless the environment where the application is running It needs to respond with the same time because if an operation past 10 seconds the central bank of Brazil is going to cancel the operation More questions actually have two questions now. That's okay. Yeah. Yeah, okay So the first one is do you use any log aggregation software like a last-search or low key? Yeah, we we started using the Elk stack, okay, but that's what we use on in our environment as well Yeah, but then we migrated to Splunk using Splunk, but we have a log library that we developed ourselves to generate logs In using a pattern so it everything will be easier with Splunk or any other two so Everything's about our logging library Regardless of the login tool we use or the environment so we migrated from the Elk to Splunk with no painless and no Other issue at all. So our login library is the main thing. Okay, cool And then the second question is I'm assuming your application is the imprecise storage What a kind of provider to use to provide those volumes to OpenShift do you like Ceph or Portworx or something like that? for On the on-premises environment, we use Oracle so Oracle database Okay It doesn't need any like file system file system persistence in the pods themselves. Just what's in the database? You guys. Yeah, you actually have like Non-stakeless applications. Yes, they do they do use Ceph and On the on-premise side is that using ODF? ODF exactly. Yeah, the data foundation. Thank you. Thank you. Yeah, no problem More questions. Yes, Glenn Yeah, hi. I was thinking you using advanced cluster management Could you maybe elaborate a bit on what you are using it for like policy management get-ups full setup of clusters? We we are at the beginning of it We're doing a lot of proof-of-concepts and tests today but mainly to We are Like testing how we could have all our clusters Manage the same the same way that our on-premises cluster are configured and managed So we were getting a lot of ready-held help for that So we are on early stages of that so we have we will have that soon soon enough. Yeah, I would say like So we are going to the cloud right now and Trying to keep everything balanced and also better standardized So ACM we are we are on early stages of like trying to establish like the same configurations for the clusters and eventually yes We are going to come to like to get ops and also using different policies and other stuff Maybe a second question then because you're a bank Do you have some other way right now that you're applying? Policies or that you're checking something or is that something that you don't account for other what sorry policies But you general for the center bank center bank of Brazil you say or like? Technologically speaking right now you're talking about like regulations. Yeah, yeah, so so Our answer with an example. So as I said picks has to be secure New policies and regulations are created constantly because we have a lot of frauds for example When we started it's 24-7, but you could transfer any amount of money After 8 p.m. Today you you have a limit. So like the There's a new regulations that you have to have a limit of I think it's 5,000 highs max you can change that with your bank, but you have a limit after 8 p.m. Until 5 a.m. to to because of frauds or anything and this the busing security the central bank security team keeps investigating and and Release a new regulations and policies for that. Hey, can you go back to that slide of the SPI? So we have two types of participants in the system One's called indirect participant and the other one's called direct participant. So the direct participants they need to have kind of like a long-term relationship they they had to Like be at the central bank of Brazil had a relationship with the central bank of Brazil for long long long long time But to make it the real-time payment systems broad they created this category of the indirect participant So there are some institutions that they don't have to follow the same rules as Banco original But they are still able to use it the real-time payment systems through their APIs So they need to follow like the strict rules and regulations But those ones they are still included, but they use it through Other participants basically did the policies and the the rules are restricted to the regulated banks That that's why indirect participants cannot Interact directly with the addressing database only through direct participants Yeah, okay. Thank you. Thank you Another question Here Man in the slide you showed 113 TPS Yeah, I think it's I That's too much. How do you handle that traffic? What is your RP management strategy? This was with picks. It was a peak so what helped us handle that was HPA with our cluster and the performance SLOs That and the performance C testing I mentioned we had on our pipeline so we can If we can evict performance issues on the QA environment so because before going to production our staging environment Because if a system doesn't meet all the requirements performance talking it won't go to production so HPA and The probes and the performance testing we have on our on our pipeline helped us a lot to achieve that Was a peak so we can achieve a lot more our environment was healthy But if we continue that for like a day or two maybe something Will some problem will pop up pop out and we will have to solve it Yeah, and uh, you asked specifically about the API management like a tool or Release strategy. Okay. Yeah, but You question on that so off-tuning right so especially in our open-shift Especially in our open-shift environment right Performance tuning we all understand regular of Tuning strategies and all of that right? Can you can you give us, you know the one Big thing that was in the way and you guys had to solve For open-shift for perf tuning Right any any one big dog that you had to you know sort of solve for that'll be helpful Yeah, I think sorry the most difficult as we said was the Culture of all our teams so our infrastructure team manage a bunch of VMs and Bermuda and all of that so we didn't have knowledge and a culture of a distributed Architecture talking about I think few people knew about what a pod that was what a pod was so the Learning curve I think and the culture the spread of the culture or the most difficult challenges we faced That's why when we met we met I I talked to him about all Opportunities and needs we had to spread that culture and that knowledge Throughout the company you actually met in an insible event right yeah, I wasn't even open shift yeah, it was a nice and Because Technologically talking it's we had all ready had support but because it's configuration installation and read the doc stuff, but when we talk about culture and learning curve of Different a different set of teams. That's when the challenge came for us Yeah, if I can dig just a little deeper like this is a Typical digital transformation scenario not only for the market, but for the companies itself because You need to have like the way that you work like Prior to those that revolution is not going to work if you want to solve the problem that way If you don't implement like SLOs SLI's SLAs site reliability engineering platform engineering and Other kind of technologies that can help you you either going to take that long to to solve all the challenges Or not going to be able to do it So when we when we started like the way it was like okay So we have to and we know how to use it. So how to tune it as well How we can spread out this message across the bank because I cannot have like the architecture team Being called like every day when you have a problem at 4 a.m If they don't manage if they didn't have like if they they weren't the ones that made the changes like to the code Just because they know open shift. That's not that's not the right answer, right? So the first the first thing that we did was like, okay, you guys have like webinars, right? So we are done, right? There are more questions. They needed us to stay here But yeah, it's not it's not all about technology is about partnership and also culture. Okay. Thank you guys