 I'm a site reliability engineer Good afternoon. That was entirely planned. We wanted a fuller room now. We have it We had no problems whatsoever We have Sherwin from Robobank Product owner for cloud as well as sorry kind of jack-of-all-trades. I suppose Yeah, give it up for Sherwin, please. Thank you Thank you. All right. Thank you all for joining This last Friday last talk Hope you had a great Day today. I saw some really interesting talks as well Some of those will tie into what I'm going to show you today so I Want to talk about Kubernetes in the enterprise What does it mean Kubernetes in the enterprise? Well, first of all, let's talk about what is the enterprise? so For me, I'm like a technologist at heart. So what I thought the enterprise is is just like Yeah, dusty old People sitting at the cubicle farm something like this, right? Who thinks about when you see an enterprise who thinks about this? Show of that. That's exactly what I thought, right? But when I got older and I saw what people actually do and the processes like the enterprise is much more than just a couple of people Or like a bunch of people in a room having meetings for me an enterprise is a cohesive ecosystem that creates value for the people that work there and also The people outside of the enterprise where where the enterprise of services for It's a highly advanced technological machine Where people work together they have different roles They have the different outlooks different skills and together they create an amazing product for their customers and That's what I think an enterprise is so for me now an enterprise doesn't look like this It looks like more like this This is an enterprise. It's an ecosystem if people don't work together what happens to the enterprise It falls apart or it crashes. You know what happens with this thing And it's a fun story because When I started my career a decade ago And we the the people that were there the technologist and the lead developer He called me Georgie LeForge and I didn't know what Georgie LeForge was. I wasn't a Star Trek fan but He commanded me that I can come up with like very good technological solutions in a short amount of time and Now that I'm like looking at this whole endeavor what I've been doing and stuff like that I feel like a Georgie LeForge this this guy kind of look like him if you look at it Thank you But we also have other people of course different roles This is the frontline risk that we have at the double bunk. These are the security guys They're very strict. They're very to the point, but they are fair because understand with an enterprise there's a lot of Problems that can happen and you don't want to basically bring the enterprise down and we also have another role These are the architects and This is what I see this here John if you have seen here John. He's right there Kind of looks like him to Yeah, he really Yeah, so he really makes sure that we do the right thing because he understands not only low level But also what happens on top? He speaks with management and also the people above that Which is very important to make sure the enterprise keeps on flying around All right a little bit about myself So yeah, I I'm Sharon Noidmeyer. I'm an SRE at heart. I've been doing this as I said for more than a decade now I worked at some more notable companies like Spotify back in 2013 Then I'm originally from the Netherlands, but I'm born in Suriname. I So I worked in Sweden then I went to London worked for Palantir Technologies I started my own company there as well did a lot of cool things all the way from Bare metal putting it in the racks and making sure that the fiber channel is working the Cisco switches are working Everything is redundant Everything is working fine and then the and the monitoring system is in place and backups as well And I just basically grew from there to The SRE that I am right now. I write a lot of automation. I write a lot of go I do software engineering, but mostly on the architecture and now as well with the rabble We're at the forefront to transfer to the cloud and there I help the team our team team Vikings together with also TCS to make sure that we get there the application that is now working on premise And we want to basically get the application to the cloud TCS has helped a lot of that too and why am I showing this particular picture so you can see this commit has been made by me in August 2018 this is the first time I actually wanted to deploy Kubernetes in a bank the bank was Network International It's a bank in Dubai and I was there and basically want to first get me to install Kubernetes on premise I thought okay That's it's gonna be a difficult ask because I have a lot of policies there a lot of things that you can do You kind of just pull images from darker hub with your Bitcoin miners in there and stuff like that It's it's kind of difficult, but within a week with having also first-line risk there on the phone I basically installs Kubernetes in a week on premise In the network international in the bank. That was the first time. That was in 2018 the beginning of 2018 And when I did that, I'm like, yes, let's use the cluster and then he said no, we have to do this in the cloud So here we are we were using a AKS engine and actually was not AKS engine. It was called ACS engine AKS really wasn't generally available at that time So what Microsoft did is they open sourced the engine that basically installs Kubernetes for you on Azure But it was not ready for a bank and why was it not ready for a bank? Yeah, you can see here right now There's a nil pointer when you want to use hardened images and that's what you want to use when you're in the bank You want to make sure that you're secure You want to use your own images, but if you use your own images and you install Kubernetes What happens it crashes. Why? Because you didn't check your pointers. So I needed to make sure that that didn't happen. So I Made this commit and the same day it got merged and we could actually continue to deploy Kubernetes in the bank Which is great. And this is what I call pushing the envelope because basically AKS was not ready for the financial industry at that point in time without this particular single line of code All right. Thank you. So The enterprise is not only humanities in the enterprise not only a technological challenge. It's first and foremost a Organizational challenge you kind of just go in and then install a cluster have a public load balancer exposed All your S3 buckets exposed and then think you can run a bank like that before you know What you're hacked and then you have to pay some like I don't know some ransomware money to make sure that you get your data back So to make sure that we don't make silly mistakes because like the number one mistake is like open S3 buckets And for Kubernetes, I thought I like I heard a different talk not so long ago There were like 1 million open API endpoints in the wild Isn't that crazy? So we don't want that to happen at the rabble bank. So what do we have? We have a bunch of Enterprise roles people groups. We have the cloud competency center the CCC an incredible team at the rabble bank What they do is they help you to make sure that you make the right choices inside of your AWS account so you don't open up like The API endpoint for everyone, but also if you do something that is silly There are automated tools to make sure that that silliness won't actually become a problem because if you create a public load balancer They make sure that it automatically gets deleted Already talked a little bit about first-line risk It's very important the first-line risk make sure that all the policies are also in place So we don't expose the Kubernetes cluster or actually anything in the AWS Environment or any actual environment rabble bank is multi-cloud So we make sure that no problems happen or the risk is at least mitigated or we accept certain types of risks We have identity and access management We have to make sure that we have all the accounts people who can access the cluster who can do the certain things you have to make sure that Like a group of people can access the cluster we have people from TCS right here They can also access the cluster in a particular way That's a whole group of people that make sure that you get the right identity and access We have the infrastructure architect the solution architect the business architect and much more Those are all people making sure that the enterprise stays afloat All right the placement in the investment stack. So what are we actually doing, right? Why do we have Kubernetes as I mentioned before we want to make sure that the Investments back office The the blue circle the blue square right there. That is basically that what we want to run That is the whole back office if you want to do for example Investment portfolios ordering corporate actions payments. These are all things that are being handled by the back office There are connections to your next some of y'all probably know what your next is This is where you can place your orders and hope for execution and this needs to now run on top of Kubernetes We had it on premise before when you go in with rabble and you have like your Your phone and you want to create or you want to start an open investments account This is what you see you go through the whole stack and in the end You come with like the complex workloads where the investment back office runs on Kubernetes and the red part This is where I come in or we come in as a team to make sure that that is good and That the functional teams can consume that All right next the application architecture. Well, this is something looks very complicated I'm not gonna go into details But just to give you a view of like how Complexes is there you see all the connections from the external side come in and all kinds of things are being hit And they talk like we we run now with like 40 transactions per second Which you can imagine any one transaction can multiply it to like five or even ten times as much because it propagates Across the systems and different system need to do Different things so the client can be helped some metrics So we have 19 systems integrated we have 44 APIs provided to the front-end applications eight different communication protocols and Seventy different types of batch processing and that's all in a nutshell what you saw in the picture before now some fine grained components I Myself particularly interested in the Kubernetes part and what are you running? So these are all the components that you're running. This is DevOps That's the DevOps squad that we that we have this so basically we have like we have Splunk with Dynatrace Amazon CloudWatch That's the monitoring side To deploy Kubernetes we use Terraform Customized that's basically The building blocks to deploy Kubernetes on so Terraform to deploy Kubernetes and then we use customized to do the rest and Inside of the cluster, of course, we have flux What's where's flux on our list right there? Right there So use flux CD so we use get-ups to make sure that All the credentials stay in store inside a cluster and the state is reconciled with the get repository and Also the Docker registries Right a little bit more metrics. So we have three accounts and five environments Well, it's we have more accounts with three active accounts at this point of time and five environments How can it be because like normally I will do like one account per environment But it's a different setup that we have here because if you have to if you creating an account You have to go through the whole process There's a lot of people as I said before Involved to make sure that we do the right thing and it's much easier if you want to create a functional environment That you can do that on top of the Kubernetes cluster because you can use the Kubernetes to run different types of workloads So why not just have like different node pools that you can use that are it's specifically Used for a particular functional environment. So we have performance tests. We have acceptance environments We have all kinds of other environments for like for the station Account we have the development account, of course This is where the initial CICD process starts from the rabo side and we have also production right now It's just one one environment production, but it will be split in multiple environments You see also the amount of nodes that we're using and staging has the most nodes 51 nodes 720 cores and Almost three terabyte of memory So yeah, this is a bit it's a pretty expensive, but yeah We are we are an enterprise and so the Operational expenses might be a little bit higher, but the capital expense is much lower because you don't have to use any on-premise systems anymore As I mentioned before we do get ops This animated gift so immediately you see what happens flux is running inside a cluster we have a developer Developer changes the git repository and what happens flux make sure that the system is reconciled But we also have darker images and if there's a new darker image that gets pushed to the registry Flux will automatically deploy that image to the Kubernetes cluster. All right So We are now in the process of going to the cloud It has not been an easy process. It's actually a very complicated process There's a lot of things that we've touched. There's a lot of things a lot of communication that we need to have It's also there's also technological challenges What I see in a lot of companies is what they what they do usually is they make their own bespoke solution on top of Kubernetes we already have a lot of YAML files and what they then do is you create a template system to make sure that you don't repeat yourself And I think that that is not really the right solution to go I know how internally use templates, but otherwise I wouldn't use any templating system to To do YAML. I wouldn't make anything bespoke. So What would what would we do different now? We would first of all, I would like standardize as much as possible So there are multiple groups of people in the rubber bunk that are using Kubernetes At least two of them are using customized one of them has a bespoke solution And as we have seen in the talk before you want to remove your unicorns. You want to move fast You don't want to have 50 planes that you can make in a year. You want to have 50,000 right? How do you achieve that? How do you get agility is by standardizing so maybe not everyone will like that You have some Rockstar engineers. They want to do their own thing and of course I understand that's also very possible a very like like everyone wants to do that But for an enterprise that just doesn't scale So standardize The other thing that I find very important to do is involve enterprise Expertise earlier. We've made some decision. So inside of the class. We're also using Argo Argo is a big part of our CI CD system and it's a tremendous tool. It works very well It works very fast. You can do a lot of things with that. It's a versatile tool But inside of Rabobank, we also use Azure DevOps and the pipelines are compliant There are security checks and what's going to happen now if you're using a tool like Argo It needs to go to a process to accept that tool It's a Crowd a cloud risk assessment that you need to do before you're going to run a tool inside of the cluster There's any tool that you get from from from darker hop or anywhere else Could be compromised or has a maybe not the right security posture You want to verify that first before you're going to roll that out. So these these decisions need to make not likely Let's see. So the next one is start processes earlier So if you something a tool that is not necessarily rabble endorse like Argo What you can do is start the process early if you want to Properly risk assess that tool You have to make sure that you go through the proper channels and it can't take three months Yes, there's a lot of people that need to sign off on things People need to actually check people check the code people will do pen testing people look at Even the maturity of the project and the impact of what you're doing there and another thing is very important It's the consideration for day to operations. What is day to operations? You have first day operations as basically when you install the cluster when you Install all the tools when you just start doing things But you have to maintain a cluster. You have to have updates. You have to make you have to Install new tools. You have to make sure that you are with the right API versus Etc. So the tools that you pick right now, they only don't exist now But they also exist for your predecessor. So you have to make sure that you also take them into account Let's see So there are a bunch of key takeaways Now first of all keep it simple See a lot of people wanted to make a lot of interesting things But I like boring now. I'm now like 36 years old Some people think it's not that old but for me I've been in there in this industry for too long and I don't think like new and shiny It's not it doesn't excite me as much as actual having some stability making sure that we can continue the business Even though if someone has if someone can't Fix your tool and someone is very good at the tool and it's not there. You should be still be able to fix that Don't temple YAML. I can't say that enough Don't create your own bespoke systems And for Kubernetes, it's very important to extend your base infrastructure configuration So don't template it extend it and use customize and I also want to show you a little bit about customize Let's see. Oh now I have to mirror my screen, right? Let's see if I can do that What if I stop this presentation for a second or can I just mirror the screen? I'll just mirror the screen So who of y'all is familiar with customize? Nice very nice Who isn't? Okay, okay. There's a lot of people not raising their hands So I don't know who knows what like like what is customized? I will like shortly explain that customize is a tool that basically allows you to patch your YAML And that's so it's so you have a base YAML and you can basically say well I want to take this base YAML, but I want to extend it with a little bit of this with this snippet. I Think you can Extend no average here average here mirror for built-in display and there we go There we go. Hey if you want to check it out because I don't know what this thing is doing Yeah, come over Oh this place Decisive Mirror for display. Yes. Yes. Oh When you go there, then it does it doesn't do that. Okay, let me just fix it real quick by doing this Wait a sec. Yeah, because it does this again main display It might be power point messing it up. What's that? It might be power point messing it up. Yeah power points messing this up right now Yeah, maybe stop the presentation and then I Have to check it here now What's this? Yeah, I know I know there we go Thank you So very simple. This is So flux if you have to first install flux and we have installed flux I think this is visible for everyone. Let me just make sure that is a little bit bigger So this basically is the basic flux configuration that you can use to say, okay, I want to like Already configured to get repository and this is what you're doing death. We have a director called Kate's short for communities clusters death so That happens So if you see here already we have here the director called case base This is where all the yaml is in resides in and then we have the case directory right clusters and Then we have death just like you see here in the path, right? And then if you look at for example all the applications that are installed I'm not actually showing rftm rftm is it's not our TFM. It's our FTM It's not a typo. It's a rababank file transfer manager So of course you can't just like exfiltrate data or infiltrate data you need to go through a system and There are different endpoints you have like S3 buckets with also links machines You have all kinds of like other like endpoints But you want to control from one place how data guts gets from outside of raban inside of rababank And so that is rftm rftm has an API so you can configure rftm and I wrote a little controller there So this is like an app that I that I installed on the cluster and so but basically you see here that we're now when we now Went to went to customize you saw that went to app that like the Clusters deaf and then we have some apps here and here you see that I'm pointing to Kate's app deaf rftm So let's see Right Kate's apps deaf And then we go to rftm Here we have the deployment but look at this. This doesn't look like a full deployment, right? This is a partial file You only see what is that but a specified is basically the image here. So we have an image specified But where does this come from? So there's a customization and then you go two directories up point point point point base rftm So you go up base And then we go to rftm And here's the full deployment And so with customized is very easy you go from this full deployment. This is what you have This is full deployment But in the end the only thing that you do for deaf is you change where the image is coming from because you have to point to the repository or to the registry of deaf and that's what I'm doing here I'm pointing to the deaf registry So and it's another thing. So you've seen that I've installed an rftm controller So the nice thing about an enterprise is that it is a big ecosystem of Technology and people that come together to create value But they're also so what they also do is they create a value for people internally inside of the company So it's so we have a thing called rftm But it does it's not only a GUI it also has an API and what you can do in the enterprise There's a bunch of API's available and you could build all kinds of controllers for the API's that are already there So this is this is the first controller in the rabble bunk That's actually using an internal API and I think a lot more will follow This is an amazing process So you can imagine if there are a bunch of API's to do all kinds of things in your enterprise You can not only use Kubernetes to deploy your applications You can use Kubernetes to basically configure your entire enterprise So working for an enterprise basically becomes like the Starship enterprise You can just go and sit behind your laptop and do all the things from the console and in the end You don't even have to necessarily communicate to all those kind of people because you can just create your your YAML objects and communicate with the enterprise like that So I think this is good for now. I still have something to Let's see. Here we go. Wait a sec. Wait a sec. I need to make sure that this is gone Actually need to check it over there. Is it almost almost? Yes. Oh another one. It's gotta be kidding me, man There we go All right so one thing Show of hands how many people are actually working for an enterprise right now enterprise organization Okay, and of those hands that are up Who is not using Kubernetes at this point? Okay, you guys okay, so That's a very interesting thing If you want to know how to convince your company to work to use Kubernetes First of all, you have to understand respect your organization Right a lot of people think like okay the organization is stupid They don't use Kubernetes and that's super bad and we're gonna just fight the organization That's not what you do. You work with the organization. They're there for for a reason you make sure you follow the process You talk to the right people get people together make sure that you understand what you're doing You also understand the perspective of management like I can say all kinds of things for a month I can say yeah, we need to use Kubernetes. We need to do this or that or whatever, but in the end Management has a whole different mindset. They don't know about containers. They don't know about classes. They want to know Does the business work? Can it continue to work? Are we in control? Are we not creating like weird risks? Like they need to ensure business continuity And you need to make the case that the communities make sure that you can actually have a better business continuity And if that doesn't work you just basically ask for forgiveness Because you're just gonna do it anyway roll it out. Maybe show something and then you can back for permission later And if that doesn't work, you should apply somewhere else. We're hiring Thank you Thank you, sir. I mean That was brilliant. Thank you very much You're okay, and and I was just I was just talking with with here from Rob I want that the the very first item respect your the organization seems Kind of obvious, but it isn't bringing something in new requires Courage of course, but also respect for those that were not thinking it or that didn't want it So you have to have that your own way to exactly to talk it through. Yeah, so thank you very much Yeah, you're welcome a game for them for sure. Oh, I'm sorry. I'm sorry. Where are you going? Stay here questions, please Yes, we have one over there. We have one Hi, thank you for your presentation. I do have one question I'm here. Oh So I agree with the not templating your emails, but what about templating your TFR files Templating TFR files For what purpose just asking you Yeah, you could okay. Let's say I want to have one code base from terraform from which I can inject different TFR files for different environments, for instance Yes, so you can do that. Yeah, I know but you're allowed. What's your thought on that? Well, like TFR files, it's kind of different, right? The problem with templating YAML is you can create all kinds of problems there There's a wide space that that is an issue You don't know what it looks like in the end, but TFR files is basically just a flat file So I think that's perfectly fine to template you don't have the same problems with just a flat file That as it is with YAML Let's do that Sounds like a threat Bring it on Hey, thanks for your talk There was a diagram with all of the components and then some of the things that were Pin that a certain version like helmets plunk and so on. What is the update strategy for those components if there's any? That's a very good question There is no update strategy as of this point. Yeah No, so like yeah Basically, you can do that by hand of course, but that's not gonna work Like you have also like because we were using get-offs You can use something like the panda bot to upgrade certain types of things, but not everything can be easily upgraded So we have to look at it per component on how to see and specifically for day two operations And that's something that because we're in the forefront of the Kubernetes world and the cloud Transformation we are also looking at things like this and how to do that correctly Serving we have space for two more questions. Are you okay to take them? Yeah, sure I I think you already mentioned but I missed it probably how are you managing secrets? Yeah, good question Yeah, well we use cube seal in the beginning, but that's like There's a lot of overhead to cubes you because you have to Cycle the keys and stuff like that. We don't want to do that. We're now using external secrets So most secrets are in the secrets manager in AWS, but we expose the secrets via external secrets So it's for it's in the cluster. So the application can just consume those secrets Yeah, is there anything you can share about what your organizational team dynamics look like to enable all those repositories and all those structures because if you're trying to Build these for developers if you're trying to enable developers then are there any structures in place to have them productive Let me rephrase that question. So you want to know basically how we Expose the cluster so other people can consume it easier how How you enable your developers to release Okay, yeah, so for this particular Instance of the Kubernetes cluster that we have right now Everything is coming from offshore. So the developers don't necessarily interact directly with the cluster They basically deliver the tar ball and we make sure with our CI CD system That a darker images automatically created so they don't interact with the cluster like that. However, we are looking at Forming a platform team, which we're going to answer that exact question, but conversations are still ongoing Thank you very much. So in Please one more for showing Okay Please do not leave the room. We are going to have okay I'm gonna give you one communication service communication We're gonna do one group photo with the whole team and then we have our last mysterious keynote speaker I Okay, so We are going after after the last keynote. We are going to go everyone We're going to be serving drinks in the sponsor area And around six maybe a little bit later. We are there are too many Italians in the team to make it on time. So We are going to we are going to move and have our barbecue vegan barbecue in the bar area Okay Thank you