 Yeah, okay. Let's start. Yeah. Hi everyone. I am so happy to be here. We're both so happy to be here. I know I'm speaking. Amazing. Thank you so much, everybody. Yeah. So I think now is a great time to start. But wait, let's do a selfie. Right now? Let's do it. Let's do it right now. All right. All right. Come on. It's a beautiful audience over here. It's a beautiful day to begin. Let's do a quick selfie. So we can do it. Everybody make some noise. Come on. No, it can start. Okay. Now I feel ready. All right. So it's a good time to start talking about the terrifying migration to Kubernetes. And I am sure that when I say terrifying migration, all you can think of is failing apps and nodes crushing and pod errors and developers running around aimlessly. And what else? Crazy. And 10,000 tickets and PD alerts all over, right? It's pretty crazy days for sure. And all of this may sound unreal, right? Or at least it sounds like these are not your problems. It's not our problems. Obviously not our problems. Well, wait. I think it's a good time for you to present yourself. All right. Cool. So good morning, everybody. And thank you for joining us today. My name is Batel Zoran. I'm working at JFrog as a developer advocate in the last six and a half years before that I was a developer. And I love to talk about security and DevSecOps. And my amazing colleague today, Mae, will introduce herself. Yeah. I am Mae Baseron. I'm DevOps at Forder. And I am currently leading the migrations of 420 services to Kubernetes. And I'm a Sworn Star Wars fan. I don't know if everyone noticed that I have R2D2 heals. That's amazing. I don't suffer for nothing. Clap. Come on, clap, clap. Give it up for the R2D2 heals. Thank you so much. All right. So now I want to take you all to a great journey to a far, far away land where we have Silicon Valley. And in Silicon Valley, we are going to talk about a small company or medium sized company named Pipe Piper. Pipe Piper is a compression, is a file compression company. And as such, they have a lot of data to mess with. They have a lot of heavy lifting to do. And they needed to go on scale. Now, Pipe Piper? I mean, yeah. Yeah. We promise that we will talk about Kubernetes migrations. And we are going to do that. We're going to talk about the Kubernetes migrations at Pipe Piper. So over here, you can see the two DevOps from Pipe Piper. We have Dinesh and Richard. And what they're doing is they're planning the migrations to Kubernetes. They started by planning the infrastructure. And the way that the infrastructure goes at Pipe Piper is that the configuration and the provisioning of the cloud resources is actually done by the developers. And it is done by using a Python DSL. So they take this Python code and they generate some cloud formation templates. And this is how they define the cloud resources. So those DevOps guys, they decided to use the same Python code in order to generate the Kubernetes manifest. And that was the time to start the migrations. All right. So let's talk about the migration. And let's meet Monika. So Monika is a backend developer at Pipe Piper. And she got a sprint task to migrate the API gateway service to Kubernetes, which sounds pretty simple, right? She already have everything she need. And it's labeled. We have two days' work. What can go wrong? So two days for migration. First migration? Totally legit. Pretty legit. All right. So she started to learn a bit more about Kubernetes. She opened the official documentation. She read a lot about Kubernetes. And then she figured out that it's written in YAML. She expected for Python, right? They have their code DSL. And yeah, she decided that she can't do that. It's too hard. And you know what? Seven tickets, 14 days, a thousand coffee mugs, Brazilian questions to the DevOps. And you don't want to know what it looked like when the DevOps got Brazilian questions. I'm at lunch. Sorry. Don't bother me. That's how it looked. And she was still not done. It was horrible for her. Not. I wasn't there. No, no, no. Yeah, it's not our problem. It did not happen to us. And if we want to talk about what was the problem, then we can say, oh, it was the fact that they used an in-house tool, right? Wrong. Because there are a lot of companies who use in-house tools, and they don't experience these issues. And I think that if I want to put it more precisely, then we can say that the problem was the knowledge gap. And how do you know that you have a knowledge gap? It's when... Sorry. I'm sorry. I'm sorry. Oh, my God. I tried to look at them. You're so serious. You coughed up. Yeah, it's terrifying migration, you know? Thinking about the knowledge gap. So how do you know if you have a knowledge gap? It's when you expect the developers to do something, but you don't provide enough information for them to be able to do so. So was that the end of the problems? Not really. So it seems like everything was good. They celebrate. They have a great company, all ends. And then suddenly they got speedy alerts. Like everybody here loves, right? So they got some errors. But wait, wait. A second before that, a few hours, they just had some great, great issues. Just another slide, please. So they already migrate everything. They were so shorted. Everything working perfectly. So they decided to remove the old migration, right? Why do they need to pay for it? Everything works so good that they don't need this old EC2 thing, right? So they have their Kubernetes environments and tons of PD alerts. In the middle of the company, all ends, they took their laptops, started to debug everything and see what all of these 500 errors. So let's go back one step and talk a bit about the environment. So it looks pretty simple, right? We have our Kubernetes application, we have our bucket, S3 bucket, and we have our database. And everything should work perfectly because they just copy the configuration from the old environment. They still have their security rules. They still have pretty similar S3 bucket and databases. So when one thing was missing, actually, they already have their security rules, their firewall rules, and it seems like everything should work perfectly because the test did a bit and the services was running. But whenever we're talking about thousands of services, it's a bit more complicated. In some cases, it's really worked perfectly and sporadically, actually. It's working, right? And it seems like it's good. But in some cases, we got 500 errors from the database. So we couldn't connect to the database. The application won't run. And after one hour and a half, something like that, maybe even two hours in the middle of the company, all ends, they figured out that the firewall in the IRC rule was blocking some of the cases. And therefore, they can connect to the database. They fix it after talking to the amazing security team. But still, it's taking time, right? So the problem here is actually to unknown the unknowns. Sometimes it won't work. We don't know why. We don't know. Even if you test it, something could go wrong. And we'll talk about it later. Right. Yeah. So I want to say that there was another problem. But I think that there were many other problems to be honest for them. Again, we're still talking about this fight by per company. Not us. Definitely. Not me, not you, not you. It's not for you. So now the next thing that I want to share, like the next problem that they had was they just wanted to do a very basic day two operation, which is to upgrade the Kubernetes version. And once they started to do so, they started experiencing errors. One by one, everything was going down and going crazy. And you know, if I will just take you to the point of it, the problem was lack of configuration. Oops. Yeah. I, okay. Not to know that it wasn't the problem, but actually it was. The problem was that they had lack of configuration. They didn't have a configuration for order of termination or lack of pod disruption budget. Now I already showed you this slide because I wanted to say that this is the problem, the lack of configuration. But again, that's not really the problem. Because sometimes if you do have a problem of lack of configuration, it still does not mean that you will experience all of these errors or you can come back from them very quickly. So what does it mean if it wasn't the problem that they had like lack of configuration? The problem was very simply that they were not ready. That's it. That's the issue. All of those things that we've been talking about, it's for us to show you how that fact that they were not ready affected production. Yeah, that's it. That's it. So I want to say that there is a light at the end of the tunnel, but actually there are three lights at the end of the tunnel. Three lights. Come on. Let's look like a light. Okay. It's firecrackers, but still it's kind of a light. Let me show you. Let me show the light at the end of the tunnel. Let's talk about the first light. So the first thing that you want to do is you want to have mindset migration. What it means is that you want to first overcome the knowledge gap that your developers have. And in order to do that, you need to understand that documentation is crucial. So you have to prioritize it accordingly. You need to provide with the information that you want your developers to know. And the next thing that you want to remember is that communication is key to success. And you want to invest time and effort in communication channels. And when I say communication channels, I am not talking about creating more Slack channels. I think we have too many. I can't see Slack anymore. So I'm not talking about adding more Slack channels. What I'm saying is this, if you decide to start doing Kubernetes migrations, what you need to do is first to set up a meeting which is just for Kubernetes information. Just to set up the field to allow the developers to ask questions, to allow them raise concerns. And then you also want to do a separate meeting for discovery session. You want to sit with your developers and go over their systems to figure out what it is that they need in order to be up and running on production. Because it is very different than running on VMs. So the second one is not just production ready. We need to be migration ready. And it's much more than just be ready for migrate. So we need to plan and configure the rollback set. First of all, what will happen in case that something goes wrong, right? We need to make sure that we can trust our environment, but we still can roll back to another environment in case that something happened like we saw in PipePiper, right? It's never happened to us. And the second one is hybrid mode, right? We always need to make sure that our customers, our developers, or anybody that's using our platforms is up and running and can work with the tools that we provide them. And we have everything we need in case that something goes wrong, right? So the last thing that we want to share is that you want to plan what happens, what happens when we get to shore. Now, let me tell you a little story. Let's think about you decide to go to New York to do a relocation. And you decide to take a boat on, you know, to get to that New York with that relocation idea. And you plan everything about this cruise. You plan how you go up on that ship. You plan which room you're going to be in. You plan your suitcases, what you're going to take with you, which shirt you're going to wear. But you don't plan anything for when you get to New York. So nothing. You don't plan about where you're going to live. You don't care about the school for your kids. You don't care about even a job. And this is what it is. If you're only thinking about the migration themselves when you think about the Kubernetes migration. If you don't think about the day two operations, it's just like not planning what happens when you get to shore. So one of the things that I want to think, there are many things to plan for day two operations. But I think two of the most important things that I've experienced was having an infrastructure testing environment. And I'm not talking about just testing for the developers to test their services, also for testing infrastructure. If you want to change configurations, if you want to upgrade versions, you need to have an infrastructure testing environment. That is so crucial. I cannot even say that enough. I know I've said it three times. But okay, I think it's time to move on to the next one. And now I want to say that upgrades and breaking changes, that's another thing that you want to get yourself ready for. You want to think about breaking changes, what it's going to look like from the developer's side. Can they not take those breaking changes? Do they have to upgrade their versions? All of these things are things that you need to think about before you do them. Definitely before. So because it's never happened to you, right? My lips are sealed. But what I'm going to say is this. There are many things that you want to consider when you're doing Kubernetes migration. And I think that we can definitely do a talk about each and every one of those three points that I've just discussed. As you've seen this infrastructure testing environment, just that I can stand here on stage for 40 minutes, tell you all about it, how to do it, how it worked eventually, and how it saved my life. But we don't have time for that because our talk is almost over. So I think we have just enough time to say big thank you. You are amazing. You are amazing. We love being here. Thank you. Thank you. Thank you so much.