 All right, I'd like to thank everyone who's joining us today. Welcome to today's CNCF webinar, I'm Jeffrey Sika, I'm a senior software engineer at Red Hat, and I'm a cloud native ambassador. I'll be moderating today's webinar. We would like to welcome our presenter today, Baruch Setagorsky, head of DevOps Advocacy and Developer Advocate at JFrog. A few housekeeping items before we do get started. During the webinar, you are not able to talk as an attendee. There's a Q&A box at the bottom of your screen. Please feel free to drop your questions in there and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful of you and all of your participants and presenters. With that, I'll hand it over to Baruch to kick off today's presentation. Thank you, thank you very much, Jeffrey, and thank you for having me to CNCF and thank you for coming, all of you. So today, we're going to talk about continuous updates and what it is, how it's different from any other kind of continuous things and how we go about it and how we don't go about it. So before we start talking about continuous updates, I would like us to think about a couple of things that we take for granted. And first would be why at all we update software? Obviously, there is some software which doesn't require updating. And that's an example of a BC for probably most of you know, for those who don't, BC is a command line calculator. You type two plus three, I get four. And obviously this kind of software is not updated very, very frequently. You can see here the history of updates. Frankly, I have no idea what those updates are required for. It's not that the basic rules of math changed from 1991 to 2000, but even considering those updates, you can see that it wasn't updated for 20 years and it's probably okay. There is nothing to update there. But obviously this kind of software is an exception. Most of the software is updated, updated frequently. And I would say that it updated frequently for too many reasons, too many reasons. The first is us. We are users and what we want are shiny new features. Obviously we want shiny new features as frequently as possible. And the best are now or even yesterday. And you might relate to this if you had a phone back in the 90s, some Nokia with or without the snake game and thinking about it, if you didn't have the snake game, what did you do to get it? You actually, you had to buy a new phone. You couldn't just install it. And there was like this thing, data cable. Some of you might remember the data cables that you had to buy in the store and what you could do is that nothing. You could probably back up your contacts to move them to a new phone when you buy a new phone to have a snake game. But obviously everything changed with the iPhone 2007 and all the smartphone revolution. And now if you want a snake, you just go to Google Play Store or Apple Store or App Store and just download it. And you know what? Day after day, you have new features. You have new levels. You have new features of or your app and beat installed. And this is something that we demand from the market. We as users and then we provide it to the users as developers. This is something that we are all very used to those days and probably those companies who don't do that will become Nokia for that example of our phones, if you wish. The other reason which is also very, very important is security. And as every company becomes a software company, security vulnerabilities become the new oil spills. They become the most threaded thing now. And you can remember a couple of them that were really big deal, probably as big deal as oil spills back in the day when we had oil as the king of the economy. The way to think about fighting security vulnerabilities is a little bit as the way to think about stopping distances back when you got your driving license, if you remember. So there are two different components to stopping distance. What they call a sinking distance and what they call a breaking distance. And they are completely different in what they are. So thinking difference is the neurons that fire in our brain as a reaction to light getting to our eyes retina. And then the electricity that runs through our body to the muscles that actually push the brake pedal. And that's one kind of this formula. And the other is completely different. Breaking distance is about the brakes that hit the wheel, that hit the wheels and the chemistry between the rubber and the pavement. That's like completely different thing. Now, what's common between them is in order to shorter the stopping distance, you actually have to invest in shorting both of those with completely different tools. So with breaking distance, you come with new alloys for the brakes, you come with better pavements, you come with better tires. And for thinking distance, basically the only thing you can do is replacing drivers who are slow humans with machines, the autopilot, which react much faster. But anyway, two completely different aspects compose one battle. And it's the same with security vulnerabilities. You can think about three different aspects of fighting those vulnerabilities. You have to identify that you are under attack and then you have to fix the problem, patch it. Most of the times that will be just upgrading your dependencies. Dependencies might be an operating system patch that will close this vulnerability and entire new operating system and all the way to replacing the dependency in your code in order to patch the vulnerability. This is what we will do most of the time. And then the third aspect of it is the deployment. And this is where continuous pipelines, continuous updates become a big deal because this is where the fastest you will update, your application, the better you will be. Sorry about that. I hope that's not the coronavirus. It's good that it's a remote webinar. So even if it is, you're safe. And well, the fastest you will update your software, the better you are. And a couple of examples in this regard. So you might remember almost three years ago now, the vulnerability that took down the NSA hospitals and stuff like X-ray machines and MRI machines were completely shut down because of the virus, PETA, not PETA, whatever run somewhere, hit them. So identify those run somewhere is designed to be identified immediately. But the fix, the fix is actually upgrading of the operating system. The problem was that those hospitals run Windows XP which was out of, after end of life, by then no security patches. This is why the vulnerability that this ransomware used wasn't patched. And obviously it took years for those hospitals and the system to upgrade their operating system. And this is why they got hit. Another example that you might remember is the Equifax debacle. And what happened there is that attackers used Strats 2 vulnerability that was out there before they even identified for two months. And the update was just changing the Strats 2 version in their POM file. And then it took them another two months to actually deploy it to production to upgrade their production services. And while we cannot know if shortening the vulnerability period from fourth months to two months was actually doing any favors, but we can expect that they have been in better shape if the upgrade was faster. So basically we're talking about more like an enterprise companies or government agencies that are very slow on upgrades. And you can say, well, in our environment, this is not the case. We know how to update our software immediately as so that won't happen. And that's all good news, but it's not enough. We started the last year a year ago with the announcement of Spectre and Meltdown attacks that you might remember. And while people spoke mostly about Meltdown because 10 lines of JavaScript code could expose passwords from different parts of memory, I would claim that Spectre is much more dangerous. The problem with Spectre is that, as opposite to what this article say that they, well, here, so what they say is most malware explodes coding errors and poor design. Now Spectre, the problem with Spectre is that it's the other way around. It actually exploits good design. The Spectre class attacks work by observing how a well-designed applications affect the speculative execution in memory and what decisions the CPU makes based on certain conditions in the code. And then they assume that a lot of other software is written in the same way will lead to the same speculative decisions. So the better code you write, actually the more it is vulnerable for Spectre class attacks. And that basically means that they are impossible to defeat. What are you gonna do? You're gonna write bad code just to confuse the speculative execution? Obviously not. So the problem is really that we start with zero-day vulnerability. We start with, okay, this code is vulnerable and now it's a race against the attacker. And the race against the attacker is we need to identify the problem as fast as possible. We need to fix the problem as fast as possible and then we need to update as fast as possible. And updating as fast as possible, that's what this talk is about. Now on the bright side is that we are in a surprisingly good shape when it comes to continuous updates. Instead of DevOps report that many of you are familiar and whoever not definitely should go and read that's the most important document in our industry. And that's a report that is built by what used to be called DevOps Research and Assessment Institute, which is now part of Google Cloud with Dr. Nicole Forsgren and Jess Humble and Jin Kin and other wonderful people who survey a lot of organizations, more than 35,000 for last year. And they build, they ask them questions and measure them on four different measurement scales. And that's the lead time, how long it takes from a code to production, the frequency of deployments, how frequently deployments are done in this organization. And then the rollbacks, the failures of the deployment, what percentage of the deployment fail and then time to recovery, how long it takes to recover from a service if a service goes down. And they take those questions and they group the organizations by cohorts. And you can see that we have 20% of the industry are in the cohort of elite performance. And that's every fifth organization doing a very good job. And those are obviously great news. And you can see that those elite organizations actually deploy multiple deploys a day. That makes them obviously able to provide the value of those features to the users faster and also to be protected from vulnerabilities because they can, or as much protected as they can be because they can update immediately after the problem was identified and the fix was available. And obviously they make them easier to recover from failures and all the rest. But for the sake of our discussion, those are the two main components. The fact that they can be rolling out features as fast as possible and that they can protect from security vulnerabilities. So 20% of the industry already does that. And that's because it's not a new idea by any means. Extreme programming 1998 actually advocated for a short feedback. And that means releasing faster and then scrum reducing cycle time to absolute minimum, releasing faster. And then Toyota production system decided as late as possible and deliver as fast as possible, release faster. And Kanban incremental change, smaller batches release faster. All those are not new ideas. And this is why 20% of the industry already successfully do that. So as I was introduced by Jeffrey, my name is Baruch Sadogurski and I'm the chief sticker officer in JFrog. If you need stickers, hit me on Twitter. I will hook you up at Jay Baruch. That's my Twitter handle and by now you should totally follow me. I promise you free stickers. And if you're still not sure about the stickers, my Twitter handle is on the bottom of each and every slide. This is the most important slide of this presentation. I've prepared a special page for you. Jeffrey comes to show notes, you go there and you find a page dedicated for this webinar. The slides are already there. The video we will upload later today. All the links to everything that I mentioned including the state of the ops report and everything else is there as well. The place to comment, to rate this talk and a small token of appreciation for you joining this webinar, a raffle, amazonicodot go there and participate in this raffle as well. As I mentioned, the bottom of every slide, you have the link to show notes as well. And my Twitter handle, the Twitter handle of Cloud90 Foundation that organizes this webinar and the liquid software hashtag that I will talk about a little bit later. So with all this introduction aside, let's cut to the chase. So we want to update faster. I hope I convinced you that updating faster is a good idea. So let's see what can go wrong. And the first example in question will be Java. A micranel who is, Marco, who is the platform, I think ahead of platform for Java. The most important guy in the Java world announced more than two years ago now that Java is moving forward faster. Java will release updates faster. That was, as I mentioned, three and a half years ago. And that was when Java 8 was in the release and the announced what we are going to release every six months instead of every year and a half, two years, three years, four years. It was really not that frequent. Every six months, no matter what, you have a new Java version. Since then Java 9 was out, Java 10, Java 11, Java 12, and Java 13. Five Java versions was released since the announcement. And let's see how that worked. So we look at the state of the developer ecosystem report from JetBrains. That's the last one. By the way, they're now collecting answers for a new one. So go ahead and participate. But that's the last piece of data that we have. And by this report, we can see that 83% are still on Java 8. Just to remind you, Java 8 was out when the announcement actually came out. And you go like, doesn't make any sense. We just made a case for faster updates and Java made updates more available why people are not updating. To understand what happens, we need to understand how we actually update. For that, I prepared a small diagram for you. This is how we think about it. So update is available and then we need to decide do we want it. If we decide that we don't want it, there is nothing interested like Java 9, then we say, okay, no, I don't want to update. But if there are interesting things like every new Java version, then you go like, okay, are there high risks? If there are no high risks, you can just update. And you can think about tons of examples for that starting from the non-mission critical apps on your phone. You have a new version of a game. Do you update it? Yes. Or Netflix, new version, would you update? Yes. And there are no high risks because if something go wrong, well, you're going to watch Hulu for a day, no big deal. Now with Java version, that's your production systems and there are obviously high risks. And the next question is do you trust the update? And to answer this question, I tried to come up with a vendor that updates you blindly trust. And the closest that I can think about is Apple circa 2000, let's say, 14. If you've been on the Mac back then and you saw a new update for Mac OS, you probably, and this is, you definitely want it, there are nice features. It's super high risks that you're working machine or your life in this laptop, but you trust Apple, so you go ahead and you update, no questions asked. So this was nice. Since then, well, not so much. And if I will ask you how many of you who are on Mac updated to the latest Mac OS operating system, Catalina, that was out for like three months, I guess the percentage will be around 10%. And that's because you want the features maybe either sidecar, there are nice features and there are very high risks, you don't trust the update. So the question is now, why don't we trust our vendors anymore? What happened to the trust that we used to put in them? One assumption will be, well, they forgot how to do a QA. Apple in 2014 knew how to do QA by 2020, they forgot. Well, I don't buy it. We only get better in the things that we do. So that's kind of a very weird assumption to make. I would say that the other hypothesis is the complexity. And obviously the complexity is going up and we and Jeff Rog as the authors of some tools for managing binary files, measure everything in binary files and artifacts and we measure complexity in artifacts as well. So you can see how the numbers of artifacts that we need to manage grew over the years starting from Agile back in 2000 when we started to do continuous integration and that means building more and more and then continuous delivery. That means that every artifact now might end up in production. So we need to keep them around and manage them infrastructure as code or suddenly or our servers and hardware or software as well. And then microservices raise the complexity dramatically. Obviously Docker, every line we change in the Docker file now produces a set of binaries that we need to manage and then serverless every 10 lines of JavaScript code is now an artifact that need to be measured, that need to be managed and versions and controlled. And then obviously IoT now those 10 lines of JavaScript code leave in every light bubble in my house. So that makes it even more hard to manage. So complexity plays a very big role but I would claim that another problem is even more critical and that's a problem of having a lot of data. So this is kind of a prediction of global data sphere by 2025. Zetabytes is a lot. Today we think about petabytes as a lot and when we speak about big data, we refer to petabytes. There is one more after and that's exabyte and that's like really a lot. And after exabytes, this is Zetabytes. Now this is something that we cannot even grasp as of today. And although this source of this research is a Seagate which might have interest in inflating the numbers a little bit, but even if we divide it by 10 and we say it's 17 Zetabytes, it's still, it's not comprehensible. And that means that officially staging is dead. And what I mean by that is that there is no sense economic sense in trying to replicate production environment that will look for staging and build environment that look like production just because the amount of data is so big that it doesn't make any economic sense. And the big companies already testing in production. And one of the interesting examples that I experienced personally in testing in production a couple of years ago, can go on Facebook if you have an account and search for this hashtag and empty envelope from China. You can find a set of blog posts by my wife about a very strange thing that happened to her. She had an account in Aliexpress to buy some stuff and suddenly out of the blue she had, she started to receive empty envelopes from China from different providers or look like this Aliexpress thing but she didn't order anything, there are no charges. Everything was clean on her side but envelopes kept coming. Sometimes empty, sometimes it had one white sock or a red tape or a piece of black cloth or a hand band or a plastic ring completely random stuff. And we were really puzzled on what's going on. We started to develop all kind of interesting conspiracy theories. And then my good friend Leonid Golnik who is the senior vice president of engineering now in SignalFX came to visit and we told him this story and he's like, I don't even know why are you surprised? Obviously testing in production. They took a subset of production database and run their provisioning test on it to see that everything works and the goods get out and this kind of stuff. And well, that made sense. And this is one of the examples of how people don't use staging anymore because staging is dead. So now we can know that data and complexity are a really big problems in guaranteeing quality and the question is, okay, so we don't trust those vendors, what do we do about it? The next question is, can we verify the update? Sometimes we just say, no, it doesn't make sense. New Java version every six months it will take me eight months to do acceptance testing on all my production environments and check that everything is correct. It doesn't make sense to update now to the next Java version by the time I will be done with my acceptance testing, new version will be out. So no, I just want to update. Or if I really, really want to, then I need to do all the acceptance tests. I need to do the time consuming verification which makes the dilemma for us as software providers because now in order to convince our users who will do acceptance testing because they don't trust us because the complexity and the data, we need to put a lot of food on the table. We need to make sure that we have so appealing features that people will update regardless of the acceptance cost. So we have a trade of now features that we really want acceptance test costs and we as users need to decide whether we should update because the features are so great we are willing to live through the acceptance test costs or we want to update to this version at all because the features are not so great and the acceptance costs are very, very painful. Can we go around this trade off in some way? Can we cheat in some way? Well, we can. In this case, it will look like update available. Do we want it? Yes, or yes, because now what will last us? Auto update is turned on and now we will get the update regardless of what's going on. And this will only work for stuff that is not high risk. And examples of that is your browser. I bet the majority of you have no idea what version of browser are they running or Twitter in your browser, that's a website. There are some versions behind the scenes but you have no, even a way to know which version is that. Twitter on your smartphone, there is a version, there is an update, but for the majority of us now auto update in turn off, it updates automatically, you have no idea what version it is. And what all those have in common is the low risk. Now for example, the operating system of your smartphone. Here you probably know the version because updating the operating system is something that requires human intervention because the risks are high. And this is where you go into this dilemma, back to this dilemma, are there enough features and do I want to take the risk of updating for those features? So now let's talk about building distrust because this is what it's all about. If we can build the trust to the level that the consumers trust that again, then there is no problem with making them updating faster and joining the features and being productive against the vulnerabilities. So what can possibly go wrong? And in the spirit of the personal stories that I started with, here's another one. Some years ago, I bought Google Wi-Fi which was called OnHub back then and that was a router from Google. Amazing piece of technology, still loving it. One of the features that was really appealing is the self-improving system. Self-improving system as known today is the over there updates. It's sitting there getting updates once in a while and actually getting better and better with time. And one day, three years ago, I was on a trip and my wife is calling me and she's like, I came back from work and our kids are sitting in the dark. And I'm like, is it the power outage? And she's like, no, the electricity is fine. The internet is down. And the kids are shouting, Alexa turned the lights on, Alexa turned the lights on, nothing happened. So they just sit in the dark. So that day they learned that there are physical switches on the wall but the internet was still down. So I called our provider, our provider like everything is good. But then I get this email and this email goes like, oops, sorry, we send an over there update to your router and it reset it to their original settings. So now we cannot fix it with additional over there update because your system is disconnected from the internet so we cannot reach it. And now you need to go ahead and reconfigure it from scratch. Now the problem is to get an email, you need wifi, you need internet. I was on a trip. So I was on a conference wifi or hotel. So I got this email, we configured it and everything worked fine. But yeah, there was this problem that update can go catastrophically wrong and over their patch cannot reach the device. And in our example, it was mild, it was just a reconfiguration of our router at home. But think about controllers edge systems on a solar array somewhere in Sahara Desert. It will take like three days to even get there in order to reconfigure it. And three days are, it's a lot of production lost. The solution to it is having previous versions saved on the device to update and rolling back if the problem occurs. The beautiful thing about this is that it's not a new idea. And this is a screenshot from, I don't know, Windows 98 that did exactly that. You remember when you had your desktop computer connected to a display and you changed the resolution. There was this warning on the screen when you come back should we revert the settings. And this is because if the resolution didn't match to the display, the display go dark and then you have no way to roll it back. So instead, there was an automatic rollback building in case something went wrong. And this is exactly how it works with all cool rollbacks today. Well, they are a little bit more complicated but that's exactly the same idea. And unfortunately, Google Wi-Fi back then didn't have this feature. Well, I mentioned again, low risks because that's a router at home but I would expect other devices to have it if they are harder to physically reach and fix. Now this is obviously an IoT example about the router and the edge device that I mentioned are from the IoT domain but I would argue that the T in IoT stands for things and things is everything including the servers that we update in our data center the servers that we update in the cloud and basically everything else. And if you are talking about fun stuff to update in IoT domain let's talk about updating cars. Again, what can possibly go wrong when you update cars? Here's a wonderful example Jaguar I-Pace released last year very luxurious obviously electric car that had a software bug in their brakes. Well, when that happens, what do you do? If you don't have software updates then you just have to do a recall. This is what Jaguar did. And the problem with the recall is that they are very costly. You have to have a network of services that will do the recall and will implement the fixes and you pay a lot of money for those recalls. And the other problem and one can argue it's a bigger one you can not actually force an update. What are you gonna do? You're gonna send them a mail like a letter and maybe in a red envelope to get their attention but in the end of the day if they are not willing to do the update to answer the recall, they just won't do it. And obviously when you have a problem with brakes that's kind of a big deal. And obviously the way to solve it is using over the air software updates preferably continuous updates. And the difference between the normal over there updates and continuous updates is this for example. So that's another software driven car and that's Tesla and one of the problems with Tesla autopilot namely is what it's called a Phantom Braking. Phantom Braking is you drive the car and the conditions are perfect. There is no interference out of the blue the car slumps and breaks. And there was a rumor that there is a fix coming out and people are waiting for this fix weeks after weeks. And the reason they didn't get this fix is because chess. You can see here an update which include this bug fixes small minor bug fixes note on the release notes and that's is the fix for the Phantom Braking. But the people didn't get it because they waited for this big feature which is a chess game to come out. And there is chess game in the Tesla that you play while your car is charging, it's fine, it's important. Is it as important as fixing the Phantom Braking? Well, depends to whom. If you use the autopilot features obviously it is more important for you. But if you are a chess enthusiast and you don't use autopilot is the other way around. Chess is the most important features. So regardless of what waited to what when you do batch updates, important features whatever is important for the user and that might differ between the users wait for non-important features, right? So for me, I had to wait for chess to get the fix for the break. Someone else had to maybe wait for this irrelevant fix for their autopilot while they're waiting for the chess features. So instead, you implement continuous updates you implement over the error updates and then you deliver the features as they come out. And now I want to make like a very short quiz Paul to make sure that you are not asleep at least. And I want to ask you how many of you are IoT or mobile or mobile developers? So let's see if we can, here we go. Are you an IoT or mobile developer? Let's see the results real quick. I don't think I have an access to the results. So yeah, Taylor, can you tell us what the results look like? So far out of the 63 people that voted 57 are not and seven are. Okay, so you see the minority of you. Seven people are the IoT or mobile developer, the rest of you are not. And that's obviously a cloud native webinar which is about cloud and server side. And you might think why are, even I waste your time talking about that. But the thing is our life as servers are developers as cloud native developers is easy comparing to the mobile and the IoT world because for IoT mobile and automotive, they don't control the availability of the target they're updating. They don't know if the computers are on or off. They don't control the state of the target. They don't know if there are implied requests. If there is something happening on the device, they don't know which version are they going to update. Maybe this device didn't get updates for last three years and the other device is in the latest version. And they don't have access to the target in case something went wrong and they need to log into it. You can always go and log into our servers and see what's going on. They don't have this luxury. So our life as server side developers is much easier and still we manage to royally screw it up. And the biggest example, it's a little bit old. It's like eight years ago, but it's so powerful that I feel that I want to share with you is the night capital disaster. So night capital was a trading firm doing automatic trading on New York Stock Exchange and they decided to roll out a new system. For some reason they decided to reuse some old APIs. And now the same APIs did something completely different, selling instead of buying, think about that way. And for some reason during the update, one of the eight servers was not updated. And new clients now send requests to seven new updated machines that did what needs to be done, but one machine contained the old code. So it did something completely different and the engineer so that it misbehaved and did something different. And what they do is, okay, we need to stop the upgrade. So they shut down the updated machine and now 100% of the traffic goes to the machine that did something completely wrong. And there is no monitoring, no alerting, no debugging and basically think about it. They are sitting and trying to debug what's going on in real time and they are actually losing millions of dollars every second they did it. The fix took 42 minutes. They lost more than $400 million during those 42 minutes and basically went out of business. And that's a horrifying story. I have a link to more detailed explanation of what happened in the show notes, just to remind you, jefford.com says show notes and there is like, it's a marvelous story and a really, really scary one. What could help? First of all, people are just bad in repetitive tasks. So instead of manually updating servers, we automate everything and this is how we make sure that eight servers out of eight are updated. And then another problem is that they did this deployment after many years of not doing any deployment to production and obviously they had no skill, they have no habit of doing that. And that was a very stressful event, things went wrong. Instead, if you update frequently, you develop skill and habit and it's not a big deal. So the chances of you screwing up are lowered. And another problem is they reuse some state on the target machine. That might be a good decision, might be a bad decision, but regardless of that, you really need to know and consider the target state when updating and reverting my require, sorry, and reverting my require revert of the state and knowing what will happen during reverting of the updates. So another marvelous example, and this is pretty fresh, that's July last year, is Cloudflare. Cloudflare went down to most of the internet with it. What happened? So Cloudflare battles, DDoS attacks and other type of attacks on the internet with routing rules that they deploy frequently and they deploy it all the time. And one time in deployment of a single misconfigured rule. Now, what happened next for me, convinced me that this world won't gonna die of coronavirus, won't gonna die on nuclear war and not even global warming. What will take this world down is a regular expression because as you can see in this example, a single misconfigured regular expression spiked CPU of a server to 100% and obviously shut it down and they propagated next thing, affected region earth. That's a quote from their status page. Some humor, definitely a needed situation. So yeah, affected region earth, everything is down because of single misconfigured regular expression. And this is weird because you expect from a very progressive company like Cloudflare to actually implement a progressive delivery because what they did is they released the bug that affected all users. Instead, you should release to a small number of users first, effectively reducing the blast of radios and then observe. And if a problem occurs, you stop the release, you revert or updated the affected users. And the most important, the most interesting part of it is that progressive delivery, it's something that the majority of us probably learned when we were like eight years old when our parents teached us how to do laundry because every detergent have this warning, always spot test on a hidden service first. And this is progressive delivery. You try somewhere to minimize the blast, to minimize the damage, and then you observe and check if it works or not. And if it doesn't work, you don't do it or you revert it or you do something else. So this comes obviously with observability. Some feedbacks are hard to trace relying on user feedback only. People yelling on Twitter is a good feedback, probably not good enough to troubleshoot the problem to the root. Instead, tracing, monitoring and logging help with that as well. Rollbacks, so now you tried it and you see there is a problem. Next thing you do, you roll it back. So fix might take time, users suffer on the meanwhile. Instead, you implement rollback, the ability to deploy the previous version without delay. Now it sounds reasonable and simple, not every platform supports it. So for example, when you are on mobile, not Google Play and not App Store allow to do rollback. If you have a problem and you wanna rollback to a previous version of the same app, you actually take the previous version, rename it as a next version and then submit it to review. And the review will take whatever it will take, three days, five days. You can flag it in urgent. If you're still not in luck and it's Friday, you won't get it until Monday. I mean, there is no way to rollback version in the mobile world today, which is a very sad state of first. So instead you can use feature flags to implement kind of rollback. You embed two version of the app and you control them with API calls and then you can turn off the misbehaving part of the app for those who are suffering. And I think the last one as the most dramatic example is MoviePass who they shut down their mobile app for several weeks to update. And that's just ridiculous because we're talking about zero downtime updates and someone go ahead and promise to come back with an app in five weeks. Probably by the time they come back, they won't have any users. That's exactly what happened to MoviePass. They went out of business after updating their app for five weeks. And then obviously the opposite of it, zero downtime over the updates of small and frequent continuous updates is the way to go. So just to summarize, continuous updates are frequent, are automatic, are well-tested, progressively delivered, state-aware, observable and implement local rollbacks. Those are all obvious stuff. Local rollbacks can be challenging. That's kind of an extra credit task, but still necessary sometimes to implement. And now what we have is we have an update. Do we want it? Doesn't matter. Automatic continuous update. And then there are sometimes no high risks or even there are high risks. We are now trust the update and then we update without fear. This is what we want to end up with. And for that is our goal to transition from Balkan Rare software updates to extremely tiny and extremely frequent software updates so tiny and so frequent that they provide an illusion of software flowing from development to an update target. And we call it the liquid software vision. This is why you have the liquid software hashtag on the bottom of every slide. And this vision is encapsulated in the book. It's called liquid software and have the concept of a continuous updates and two co-founders of JFrog, Fred Simon and you have Lyman and yours truly co-authored this book. And the good news are is that if you ping me on Twitter in DMs or even publicly, I will be more than happy to send you this book to ship you a physical hard copy of this book. Ping me on Twitter at Jay Baruch and you will get this book after this very long. Before we wrap up, I want to talk about corner cases and we believe that the majority of the software should be delivered in a liquid software way with continuous updates. Are there any corner cases? And to give you an example of such a corner case, I would like to ask you another question and that do you think that updating an aircraft mid-flight is a good idea? So you will have the poll up in a sec and you will be able to express your opinion on that. Oh, here we go. Do you think upgrading an aircraft mid-flight is a good idea? Let's see what we have. And again, Taylor, can you share with us what people say? Absolutely, right now we have 84% say no and 16% say yes. Okay, 16% say yes, that's a lot. But let's see if we can drive it even harder. So here is Airbus A350. Airbus A350 has a memory leak and it needs to be rebooted every 149 hours. If you don't reboot it, memory leak software crashes, software that actually flies the plane. So here is a hypothetical situation. You are now on a flight from San Francisco to Tokyo over the Pacific or the past Hawaii. You have like three hours, I don't know, five hours more to fly and then the co-pilot asked the pilot, hey, by the way, did we reboot this morning? Because I think we kind of run out of this 149 hours and they're like, oh no, I think we forgot. Do you think that patching this memory leak mid-flight, sending continuous update that actually fixes it would be a nice idea. You might start thinking differently than you voted and to prove that I'm not the only one and you are not the only one who thinks it might be a nice idea, here is a proof. And this is from the great book from Project to Product. You know where to find the link. Jeffery.com for show notes. Boeing 777 had a problem during the design of wheeze stabilizing the shaking during turbulence. And they couldn't debug it on the ground. So they took all the software engineers, put them on the flight, took off, went to the turbulence zone, debugged it in flight and found the problem and actually updated mid-flight to make sure that the fix helps. So this is something that is undone. This is something that definitely might be a very good idea. The secret is to trust the upgrade and to trust the update, there are very good advice in this webinar and also in the book. So wheeze that just to summarize, my name is Baruch at Jebaruch on Twitter. Liquid Software is the hashtag that you use on Twitter when you talk about that. At Cloud90 Foundation is the Twitter handle of the Cloud90 Foundation that hosts this webinar and liquidsoftware.com is a website of the book and you can learn more. And Jebaruch.com session is where you go for the slides, the video, all the links, comments and ratings and your chance to win Amazon Echo Dot if you fill out a little form. Wheeze that, thank you very much. I think we have like a couple of more questions, a couple of more minutes for questions. If you have any questions, type them into your Q&A panel in your Zoom. Yeah, we do have just a few more minutes, but if there are one or two questions, you should have time, all right? No, I think, yeah, there are no questions. Just to remind you, if you want the book ping on Twitter, I will be more than happy to ship one to you. Wheeze that, Taylor, Jeffrey, thank you very much for having me. Yeah, thanks Baruch for a great presentation. That's it, thank you for joining us today. The webinar and slides will be online later today and we look forward to seeing you at a future CNCF webinar. Have a great day everybody.