 All right. Hi, everyone. So first of all, I'm not a fan of Metal Gear Solid. I'm just a fan of cultural references Yeah, I've said I am Hinnick and I'm here to talk about last year About my last year because after I've wrapped up Python US which was in Portland Which is the Pacific Northwest of the US. I took advantage of being that far away from home and Took the classic road trip people just do usually in white Cabriolet's I went from San Francisco down to LA up to Las Vegas through the mountains back to San Francisco And a small detour to Hawaii Now this little tour plus Python took me more than five weeks Which means that I've been more than five weeks absent from work and with every photo I've posted People kept asking me how I do it and whether I still have a job Now it turns out I still do have a job last time I checked and And that job I'm responsible for almost 70 projects And it still worked out that well that I'm gonna even take seven weeks this year when I'm gonna Spend a month in Cape Town, South Africa. So let's see if that's gonna be another talk But it also turns out that the answer is a lot longer than 140 characters So that's how here at the airport of kawai with a seven dollar plastic cup of beer This talk was conceived and I'm here to tell you my story or more like the bits of it that fit into 45 minutes That I think are vital for my sleep I'm gonna touch on a lot of topics It's gonna range from DevOps see things like people are usually used for me, but I will also talk about General software engineering so there should be something in there for everyone and Each topic by itself is a talk by itself So as usual there will be a link at the end of a page with all the links for you to study If you want to know more about certain subtopic Now with that said let's get going Like every big change It all starts with your attitude you have to start prioritizing quality Which unfortunately means that you will spend a significant time on non features and That brings us directly to incentives Because nowadays everything is top priority if you ask your manager if quality is a priority They will probably tell you that this is the biggest priority ever However, how are the incentives if you don't all have time to write tests to instrument your systems It's kind of like a construction company that claims that safety is number one priority But forces their workers to buy their safety gear out of their own pocket and It's true that reliable systems do take time And if your only performance metric is to ship new features you may have a conflict of interest here However, the good news is that there is a solid business case to be made for rely up for spending time on quality and reliability and That is that the fewer Urgent interruptions you have because something is going up on fire The more you get to work on important stuff or you put it the other way around if you Build a lot of unreliable systems unreliable programs apps, whatever there is an inflection point where you just Stop being productive because you and your co-workers are busy fighting fires all day And you do not get to ship a new feature at all And we've seen it with a lot of startups that have a very fast trajectory and then like two years nothing happens Well, this is what happens And this is also how I ended with 70 projects because I'm forced because a very small company I cannot just tell other people to fix my crap. So I'm forced to build quality stuff and just accumulates This of course takes some long-term thinking Which is certainly not too common in our line of work and unfortunately that is the part I cannot help you with Yes But yeah, let's get more practical So the first thing I'm gonna defer to authority and no one less than Tony whore who gave us quicksert was quite accomplished with all things concurrent systems and What he says is the price of reliability is the pursuit of the utmost simplicity and if you prefer a Dutch sage Which at a Python conference is completely reasonable then dice grass said basically the same So you need simplicity Otherwise there will be no reliability and at this point it's important to stress that you must not conflate Simple and easy because easy solutions usually are not simple easy solutions usually a hack and Simple solutions like solutions that will serve you well in a long term. I usually not easy to find and I find a good way to approach simplicity is from the other side. So let's talk about complexity for a moment So complexity in software Is something I would say is the number number of concepts You have to keep in your mind when you are trying to reason about the behavior of a system of a program of a piece of something and Especially if you how many things you have to keep in mind when you're trying to reason what happens if you change a little corner of a system and Humans are naturally limited in the numbers of things we can a juggle both literally and figuratively in your mind and if you have too much things going on too many balls in the air you start dropping things and That's when normal accidents happen This term has been coined by Charles Perot in the wake of the three mile island incident, which was nuclear power plant incident in the United States and It's called normal Because they are inevitable in extremely complex and tightly coupled systems Now if your program is a contraption like this It's impossible to reason about changes What will happen if you play with this knob? it's impossible to tell and What breaks if the part with the knob breaks? It's I would say it's impossible to say but actually it's very easy to tell the whole thing is going down in flames That's the problem of having tightly coupled complex systems Now the irony is that if you try to make this thing safer You're adding more complexity What does it mean? The system is more likely to fail So you cannot fix a system or a program that looks like this. You can only rewrite it from scratch Now when you're talking about complexity We have to differentiate between two types. The first one is the essential complexity, which is a good complexity That's the complexity your customer or your boss is paying for you to solve It's inherent to the problem you're solving. The other one is accidental complexity This one comes from using wrong abstractions Having cumbersome deployment procedures that stop you from deploying or just using ancient and inadequate tools like Python 2. Yeah, I cannot drop this mic so so Obviously only in a perfect world you can work only on On essential problems, but still you should always keep in mind what problem I was solving here Is this essential or is this accidental? How is your ratio between those two? Now what a simple software and that of course is a talk by itself If not a conference by itself, but given we are in Italy. I really like the ravioli metaphor now Given that normal accidents happen in tightly coupled and coupled system tightly coupled and complex systems It follows that you should prefer to have simple objects that are self-contained and it have simple relationships Just like the ravioli Because if you look at it They are small. They are self-contained. Nothing is leaking out unless you overcook it and I heard in Italy There's jail time on overcooking pasta Yes, yes That's an adequate thing to do now in other words your objects your functions should do as little as possible But more importantly they should it should know as little as possible about other objects and other functions And this gives them a clear interface which you can program against and The few assumptions help you to have simple relationships because as many object oriented mentors say before Dependencies will kill you and by dependencies. I do not mean things to install from pipi. I Donald stuff is doing great work install as much as you want but dependencies between your objects and especially I really like to Not have bidirectional relationships, which are almost always bad I like to think of an object graph like an family tree if you have loops in your family tree It's a bad sign right, so Think about your object graph similarly Now all of this if you follow that gives you system components that are really easy to test because you have clear interfaces that you have to Either fake out or a mock or whatever you're doing and it's few dependencies. This is good Now bad designs staying on the pasta theme Um, it's the opposite it's back big classes Also known as God objects and a God object is an object that knows too much and it does too much And those are obviously very hard to test because just to instantiate such an object You need a dozen of other objects and you get yourself to fight accidental complexity really really fast The sad thing though is that God objects are pretty common in the Python ecosystem and the reason for that is That writing classes in Python is a bit annoying, but I will touch on that later Another mostly self-inflicted complexity is Basin your design around subclassing or as Corey puts it You will regret it and I'm not here to Hate on subclassing per se although it's I think it's well-known that I personally try to avoid it But subclassing is subject to a lot of misuse So if you think back what subclassing has been invented by its inventors for is specialization Not code reuse if you're using doing a subclassing for code reuse you're kind of Making the inventors of subclassing sad So don't do it and there's a bunch of rules around it like LISC of substitution principle the open-close principle All these things and if everyone followed those rules there wouldn't be any regret Corey is talking about here and it takes actually a lot of experience in design and Modeling to do subclassing really well now The thing is subclassing makes your software always more complex to understand. It's easier to write It's less typing no question there But it is harder to understand because you end up with namespace confusion So where is this attribute coming from can I name my attribute like this? Or is there some attribute up there in a inheritance tree that has the same name and Everything will break You have to understand the MRO your logic is not only distributed between in one level like one method calls other No, you have to traverse hierarchies to understand where your calls are going through so This is all not great, but it is about making trade-offs So I'm not saying you must never subclass, but you should be aware that that you are doing a trade-off here between easiness and complexity and More importantly, this is not where you should start so a subclassing may emerge as part of a design out of practical reasons, but it shouldn't be the thing you build everything on and So don't start projects by inventing hierarchies and As a reminder many new and modern languages that came up in the past years like go or us Doing just fine without having any subclassing at all and I'm gonna say that Swift only has Subclassing because Apple needed to ensure compatibility with objective C So Relatedly Meta classes very powerful Often overused there are a bit of pythons Monads because people tend to read into them then they find them interesting and then they've read a blog post Explaining everyone how great and simple it is, but it isn't You're again paying with complexity for mostly syntactic sugar So leave them to David Beasley. He will do something nice and depraved in his evil layer and Everything one is heavy. So just to be clear. I'm not talking about abstract base classes here Which are meta class and subclassing. It's just the way they are used So that's something different. Although I personally prefer the approach of the soap interface which using Which uses class decorators for it? But yeah, ABC's is not what I meant here Now I mentioned that writing classes is tedious in Python. So I decided to do something about that And I brought address. How many of you have heard of address? Okay It's getting better by the year so those whose arms stayed low are probably restless in your seats Dying to tell me about named apples, right? So turns out I do know about named apples and maybe I know too much about named apples And it's also such a thing that people discover them and it starts reading out that it's the best thing since sliced bread So first of all, that's a low bar because carbs are really bad for you But other than that named apples Have a history at least that's what I've been told by some ancient Python sages They're made if for the standard library if they return a tuple so they can attach a name to the tuple So it's more readable. It makes sense right named tuple tuple with name so And they're great for that But they are terrible as a class replacement Because then you end up with the tuple type in your inheritance tree, which means that you have very odd rules for equality You have accidental iteration. You can accidentally unpack your class which gives you very weird errors and Yeah bucks which Don't make any sense or you won't attach a method to a name tuple. Well, you have to subclass it You want to influence the initialization have fun implementing down during you? so but people do it because it's convenient and That's why I wrote errors because named apples if you ever wonder when to use them It's very simple. You have a tuple and you want to attach names to it You send a name tuple other than that write a class or use address which will write the class for you And it has many more goodies like validators converters default values including factories. So for the few poor Didn't know about it. Check it out and you may be asking you know Is this a serious project you should put into production and I'm so glad you asked. Yes, it is I have stickers get them Now moving on operational complexity Which is a complexity of running your infrastructure or parts of infrastructure and I think it's not controversial to say the distributed systems are hard, right? Now let's look at a very very simple one which many wouldn't even consider a distributed system We have a client that speaks to a content delivery network, which is not part of your infrastructure We have an application that has a work queue salary for example a database and a radius cache so far so standard Many run something like this I run something like that now the problem here is that every box You have there is a point of failure if any box goes down the whole thing is down, but wait, there's more Every arrow between those boxes is a point of failure to so in this case You come up with ten independent points of failures Which if you know something about probabilities they almost add up and The reason is simply that network partitions are a thing You may live in denial because it never happened to you before But it will hit you to eventually so it's better to accept your fate and learn from others They are kind of rare if you don't run at Like a huge scale, but especially if you run at scale rare occurrences are kind of common Now all I'm saying is think twice before you add more boxes and more links between those because each of them Introduces new exciting ways to fail and with that in mind. Let's talk about microservices Microservices is about splitting it up an application in many small web services There are many great reasons to do that But you end up with many more red boxes and many more red arrows between them So your big monolith that has certain annoying properties. We all know about ends up being a highly distributed system Which you may or may not be equipped to deal with well Because now now you have new decisions to make for example. Are you gonna go for this tangled mess? Which is so tangled that I was too lazy to make this slide myself and took it from Andrew Godwin Or are you gonna use a message bus? What message bus? And all the new problems that come with that like service discovery You cannot really run this without service discovery which service discovery Aggregate logging you cannot just look at the files anymore Tracing you shouldn't even think about microservices without having a distributed tracing In place which by the way, I heard there will be a talk later today about it. So you should check it out And of course all the other fallacies of distributed systems that we like to forget and that come back and bite our butts later Now what you really want at what you really need a boundaries You want to define and adhere with Between your Boundaries between your modules and your packages. That's what you really want You do not need a network between your classes Okay, and then you can have also separate teams working on separate parts if you have clear interfaces But once you have that it's very easy to graduate into Into microservices because the boundaries already there now you just put a network between them and there is a place for microservices if you need to scale out because as we know we have a thing called the gill in Python Which is kind of annoying or if you live in a heterogeneous environment like I do I need to interface PHP and pearls so Web services it is now in that way I need to stress that complexity is not the devil by itself But it's a price you pay to get things done So you should more consider it like a currency, but you have to be conscious of your budget Because the budget depends entirely on you on the time you can invest in running something that on the human resources You can throw at this problem And the price can be quite high. So if you take for example, Kubernetes Which is an amazing product? But it's super complex to run It's easy to set up. I've set it up like five times by now and it's really really easy but Do you really know how to keep it running properly? with all its HCDs flanels from ethos docker each of these parts are pretty complex by themselves so if you can afford it using your complexity budget Great get it run it put some people on it, but if you can't If you put tech in production, you don't master because it looks cool Dante has some opinions and predictions about your immediate future Now speaking of stupidity This is obviously an hyperbole because things only seem stupid in hindsight But people tend to act stupid. I tend to act stupid Just today I forgot my wallet But you act stupid for good reasons too for example if you sleep deprived because you just Came over Atlantic and our jet lagged or the baby cried all night Or you're at the busy airport on your way to a conference and you to quickly deploy a hotfix or You're hustling really hard for to satisfy some well-meaning venture Capitalist that has the best intentions for you and your future Now I have opinions on that and my opinions are the same as the one of John Allspaugh the city of Etsy I don't believe in human errors I believe that if a human causes an audit outage It's a system that failed to them and if you remember the big S3 outage earlier this year There was a great post mortem written by AWS and you should really read it expect like any post mortem is worth reading but this one is also very good and It turns out that a human used a tool the wrong way and it took down half of the internet Now the post mortem doesn't focus on that it focuses on the tool and how to prevent this from happening again Because people make mistakes and it's your job to prevent them from doing that you need to protect them now What when building tools and API's you should always assume that the operator is dragged from the dentist Consoling a crying baby or is just sitting in a boring conference talk There are many reasons to be distracted from the thing you do and still need to do it so take in an account It's kind of like products for people with physical disabilities It always turns out that those products are better for everyone and That's true with this is just as well And if it takes just one click or one API call to lose all your data Someone will make that click and will make an API call Maybe they were just cleaning their phone and just hit it by accident But they will do it. There was other data was the story of a junior developer who kind of accidentally dropped the database of some startup and He got fired and the CTO was threatening him with legal action and everything and it turned out that setting up a development and development environment Consisted of running a bunch of Commands you had to copy and paste from some document and you have to replace passwords and usernames and If you have sharp edges like this, it's your fault if people cut themselves or cut your company So in this case they have they should have fired the CTO right away But yeah, you know what it is Anyhow part of that is how you handle input and it doesn't matter if it's The input is maliciously Broken or by mistake you still have to be careful about what you let in and I fundamentally disagree with postal law here I think you should be conservative in both what you send and what you receive and This by the way was neither malicious nor mistake a mistake. It was literally the only thing I liked about them Francisco now in validator I Like to think of it of a time bomb and I'm talking about things like an invalid date string for example If you have something like that you let it in your system It starts wandering through your system and the deeper it gets the more damage Will it cause and the sooner you catch it the better can be a response. So just for an example You let the user Give you on a date string if you catch it ideally in the browser you can you can tell the user Okay, this is wrong. You have to do it like this if it reaches your ORM You'll probably show you will get an integrity error and the user gets a 500 That's not good. Other edges are of course your thunder in it or command line parsing just so Validated edges Almost I think validation is not enough I think you should always strive for a simple canonical form of your data that the rest of your system can rely on Because your business your business coach would not know what even Jason is it should just take dicks or on a native Python data types or if you know the structure of the data it should actually use a class and I find this The decision between the classes and dictionaries is actually pretty simple if you don't ever iterate over the keys of The dictionary you probably should use a class because a class is much better at catching typos and other little accidents while developing it while Dictionaries make sort of work and just shadow some mistakes for you Now passing strings around in general is also Not great Because if a function takes a string it becomes a parser It might be a simple parser, but it's still a parser and parsers are not that simple And if they screw up, it's it can be a really bad So I don't like strings in API's at all So and if you need a symbol or something in Python 3.4 later We have enums and there's also a pi PI package that will give you the same classes and they are great Now I could keep talking and talking however given that complexity leads to normal accidents and computers kind of complex right and Distributed systems are kind of complex to the second It kind of follows that failure is inevitable Everything I told you before is about minimizing risks and you should do it, but eventually something will fail So if you work in computers Failures are part of your life where you like it or not so in practice your reliability will land on a spectrum somewhere between Twitter in 2007 which had 98 98 percent uptime, which means a downtime of six days and NASA in 1969 will land it on the moon Despite a human error on descent, but the software was robust enough to compensate for The stupid mistake of the astronaut and I think we can argue that astronauts generally are not stupid and yet they make mistakes Now the problem with NASA's Reliability is that you need an actual genius to write your software a genius that will just Invent software engineering on the site while writing the moon landing software no big deal So you might have to scale down your expectations unless you have miss Hamilton on staff in that case I would like to apply for an internship So failure is inevitable inevitable all you can do is minimize risks You can prepare for it to happen and then you have to deal with it Don't ask me how long it took me to get this done So So and I'm gonna give the rest of the talk away right now a long vacation is the result of good failure containment and Solid recovery and let's shift gears here and talk about that So you've embraced failure. How do you expect failure? So if you need a system or a program do anything Reliably you then you need to monitor it. It's that simple because for all you know It's down if you need to have confidence on it working. You have to monitor it and So you need to check for outages you need to instrument it You need to instrument your system and you need solid error reporting because a silent failure is terrible I've talked about these topics in the past two years my taste didn't change at all I still love Prometheus for instrumentation and monitoring and I still love century for error reporting full disclosure David Kramer may or may not have a video of me singing Beyoncé in a karaoke bar in Bilbao But both is open source Century has reasonably priced paid plans which you can scale in a scale or two. So There's no lock in either way Now your code, how do you expect failures in your code? Well, if it's local, it's simple You just try accept and let's that's it Remotely you're lucky if it's that simple because instant errors are rare and Fortunate like connection refuse or a 500 or something like that. We know immediately. Okay, something's going wrong the worst failure scenario is that nothing happens So whatever you do remotely you have to put a timeout on it and One missing timeout in a database driver was enough to ground an entire American airline once so always put timeouts on it now if You have a timeout What do you do you've just hit the timeout? Do you carry on like nothing happens? That gives you very slow very useless requests. It's a bad experience for users because they have to wait 30 seconds to get a 500 and it's giving more load to a system that might be overloaded and for that There's a circuit breaker pattern. How many of you have heard of it? Okay, I'm gonna make it really quick. It's very simple very useful so It's kind of a local proxy between you and a remote system and it on a normal state It's closed like a drawbridge. Okay, so the requests go to the remote API the results go back simple Now if something fails, it doesn't have to be a timeout but timeout is the best example for that and it fails More than once because in distributed systems. Let's face it things fail once all the time So if you have a certain amount of failures, it will Switch from closed to open and the requests have to wait like at a drawbridge at this point It starts ignoring the remote system and it's just giving arrows back to the client very very fast So after a certain time It will send out a probe one lucky request from the client to check whether the remote is still broken and if it succeeds great We are close the drawbridge. It's closed again Everything is like before if it fails. Well, it stays it stays open and we will try again later This is very simple very effective Now I've said that adding more components to a system is usually bad However, it's if it's the same component and if you hide it behind another component It is way more reliable than your crappy code Then it might become something good. So for example, if you take something like a proxy Which I'm not I don't say lightly is very good software. It's a piece of software. It never let me down So if you put multiple of yours and protect it with a with a proxy Your reliability is actually going up and in military. They have to saying that two is one one is none So meaning you should have at least two of everything and if something is really really important You should have three of them so you never even get into the situation that you have only one of something Now this principle made the internet almost Unbreakable unless everyone uses the same DNS provider or Or unless everyone uses the same storage system in the same region Including the monitoring panel of say storage service And this works at any level at the network. You should have more than one switch more than one uplink You should have more than one server You should have well more than one data center. That's like level up game. Not everyone needs that But you certainly should have more than one backup Because if this is all it takes for you to lose your data, you do not have backups and Also, if you do not test your backups, you also do not have backups ask kid lap. They have a story to tell Now if you want to be dispensable You must not be a knowledge silo So if people ask you something regularly Write it down if you have regular tasks and standard procedures. It cannot be automated Write them down and if you have something that needs to be done in case of emergency write it down Thinking that you will remember it when you need it is a very dangerous fallacy and it had been a staple of aviation security to have To have a checklist for everything like literally everything ask any pilot and Having a contingency contingency plan you can stick to if a shit is hitting hitting the fan is Priceless you can believe me and this can be also communication. It doesn't have to be something you do It can be something you tell someone like your social media team So they don't lie to your customers that everything is dandy and they should start to reboot their computers or whatever So finally how to deal with the failure in your application? Well, your database is gone. I bet service is returning invalid data or the infamous timeout. So first Failure containment don't make it worse Because cascading failures can happen and that means that everything just melts under you even completely underlight unrelated systems and In relation to that let's talk about something very simple with a very big impact and our retries Retrace are essential in distributed systems because if as you've seen before Transient errors are completely common. So you have to take them in account, but they're also very very dangerous because if you use them loosely You can dust yourself or even worse. You can dust a third party Which can mean that you land on a blacklist or you get even into? into legal problems so What do you do you back off? But how long do you back off if you just run into a deployment or a packet got lost or whatever one second is fine If a system is overloaded and you're just waiting for your schedule to provision more servers one minute Maybe adequate and if there's like a hardware failure Maybe a switch and some poor schmock has to go into the DC and replace hardware Well every five minutes just check if if he if he or she had done it and try again The trick is to do all of it you try with a very short period which you then raise exponentially and this is pretty good But there's still a problem because if all your services do it at once You still have a lot of load in the same moment And that's where a jitter comes in a jitter is just a random number that you add to the computed exponential bike off and It will space out the retries between your systems. So you don't get this thundering heard Okay, so this is intuitive. I think so but let's dig deeper. Let's say you have a policy of three retries per client click whatever and Let's assume that the back end is your responsibility and because you read the orange newspaper page You want microservices? So let's do it So at this point one request in the front end like or like a one click means nine hits to back and a in the worst case because three times three But we are not even close to microservices let's add one more layer and You're a dusting yourself again. This is called the combinatorial retry explosion and One user click means that a C gets hit with three times three times three twenty seven 27 requests now imagine if C was slightly flaky because because it was overloaded before Now it's toast And the simplest solution the one I use is that you retry only at the top like in the client but this only works if You know where your top is Which may or not be true and then you have to do something more complicated like per request retry budgets But then you have a state that has to wander around with your requests That's not that simple anymore And this is a good example for complexity through microservices because such a problem just doesn't exist in a monolith There's a lot more to be said back pressure have it have a way to signal to the up that you are overloaded Unbounded queues do not have them it's mathematically proven that a unbounded queue is worse than no queue at all do not use unbounded queues and There's a bunch of links on my page to that Now next one insight into failures something fails You want to know what happened and why did it happen? Because nothing is worse than a silently failing backup script again ask get lab In the best case you put something like century into it But at least something email if something doesn't work and this is true in code to and If you do something like this Especially in libraries. I have a lot of opinions about you because how am I supposed to debug this? I have no chance The same is true if you do something like this what I like to call a vanity exception like an exception That is special to your very special library very special application. The error detail is still lost Don't do that instead to exception chaining So this is the Python 3 syntax. This will attach the original exception in dunder cause and You can't disrespect what actually happened if you're using Python 2 you have my sympathy and There's a function called race underscore from in the 6th library So you can use it to but really if you do not know how to What to do about an exception just let it be Chances are that the user Knows how to cope with it better than you do deep in the guts of an application So the next one is a bit counterintuitive if your application is unfit for work Like the classic thing is that your connection pool turns invalid because this database server has to be restarted You could start adding complexity to reconnect the pool but only in one thread and serve some intermediate error to your user or you just add one line and More often than not, this is just fine And it also runs assist or so sys.exe. It also runs your ad exit Handlers, which means that you get to clean up properly And this is called crash only software and it's not something I just made up. There's solid science behind that There's a lot of papers which you can which you should read And there's a bit more nuance to it like micro reboots, but I found them a bit hard in So to sum it up feel fast feel loudly a crash is always better than a hang as a user I prefer a 500 over waiting for 30 seconds and get a timeout or then an error And depending on your audience you may would you can just present a stack trace to for example Redis takes it to the next level. They have if they have a crash They show you a stack trace But because most of the crashes are caused by faulty memory They will also run a fast memory test to tell you that your memory is flawed before you open a Buck on there. I think good up So once you have that you need to focus on recovery and this is where the MTTR reigns supreme The MTTR is the mean time to recovery and if you've accepted that failures happen and maybe started writing Crash-only software it becomes much less important that something goes down It becomes much more important that it comes up fast And that's why humans cannot be allowed for the restoring of service because if I pull the plug in your DC You and your interest starts getting up and a lot of systems need hand-holding Hand-holding to come up. How do you choose? No, this has to happen on its own And the prerequisite for that is that your app Does not expect what's there like the database you try to connect if it doesn't work you try again again again Maybe you just crash if your process manager allows you to add back offs, but I just use loops and back off Now what is the secret to a long undisturbed vacation? apparently nine dollar mites and plastic cups But also more practically what you can do about it build fault-tolerant systems That recover autonomously and throw your phone into the sea This is the promised link the QR code would bring you there to follow me on Twitter buy your domains from Vario media. Thank you very much