 Hallo! Letzter Talk im vollen Saal auf der GPN 21. Letzter, erster Tag. Es spielt sich schon an wie ihn. In den Content Notes zu diesem Talk steht Contains French and the Speaker has a thick French accent. Könnt ihr mich verstehen? Gut, ihr kommt mit österreichisch im Akzent, sondern dann habt ihr überhaupt kein Problem, Garfield Airlines zu erkennen und macht über OMG GOV H BBQ. Viel Spaß! Merci beaucoup, Monsieur Leirach. Wo ist das Publikum? Erstens, natürlich. Es wird viel frischer Content sein. Komm schon, es ist ein zweites Slip und ihr lacht schon. Das Publikum wird heute sehr leicht sein. Und seit dem Publikum ist voll, hoffe ich, dass ich nicht viele Leute habe. Aber ja, alles ist in French, die offiziellen Dokumente sind in French, die pressen Artikel sind in French, die Memes sind in French, und so weiter. So, erstens. Lappéritif. Was sprechen wir jetzt? Wir sprechen über OVH. OVH ist ein frischer Unternehmen, von dem in 1999, by a guy named Octav Klaaba, er ist Polnisch, und er ist inspiriert von seinem Studiumname, Oles van Herrmann. Es kann also mean, we host you, in the original version, that was used on the FTP-Logging. They are doing whatever your standard cloud services are doing. Bermeter, VPS, Domainname, Public Cloud, ISP are they doing as well, and many other things. They have nowadays a lot more servers than what they had at the beginning in the north of France. They are having, around the world servers. But what interests us is of course Strasbourg. Yeah. Now for a little bit more context. They got 70 terabyte global network capacity and 34 data centers. But what's interesting is Strasbourg. As you can see, they are here in the portory section, just next to the German border. This is important for later. It is important for later. It is important for later. It was at this very point an old Arcelor-Mittel emplacement. They are still there. They are at the end of a port dock, so you have to drive 1.2 kilometers. I did that two weeks ago to a scoot. Had to see the security guard telling me, what the hell you are doing here. Yeah, I know. And you know the camera everywhere and I have seen you since the start. Yeah, I hope so. So yeah, I did a little bit of scooting for this talk. Just a little bit of Arcelor-Mittel. I don't know what they are doing. They got two times 20 megawatt power lines there. This is also important for later. And yeah, as you can see it's an industrial place just two or three kilometers away from the center, the European Parliament, European Court of Justice, lots of things European in it. And since it's in a portory sector let's say there's a lot of companies that have to say it. Going to fire. For example here Scholl Recycling. 50 Tonnen of plastic waste going into flames. Again, three seriously injured. A grain silo has exploded. Again, polyurethane fabricant. Again, malt fabric that exploded. Again, a rain of aerotherminals that is doing logistics in a porotonum de Strasbourg. By the way, I checked their website and they are usually guys, high TTP, in the year of the current year. Come on. Guys, I'm in. I fucking swear. But you think I'm done? I'm not. Yet another recycling company that went into fire. So, that was it for a l'apéritif. Now let's go to Le Pla principal. The building of OVH itself. As you can see at the day of the burning we had four different, five different building. The fifth one wasn't operating. It was still in construction. And what took fire was the SPG2 and it leaked slowly on the SPG1. We have another plan here. We have here the two power transformator and this little shiny boat we'll get to that later again. It's a very powerful boat. So, the plan this is coming from an official document, the other slide too. I'll give you all sources at the end. As you can see you have at the top of SGB3 a diesel tank of 10,000 liters under SGB5 they're the same. And according to the documents that I've seen, there's a little bit more of underground fuel storage so there's sort of 30,000 liters of fuel. Yeah, we'll get to that later. There was already an incident in 2017 in November. What happened? Both power lines didn't have electricity anymore. Bad. What's worse? The diesel generators haven't started. So, what happened? All four buildings were down. And they had the power turned off and they were taken here. You can't really see the Overline. But basically they promised to shut down the servers in SPG1 and SPG4 and migrate everything to SPG3. This wasn't the case. It has been 3.5 years between this announcement and the fire. Nothing has been migrated from what I've seen. So yeah, let's get to some plans. Sorry. You have here the SPGB1 das Prototype. As you can see it's just artillery containers stacked one over the other. And they took a canvas sheet to make the cooling tower effect like a chimney. I'll get to that. It's this plan. Another one of the container. And here you have what's inside. You can see the servers so. You have on one side the water cooling and on the other side the smaller side there's fans to let the air go out. And here you can see these buildings on SPG2 stacked one on top of the others. As you can notice there's wood on the floor. The end building still in progress for SPG2 that was 5 floors. So in just another side quest. This news article happened 3 days before the fire. They announced they wanted to go into the stock market. It has been moved on 6 months later again we'll get to that later. The Klara family owns 70% of the stocks and it started at the stock exchange at 18.5€. They had a range between that and 20€ they went for the lower. But as you can see I'm not a volunteer. It was really good. So, now the spark. The fire itself. It was detected at 35 Minutes there was an alarm the security station. They noticed it very quickly the guard enters the energy room the batteries and UPS on the number 2 energy room, sorry. It's located on the ground floor of SPG2 thick black smoke. 2 Minutes later the entire building is evacuated. You had 2 other technicians working there. It was a night service. They quickly called the fire brigade. With group leader pump one turntable ladder and here the famous pictures of the security cameras. Yeah On the left you have the batteries this lead and on the right you have the uninterruptible power supplies just just for the record there's 800x30kg of lead batteries that's 13kW So yeah the fire brigade arrives and they requested to shut down the electricity at Electricity Strasbourg basically the one maintaining the power lines. Thick black smoke appears at the ground floor and electrical arcs are forming at the door of the power room. You have 1m wide flashes and very very loud disorienting banks and they deployed our water lines while waiting for the substation to be powered off. Yeah The emergency power supplies the diesel generators were turned off because this time it was working. You really shouldn't but this time it was working and of course on the other building too the fire brigade 47 minutes later attempted a recon on the power room where the batteries and the uninterruptible power supplies were the thermographic camera records 400° on the room of course they couldn't get in and at this moment the fire brigade just learned that's actually lead batteries So yeah that's the CPU monitoring from a brave willing server that did its best until the end You'll notice it's not 2 o'clock yet but yet the server was still running at 2 o'clock so that's one hour and a half after the first spark almost 19° well it was probably more than 19° to be honest So yeah Strasbourg Electricity Resort arrives on the building at the campus in the first turn of the electrical substation because the substation belongs to Oveache not their problem as far as I can recall the fire of course in between was spreading and it was ruled out that this was not possible anyway because it was too close to the fire this engine is made to cut off a bit above the other distribution substation so yeah last a bit the temperature on the second floor saturated temperature safe sensor excuse me and then they remotely finally after one hour 15° they remotely turned off the electricity at the substation further away but the server is still powered on at this moment one hour 49° exactly the german lightsteller informed that there was a thick smoke cloud smoke coming in because of course it was just 15 meters away from the german border and there's a lot of batteries so as you can see this substation from what I understood belong to Oveache but yeah let's recap we got the uninterruptible power supply with rectifier inverter static bypass and the batteries we turned this generator off we turned the power supply off it was still running why? well of course you had batteries so the servers were still running on the lead batteries there was no emergency procedure and technical procedure to easily shut off the uninterruptible power supply which has added difficulties to the fire brigade of course there was of course you understood no emergency shut off button and it was difficult to know when exactly the batteries were empty but it was estimated that the batteries lasted 20 minutes so until 2h14 in the morning then let's go on there's still power and it's now totally burning and the fire of course is propagating to SGB1 and the boat Europa is coming to help as I said it's a big boy and according to the expert note the expert note the building was secure electrically at 2h30am then you have a nice little pictures from the fire brigade of the Baron you can see the Europa boat here and SGB2 a little bit lit and SGB1 in front slowly catching the fire regarding the cloud itself it was 200m high and they measured officially nothing significant the problem here as you already noticed is that the concept of the data center itself only helped the fire as we say it was literally working as intended yeah we can see Europa in action and through fire holes yeah they used water and foam they made again measurement because lead, electrical components and some stuff water and foam so then finally 3h28 the power is totally off it's almost 3h the fire is finally under control at 6h45am but it's still impossible to enter the building since it was so hot it has destroyed the structure and it's unsafe to go in the fire is totally extinguished at 10am and the fire brigade intervention was over at 1813 so yeah the little boat as you can see it pumped a lot of water and was really helpful it's actually a cooperation between the port of Kiel and Strasbourg it belongs to France and Germany European cooperation and yeah we had of course the two fire trucks with both that amount of water which brings us to a total of 20.000 liter per minute that's 171 bastards per minute according a bastard with 140 liter 40 liter according to the fire brigade of course the other problem was as you can remember on the map it was at the end of a dock which means it was also at the end of the municipal water supply so that wasn't enough so contributing factors to the fire of course there was no automatic fire suppression system inside the building there was no inert gas nothing however the detection has worked perfectly they noticed it quickly and acted quickly that I must say it there were fire doors but they were kept open I'm not making this shit up this is in the governmental report about this incident the fire suppression again would have helped control the temperature and the progression of the fire but since they were known it was a fire festival electricity there was no easily accessible emergency power off switch they had to let the battery die down they had to cut everywhere outside there was no button to turn it off electrical stop station to close from the fire making it impossible to operate as I said before the diesel generator started despite the fire the overage technician had to manually stop them and the battery were giving energy until they died there was also an added delay for electricity so that they find the power supply up ahead and turn it off so yeah in between let's look at the monitoring of overage 10h after the fire so that was SGP1 2 3 and 4 everything working fine guys not fun aside what the hell was going on why was it burning on the first place truth is there are two hypothesis and we don't really really know what's going on they're still looking into it the first hypothesis there was a lot of technicians call to the company who produced the UPS there was a lot of switch to static bypass it was unexplained and yeah it happened at the same time in the UPS and on the batteries the batteries itself were routinely maintained the last maintenance of the UPS was the very same morning the second hypothesis is a peaking humidity that was detected in elven 15 the day before technically can zoom it on it that's again the official governmental document they're miked this is conditional have been liquid the cooling system installed in proximity or it is related to the maintenance or a short circuit in the lead battery due to the humidity could have make it happen however lead batteries are very robust and the batteries had no monitoring but the lifespan recommendation of the fabricant were respected as I said the investigation is ongoing so we don't really know what's going on then impacted services according to netcraft 3.6 million websites over 464.000 distinct domains were taken offline after the major fire more than 80% of the uvh ip addresses were down und unsurprisingly french domains were affected of course sorry for the wall of text but 2% of the french domains and a few thousands top level domains so what exactly was affected lots of non-profit small online shop personal project personal servers it was maybe there governmental organization were hosted there Diffenseur des droits defender of rights everything regarding discriminations and so on numerous town websites the open data portal of the french government was down transportation services there was a website of numerous transport websites that were down including the airport of Strasbourg we had a virtual e-learning environment for lice in germany realschule gymnasium und so weiter so this means you couldn't see the notes of the students what homework they had to do and so on we had various support organizations your local handball team and so on webradios stopped at streaming because they were hosted there too but yeah Centre Pompidou museum was also done just a few examples if I have to list everything we're still here tomorrow morning den wordpress-plugging vprocket und imagify were down encryption software the website and github then local github chess portal b2b electronics news website an Algerian bank was down and certification agency for vehicles in the united kingdom were down i kept the best for the for now there were also cyber command and conquer servers hosted by OVH that were down so cyber malware was reduced that's the director of the global research and analytics team named great at kaspersky lab yeah so apt including charming kitten apt 39 related to iron services behamuts cyber kind group and ocean lotus from vietnam down but germany was also affected and it was a sad day for memes very very known website in germany was down and I find it extremely ironic that it's hosted in france when one of the most used tags is it's wrong french to be french as it's french to sign yet your servers are in france so yeah we're talking a lot about down servers but when your service is down what are you thinking about recovery plan and the first thing in a recovery plan is backups and yeah it's in baguette what basically happened is that uveash sells their backup options for 5 euros per month a lot of small businesses skipped on the expense and since it's the cloud they hosted it there and they lost all their data what did they do to get their website back that's right they rushed on google archive and archive.org to get their home page back perfect example we have here a jewellery store la pique cour that sends orders everywhere in france the website was gone no backups of course the customer databases contacts well they were gone there's an estimated 2000 euros financial damage and for this uveash proposed 30 euros as a commercial gesture you are losing but that's 6 months of hosting and in the article she said i will go somewhere else with a backup solution so she learned the lesson and even if you had backups there were companies who were a little bit more tech savvy and thought i'm going to take backups great idea the backups were on the same site so they were a little bit pissed off let's say it they went to the tribunal and the court answered uveash has to pay 100.000 euros they asked for 6.5 million the judge ruled uveash made no mistake or fault but they had not respected its part of the contract stating that backups were physically isolated uveash will appeal the ruling there's another company they asked for 330.150 euros they got 150 euros at the tribunal the backup server survived but uveash restarted the servers and they haven't warned the customer what happened there was a con job it started and it rotated everything erasing the latest data can't make this shit up so the customer got his backup server back but it was empty absolutely empty there's they also appealed the judgement and there's something like a class action from 140 companies that are in the same case and they are asking for 10.000.000 euros that's a story for later of course it's still in progress but even if you were smart enough to have backups well not really sometimes you're really not trying your backup sometimes something missing they got everything except one thing the gamer progression it wasn't backed up offsite the players so it was bad to start with the counterrector from level 0 but everything else was back so yeah not so bad and then there's other cases like ocean supermarket from what I've seen they had a contingency plan and they were back in a few hours so yeah there was a second fire I voted what the hell the one container was affected with 300 batteries and 25 kilo each they were unused and not powered on and yet it went on fire they used foam true employers felt unwell due to the fume but it's okay and they promised to shut down SGB1 forever but hold on Krabba you said that in 2017 I mean I mean yeah and this time he's really saying we're not restarting it ever and as I said I was on site SGB1 was torn down and if I recall correctly SGB4 too yeah so how you learned your lesson they took very early smoke detection apparatus for the SGB5 and automatic fire suppression inert gas they got a new dedicated data center for backups and the price tag of this bad boy is 30 million euros so as I said before they went into the stock market and they started with something like 18 euros in 50 cents they had to postpone for 6 months their entry into the French stock market carcant because of obvious reasons but yeah how it's going son I took this tonight 6 p.m. since their entry they lost 50% so under 10 euros I'm not a Wall Street bad treat hearts but it's not looking good in a portfolio yeah so the document itself as I said it's from the French government they have an office dedicated to industrial risk depends on the ministry of ecological transition it's online you can google it very easy to find second one is the fire brigade which had a report of 3 sites giving hot feedback on what's going on what went correctly what went not correctly and yeah the rest is online dedicated website for sysadmins and netadmins everything was linked down right and yeah I promise to keep it short so that's it from me already there's a lot more to talk like a technical issue government's slight planification legal pages to build a data center they even took private jets to bring technicians between Lille and Strasbourg back and forth that is coming back quickly and yeah I got a call key it's much better and of course it's a CCC event I got a deck too so that's probably enough thank you public you were quick so unfortunately we are not doing Q&A for this talk I'm very sorry but he is available so buy him a drink of his choice and if you have further questions on how to burn down the building or how not to grab him for a talk ask him questions he's available please give another very warm round of applause Garfield Airlines merci public