 Willkommen zurück in Halle aus unserem Chaoszone-TV-Studio. Willkommen zurück zu der Chaoszone-TV-Studio. Der nächste Talk ist von Honkhase und Egovernante. Egovernante ist ein Projektkoordinator für Digitalisierung in der Stadt. Honkhase ist ein Cybersecurity-Konsultant. Er ist auch aktiv in der Agi-Kritis für Kritik und Infrastruktur. Er hat Spaß. Hi, und schönen guten Abend zusammen. Hi, und es ist toll, hier zu sein. Vielen Dank für das Lesen. Für uns werden wir ein bisschen über das Insidenz in Landkreis Bitterfeld sprechen. Seit ich in meiner originalen Arbeit zurückgekehrt bin, als CDU-Fraktion, habe ich oft gehört, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, dass es mir nicht so wichtig ist, Wir haben jetzt Zeit und Nervs für das. So, Rebuilding, Anhalt, Bitterfeld. Was ist passiert, was ist passiert? Am 6. Juli, ein Employee, seht ihr dieses Video, das ihr seht, jetzt auf dem Slip. Sie haben nichts gehalten. Sie waren persönlich nennend. Ein sehr clear Statement. kırdbergebe North China-Radio, was wir hier in Santaos he한다, column some want to hear in e. constitu you need to know, have a tit 4. Get used to that in public Administration. Calling ID should probably be enough. They will take care of it. The call was 6 45 and student something that is going to were executed at around five o'clock. The first encryption activity was on the sick of July at 4.30. On other systems, the encryption started at 6.30. And on all the systems at different times, the security locks and other event locks were cleared and afterwards a remote desktop session was closed. The remote desktop protocol was also deleted. So we probably won't be able to learn that much in the end, because we also we only know when the session has ended. And we have to assume that the encryption was started manually because the systems where the lockout was complete, they stopped encrypting data. But we don't have the locks. It's very hard to tell what activities the attacker did on which systems. The attacker probably moved around the systems and chose what to do individually. They used PowerShell scripts to install these backdoors. And it can also be said that the encryption was very fast. There was a lot of damage. That means that we had to assume that the whole system is compromised. And that means we have to do a holistic approach. And in the press statement, this is how we told it due to an unknown source. There was an infection of various servers in the network that led to encryption of not unspecified number of files. So we have disconnected all critical systems from the network to prevent a loss of data. And we are in the disaster mode. There was a team for special events. I've talked about that at Stiftung Neue Verantwortung. But from the information, so the question is no longer why do we have this disaster? Did we have to declare the disaster situation? It was clear that IT would have to be offline for a prolonged time. And it is definitely more than just an everyday damage situation. So if you look at just the social services, this is going to have an impact on citizens for a long time. The services for our central IT processes couldn't be rendered anymore. And we didn't have a backup. And we weren't sure if the backups we had were also compromised. So that wasn't an option as well. And the declaration of the disaster situation has a lot of consequences. Also legal implications because of people couldn't register their cars. We couldn't give them dates for their. So, let's talk about the administration of the district, the social services, where the central part, where people were calling us right away. And the task they have is to calculate the social support and also other financial payments. It's very essential to the lives of people's lives. Also the financial part, where they have to calculate and pay the money. So it also entails the services for foreign persons. So, there's a lot of things that fall into this. So, for example, endangering kids. And these are all processes that are essentially important for citizens. And they also financially, so they also have an extensional threat to your financial well-being. Also, Privacy licenses, not only cars, but also agricultural transport. And people live, they're depending for the livelihood of these licenses. In the current situation, obviously the health system is also very important, sehr wichtig, preventive Diseases. Es ist natürlich in Covid-19-Times. Es ist jetzt sehr up-to-date, es ist sehr wichtig. Ich will nicht weiter in den Detail gehen, aber die Städte-Administration hat auch weitere Tatsache, die sie erfüllen müssen. Sie müssen Prokurement, Announcements, die politischen System organisieren. Und sie stoppen nicht arbeiten. Die erste Frage aus der Federal Information Security Agency war, ob wir eine Möglichkeit haben, in den Federal-Elektionen zu konzentrieren. Ist das ein Problem? Und natürlich ist die Schuladministration viel Geld verloren, wenn wir nicht bestimmte Federal-Funds animal-Health. Es ist natürlich auch kritisch für Produktion der Food. Wenn wir schauen, warum die Disaster-Situation geklärt wurde, didn't know at that time, when we would be able to re-enable our systems and if when we do that, the encryption would resume. So, this is why we had to buy a new network equipment. And normally you are bound by public procurement law. So, if you are in public administration, you know how long this may take. And currently, when we look at how this could be simplified, in our case the declaration of the disaster situation enabled us to buy hardware without following the procurement process. So we didn't have to check whether the money is available in our plane. We didn't have to call in the procurement people and without accessing the data it would actually not have been possible anyway. So we created an emergency network and three places and so we could just ad hoc ask external companies to do support tasks. So every time so everyone who even knew how to spell IT was called in to help us. Other colleagues who were originally in the school administration part of they were and I was in the law department. So when I was declared the technical head of operations at this point, I wasn't alone anymore. I had a lot of people at my disposal, IT people and people from the administrative processes and the experts in their respective fields. And from day one, I did didn't call them Sachgebiet IT, but just IT. And are they called IT? I was asked. So this was something that we were able to to clarify very quickly. In the first hours and the first days and weeks it was about accessing meeting rooms, communicating with external people, getting hardware, accessing people at night and very profane things like food for employees and all of that depended in some way on the declaration of the disaster situation. So we're coming to the crisis team and incident response. We we kind of collided with that already. There's external teams that we have to interact with. The crisis team the decision makers and there were several in-house teams. Und so we did have some collisions there. The those that stay longer in the public administration and those that want very fast decisions. Leads to behavior. And also there was a new head of the administration. Just three days after declaring the disaster situation, which led to a lot of discussions that wasn't really productive. We could could have done without. As someone external, the Ministry of Finances was the Federal Information Security Agency. The third NORD was involved. Someone from a professor from Hochschule Hartz. And everyone was very helpful in their specialty. And the technical team leads one and two. They were involved with the one was involved with the getting the hardware back to back running. And number two was involved with the administrative processes. Und even though that sounds like this was involved, that this was interconnected, the that doesn't work because the administration works so they keep on working. The telephone worked, fax worked. So the doors were still open. So people were still requesting stuff from the administration. So I already mentioned that citizens had existential issues with the district. So people maybe bought a car or crashed it. So for some reason they have to remove their registration. That's possible. But registering your car is only possible where you live. This is very strict. And this leads to these administrative processes in in other districts. They couldn't help us because the data that they needed to register a car wasn't there for the other district. And also there's time limits. So if you don't react to a request, it may lead to the administration agreeing to it by default. And these lists were flexible in the beginning. So there was a continuous discussion going on between the different directions of the rebuilding effort. And in the meantime, there were quite a few different decisions that had to be made, for example, would the administration go into the cloud? Do we have to involve someone from the Workers' Council? Do we need to have the forensics team continue to work ? There was a question whether we could outsource certain services to Datacent. There was the blackmail letter where we were considering how to go about the data that was involved. And there were a few more challenges that will go into later. But let's go back to the organization where a few things that changed Lee like right after the CUT case on the last slide I had. We basically put together all the IT members from the different organizations and for example, the education administration workers were also involved, as well as the IT security officer had to be changed. We didn't create a new position for that. And there were certain restrictions around who was eligible to be a name for that office. So we needed to find someone. There were several people who were interested. But the state is involved in educating those candidates. And we need to change their contracts so that mit Bordmitteln möglich war. We had to make sure that the things that we that we were asking of them, that we have the budget for that, that what we did was within the limits. And we were obviously making decisions for the future here that would have a long lasting reach after the event. So this is not just for the short term. So we need to implement the IT basic protection. So it's a a terminus Technicus in this way. And they need to create it in a way that they can afford later on. And here we also need to have the political foundation in place for this. Like we didn't get any approval for new positions. So the people that we needed we needed to get permission for. So quickly, let's go over the rebooting of the business processes. This is what we had to say again and again in the beginning. There was no. There was no big business to be had here. We needed to make certain that the suppliers knew that. And it was supposed to rebuild and not created separately in a new and it needed to be reintegrated into the old network. So we also needed to make sure that we utilized the capacity of the providers to the fullest. The good news is after prioritizing a lot in this all works again, we cut short a little bit. Becoming to the balance sheet for the basically the bill that we got in the end. It was roughly two million, quite a bit more than the ransom that they were demanding. But with the planned IT budget you had for the next year have to be covered. So obviously all our planning was for naught. And there were obviously a lot of conflicts now between the different officers within the organization. And we have a trust issue now with regards to the digitalization. Because a lot of people are now saying, why didn't we just stick to paper? This would have been easier. Now we kind of lost our Internet, which quite a few people are frustrated about. They went to the Home Office just so they had the technology like their own private IT, because we couldn't provide them any machines and we had to allow them to utilize their private IT to have BYO. And now that they educated themselves on working this way, it's kind of difficult to go back to the status quo beforehand. We obviously, we weren't in the worst position. We got quite a few other organizations that reached out and promised aid and helped us out with expertise and also with capabilities, even the Army helped out. But it also means that now we're kind of responsible or we feel responsible to initiate the knowledge transfer and we're kind of assuming that we were hit quite at the right moment. We were the first, but we're definitely not going to be the last. And it's paramount that the knowledge transfer is now happening. And on the side, we've already answered so many questions. We've been to conferences. We have been to hearings. It's quite obvious that there's a lot of interest. And now I hand over to Honka. He kind of pissed me off so hard that I have to make sure that he also is part of the talk now. OK. I just want to be sure that my taxes that I have to pay are put into good hands and to good use, you know, and the problem with critical infrastructure, especially around state and public sector, is a hot topic. And we've already seen quite a bit of problems in the past here. And this is only going to be worse. So the dumpster fire you have on the left is kind of emblematic. So often I have the feeling that there's nothing that I can fix. We have to burn it all down and then start it fresh. So the question is what do we really need? What we actually truly need to have is a way of cyber resiliency. I'm a cyber grandpa, I'm allowed to say that. We need resiliency for the cyberspace of especially of critical infrastructure and public sector is part of that. The ego went on to the earliest said that there are actually quite important business processes in the administration that cannot just stop for a while. We're not just talking about fun and entertainment. This is this is criminal conduct and we're dealing with gangs of organized crime here. And what we truly need, how do we how we turn the ship around? How do we get the resiliency that we need? And I definitely want to name the Mayors and other leading members of the administration. They need to have experience with doing crisis. Let me just throw in Ataal, which was there was a flooding this year in Germany. That was quite heavy. They they had the same issue. They didn't train for the problems or the exercise that they had to do. And then they were surprised by what happened. So training is important. And honestly, we can't have business processes still depending on Windows 98 in 2021. We actually have seen this in Berlin and it used to be on Windows 98. Und even after we looked at it, it still was and that's a fail. So it needs to be digitalized from start to finish for the end. This has to be the goal of the administration. And what we definitely also need, but that is not there yet, is there are just no directives for the public sector. So there's a paragraph 8a of the DSI for eight of the critical infrastructure sectors. There are very detailed instructions about what they have to fulfill. And with public sector, it's basically not there's no directives at all. So it's basically open field and everybody can do what they want. Something has to be done around this area. The current government has to do something about that. They have to wake up and put out some security directives that have to be followed. And what we also need are so in the response teams on the county level. We've already we've been run over by this kind of attack. And it's not enough to have like half an a person working on that full time. That's they've never been involved with all of these business processes. They don't have the experience with that all time. You cannot do the work. You cannot do the coordination between the different officers. They basically look into the into the the rulings and the legislations. And that's that's about it. And they don't even know what they're who they have to interact with. So we need information collected somewhere. And so this is my passion here. The hour we call it the cyber health. So that it's basically like an NGO that would help out in the in times of cyber incidents that are affecting the public sector, especially on a large scale and that help out with transferring information. And we're working together with the public Office of Information Security and together with the KS Computer Club. And we were brainstorming how to get the volunteers involved. And how would we be able to help out when we have a largest cyber incident? Einfach mal viele Mote optimiert wurde und lahmgelegt ist offline und will be for many more months. So what we want to be able to do is be able to to create an ability to act. So what we are planning to do is not to help the critical infrastructure people or their financial gains or to have their production facilities new and stylish. No, it's only about being able to supply critical services to the citizens, which means in public administration that the administrative processes work, that people can contact the officers with their problems and their requests, that social support is paid every month, so that you don't just have to say, well, we just have a ransomware problem currently, you can't get your money. That doesn't work when people really need this support. So improving IT security isn't enough. We will have to do that anyway. We don't have to incident response and crisis response is also very important on the basis of volunteer helpers. We do have enough know how in the community, if this works on our terms, then we want to help. There's not weird security companies that the Federal Incidents Response Team is joining in. Maybe you don't even have to call the army. The army should be the very last, the absolutely very last in the chain. They shouldn't be calculated. They should be, in the best case, we never request the help of the army. When the shit hits the fan, then we need the army. But not in these kinds of situations. And this is what the Cyber-Hilfswerk is supposed to be do. And this is why I was kind of forced to into this in the expert team and I will try and get this started. So we don't have to burn everything down, but maybe it's just we have to be very angry. All right, so we are in the behind the scenes category, best of yelling at customers. So if anybody didn't, it doesn't show up for the lightning talk, then I can read for some emails that I've got. No, it's just a joke. There were some situations that in their just in the moment, when they were happening, it was very unbelievable. On the one hand, obviously, if someone really wants to help, wanting to help is not necessarily helpful. So if people think that it's a good idea to save USB-Sticks to their files to their USB-Sticks and take them home with them as a backup and then bring them back, this is going to be a problem. What we had several times is IT people were asked some computers were supposed to be brought online again and it's bad if you're not asking the people responsible. And that leads to computers not being available because they are somehow in use now. And it was also interesting in the technical leadership team, whoever was it like. So after ten minutes, someone turns on the microphone and you just say no. It's technically not possible. So, even if we really want the administrative district to go back online, there's some order that we have to keep to. Some people wanted to use that as an opportunity to use digitalization projects. And we can't start with an empty license plate database, because otherwise people will end up with duplicate license plates. That's not possible due to liabilities. And we did have the CEOs talking and one was really unhappy, because we were asking technical details and he said, well, I'm going to tell on you and you're not going to get any more companies and even though these decisions were made based on technical expertise, things like people wanted to really tell us how to how they can migrate to the cloud, which is really not helpful, because this is not what we're going to do. And losing the trust is one of the problems we have there. So if you have multiple partners, then when they were talking about how they would continue their work, they didn't want us present. So it's very important to have an IT-Project-Manager in-house and the district, just like any other office, needs to have this kind of personnel, if we don't want to be tricked. And no computing center wanted to have our data. Don't send us our data. We were doing all the services. So it's also great when a company tells you, well, we talked about this price, but now it's three times more expensive. And they don't talk to the IT people, but directly to the head of the district. They were talking to the male people outside the sniper that will take you offline. The moment you go back online with the female colleagues, they were saying something like the stalkers are waiting for you. So, very weird. And when we others said, don't use open source software, because hackers use open source software, these are the people that attacked you, a very difficult way of framing it. But you may also have luck. For example, you get a cheap office. You get seven managed services for 10 percent of the prices. In the first year and the kickoff, they wanted to have a TV team present, which we didn't accept. And then the service providers that we were working with for many years, we were asking for their offer. So it's great, if they put the licenses right onto their offer as well. So, the administrative district isn't the victim of the attack. In the expert team, I believe, we have a great way to and we also have a research project. So we now have the awareness and there's a research project being launched. And we do have digital Geminois. And there's an open source play. So, that's a good thing about this Catastrophe. The administrative district, the offices are really due to do their homework. So this is now the time for questions. Genau, fragt uns Dinge, ist die Ruhe in Antworten. Ja, sehr schön. Vielen Dank. Thanks to that. There are a lot of questions already. So a lot of the questions revolve around if you can devolve some more details of the attack. We don't know that much, unfortunately, because the log files were deleted as well. We can't really give that much in terms of details. In general, with incidence response, if you try to help the victims after a ransomware attack, the attackers, again, this is organized crime. This isn't just your general criminal. They enter these systems and they check exactly how much you can afford to pay. And the administrative district, they also leaked 200 megabytes of data to tease them into, well, maybe you want to pay something. And they will actively destroy online backups or encrypt them as well. So for a long time, access credentials such as SSH keys are being looked for. They are emptying the syslog on Linux systems. And when everything has gone and most of it has been encrypted, you can only look for what's remaining. There's a lot of attack vectors that could have been successful, which one, in effect, it's almost irrelevant. You want to be able to rebuilds. You want to be able to provide the critical services again. So it's often the case that you can't really tell exactly. And we also assume that the attacker has been in the system since the beginning of the year. And we also know of a much larger Data Lake, das doesn't improve things, but that's how it is. Ist bekannt, ob tatsächlich Daten abgeflossen sind? So is it actually known, if data has been extracted? Yes, data have been copied off 200 megabytes leak, but there have been copied off a lot more. There's been a statement. So 62 gigabytes of data bait were copied. We don't know exactly what these 200 megabytes were leaked. And obviously we know what it is. There's protocols from the and all in all 62 gigabytes were lost. That is actually pretty not that much. We also know of other incidents where terabytes are copied. So you're still in Reconstruction. So one of the questions is is there data material that has been lost forever? All of the mail server data by the environmental office from one of the locations in Bitterfeld and also the Internet has gone. And everything else, we are still looking into what data might be missing. We know that data from the environmental office is gone and also the complete mail server and also the Internet is no longer there. Eine andere Frage. Another question. How big was the ransom demand? If that's something you can double? No, we actually published this 500.000 Euro in Monero, so relatively cheap. Other attackers request much more money, but they didn't get any. Hanka, you gave some recommendations where volunteers could help out in the public sector, so they don't have to bring in the army. Was reaching out to the military actually called for, or is there something that is being done? So this doesn't happen again in the future. Well, whether it's called for, Igor Van Ante can probably judge better than I. I don't know the details. I haven't been involved in the internal things regarding this incident. She may think it's appropriate, but my position is that this doesn't happen again or this isn't just a default to call the Bundeswehr. This is why the Agi Critis wants to create this cyber relief service, the Cyber-Hilfswerk. And when a critical infrastructure is attacked, then we from the community help as volunteers and we come there and repair the service, the critical service together with the people who do that professionally. And then we leave once it's done. This is the aim. So we don't have to call the army again. They should only be the most extreme cases. Das war der erste Fall. OK, so I think that it was that was the case, it was the first case. It shouldn't be standard, but in that moment we weren't able from a technical perspective to help ourselves. So the technical services that they rendered for us. As of today, almost all of the districts are on this level of preparedness. Oh, I hope it prepares. Yeah, hope dies last. Schönen. Äh, eine andere Frage. Thanks. So, how far are you along in the reconstruction process and how are you going to improve on the situation? But this starts with the internal agreements, with the awareness. Die IT-Sekuritie-Offices. Ähm, und wir also haben multi-level Security-Concepts, das doesn't allow Workstations, Desktop-Workstations, to have all the administrative privileges. The passwords expire at some time. Und regarding the rebuilding, the Active Directory is restored. The administrative processes will go online one after the other. And these different offices will be able to have their desktop PCs back online now. And they hadn't, so we have four or six offices, where every employee has their own PC now and not just a general emergency PC. And do you also use Hacker Software such as Open Source? Oh, yes, we do. Very good. So, in particular, a question about the Active Directory. When you did the forensic analysis, was there a Golden Ticket attack you saw there or? I cannot say that. I don't know. I can say something different regarding Active Directory. So we had to look up whether every employee is actually, I mean every entry to the Active Directory is actually related to a real employee or if someone new showed up during the attack. And that was the first thing that we did. And this took a lot of time. So we had to kind of created a list of all the colleagues. Absolut, ja. And, ja, it's, it grew historically this list. So we had to check it. Sonst gab es Backups, zum Beispiel in einem externen Rechenzentrum. Did you have any backups, maybe even external? No, I mean, yes. Some processes had backups and we didn't know it sometimes. So the privacy officer was kind of irritated sometimes. So there were backups, but not all of them were an external Computing centers and now we have to check check them for compromise. So we assume that 80 to 90 percent of our data will be restored with a lot of manual work, of course. And I just want to add backup is nice. Restore is a lot better. Everybody says backup, but nobody wants backup. Everybody wants restore. And you only get that when you have a very structured backup concept with grandfathering Models, with rolling backups, with offline backups. You have some 160 processes. Ja, if you have 160 historically grown administrative processes, it isn't just one backup on one tape. And you put this into your bank locker and then you have this golden medium that you bring back when the attack happens. That's just not how it happens. So a backup concept that includes restore within a acceptable time frame is a very complex task. OK, ich kann mir vorstellen, das ist jetzt in den Wiederaufbau. Ich kann mir vorstellen, dass das wird ein Teil des Wiederaufbauungsprozesses und der Improvement sein. Ja, sicher. Ja, absolut. Exakt. Wenn wir nur auf das System sind und im Ziel sagen, wir haben jetzt wir haben Administrative Processes, die nicht in der Target-System sind. So wir haben jetzt die Daten, die wir haben, zu reimporten, wenn das Finalsystem in den Platz ist. Wöchstest du davon? Na ja, alles Gute, ich bin Spaß am Gerät, das gehört ja. Ich wollte nur die Fun mit Ihren Devices. Das ist Teil der Chaos, oder? OK, vielen Dank für Ihre Insights und für das Publikum und die Digitalisierung, die Sie hier befinden. Vielen Dank, Hongkaz und Ego-Venante. Vielen Dank. Vielen Dank und bis zum nächsten Mal. Sehr schön. Als Nächstes um einen. Thank you for your attention, also from the translation booth. You just heard the talk Rebuilding the Administrative District Anhalt Bitterfeld by Ego-Venante and Hongkaz, translated to you by Castel and Tribut, that's me. If you have feedback for us, please use the hashtag C3Lingo. Bye.