 Talk today about cloud forensics. My name is Alexander Adamov, and so I do security writing at Mirantis. And before we start discussion, I would like to ask our guest to introduce yourself, please, like, shortly in a few words. Who are you and what do you do? Yes. Hello, my name is Kemindar. I'm Chief Security Officer at Network, obviously not for shares. So, but that's me. Hello, my name is Anders Karnsson, and I'm a university teacher at BTH in Sweden. I also am instructor in forensics at European level, where I teach police instructors for BKA in Europe. BKA in Europe, Paul. Okay, so before we start the discussion, I would like, we will state the problem, but first I'm interested, how many of you are familiar with the forensic and know what forensic is? Okay, cool. So before the meeting, we had a couple of discussions with the forensic expert with police, and just to figure out what is the state of the problem, so in which way they do forensic like in real life, and have they done any forensics in clouds in virtualized environments? And when we mentioned, like we are going to talk about cloud forensics, they just open eyes widely and ask, is it possible to do forensic in clouds? So I would like to ask Anders Karlsson, who is a trainer and he's the author of forensic courses for the police. Can you please explain in a few words what forensic is and what children we have in cloud forensics? To start with someone maybe just shortly, forensic. We have a part different side of forensic, and initially it's like to put on the Sherlock Holmes hat and start finding small digital fingerprints inside the computer or, and there we have done lots of work the last years. We run live forensics, memory forensics, and the thing is that the difference comes when you take this evidence or these digital fingerprints to court when you have solicitors or lawyers to try to break them apart. You have made a hypothesis about something that's happened and it could be fraud for bank fraud or whatever or a hacker or a simple word puking at Facebook and you want to have some suit for a lawsuit. Do your paperwork stand up in court or is it just paperwork? Yeah, that's what we're struggling with the forensic and the huge thing about cloud is that, okay, we know how it works with single computer, image forensics, network forensics, everything, but when we come to virtual machines and we come to cloud, we have, yeah, it is a very complex world to put together a chain of evidence. I think that's, it's extremely complex. Yeah, just to add a few words to under speech, I've been doing an incident investigation which is quite similar to digital forensics. So in both cases, we need to collect signs of attack. In digital forensics, we need to collect digital evidence. So we need to also to keep in mind the data we collect from the clouds, they should keep integrity. So we do not temper the data so we can bring it to the court as it is. So I've been doing an investigation several times and usually I see there is a lack of investigation strategy in organizations. So when you, for example, ask to bring information about the attack, data about the attack. So for example, in ransomware attack, people like administrator usually send you encrypted files. Yeah, that's actually fine to have some private photos, I don't know, encrypted with AES-256 but this information is totally useless. So this is like a big job to collect evidence like logs to find out the traces. So how this cloud environment or corporate infrastructure was attacked. So what is the breach, what information was stolen. So to do that, we need to like administrator IT. IT staff needs to do a lot of work and it would be nice to have it automated. So before continue, I also started searching if we have any project about forensics in OpenStack. And I found that we had a project for forensic project for OpenStack Essex I believe in 2012 and this project was called Frost. So here you can see the diagram of the project. This wasn't production project, this was like scientific research and this is a project shows proof of concept or forensic tool for OpenStack. As you can see the researchers, the authors of this project, they implemented extra functionality to compute service and also to dashboard. So this is actually what we want to have. We want to have a button on your dashboard and if incident happens, we need to click the button and to collect the digital evidence from our cloud. And but I checked the repository, the Git repository and actually it hasn't been updated since 2013. So this project was like a dead burn. So when we talk with the police and with the forensic experts, we figure it out that they still do forensic in conventional ways. So they analyze and infrastructure, even virtualized infrastructure in a regular way. So they take the dump of the memory, they take the data from storage, they try it, if the traffic is recorded. So it's okay, but in most cases we can get the cashed traffic from the company. And so they do not use like any specific tools for virtualization. So they, for example, the most popular tool is the volatility framework. It's a toolkit to get the dumps of RAM memory, volatile memory. And it is good because it supports Windows, Linux, Mac OS and I even did forensics for Android devices. So you can install volatility tool on Android device and take the memory dump. So that's actually the problem. So we don't have like any functionality, any project in OpenStack. And so that's why I invited experts, Kim from City Network, just to share experience, how they do audit, how they get logs in clouds. So just to share this information with you about best practices, how to do forensics, please, Kim. Yeah, thank you. The thing is in the cloud, we have this problem, like when the police comes and say, we want the physical storage. And we say, but it's a customer VM. So you need that storage. They don't really understand it. And that's the problem when the police say, we want your entire self storage. And I say, no, you won't get it. We want it. Okay, here are 10 racks. Go ahead. They don't really get this in the cloud. So the thing is with the logging in the cloud, you can, if you run OpenStack, be forensically sound by the logs themselves in OpenStack. The problem is today that the logs needs to be in debug level. So they're insanely huge. They're insane amount of logs. And most of the logs in debug level will be pure and simple nonsense. But that's what required by the law today to be forensically sound. This is sort of a problem I would hope that OpenStack would solve in time with maturity to have a specific security event logging that you pipe to a specific log setting. So you don't need to have debug level because you get crazy amounts. So it takes a lot of our backup storage, actually. That's pure logging. Then you have this other thing when you have a specific case against a specific tenant. I can't send them all the logs. But if I don't, then they're not forensically valid because then they're filtered. And that's not, it should be unchanged, unfiltered, these things for a log to have a forensic sound. And then I need to give the authorities the opportunity to take and review all my logs that they are actually forensic to sound and then filter because I will breach confidentiality against all my other customers. If I give them debug logging data, except for the one tenant that that's actually they have a case against. This is also something that I think needs to be addressed in trainings, in IDs, a process. How can I be legal? How can I reach a neutral authority that can actually do log filtering in a forensic sound way? The same as to gather evidence in any other way without compromising the integrity and security of the other tenants. That's... As an extent, we have two types of problems. In forensics, we have legal issues. So for example, we cannot share the logs, which includes not only the compromise of tenant information, but other tenant's information, which in which we, as you said, violate confidentiality. And the second thing is about technical issues, like, as I said, to have some magic button, how to collect the evidence in a matter of minutes, better in a matter of minutes, because from my experience, when we did the investigation, when we provide some SLA for the corporate customers, we had four hours to analyze the threat and to come up with a report showing, explaining the payload of the attack of the malware and how it has been penetrated the network and what information was stolen, leaked, and how to remediate servers and infrastructure. And so this is just imagine, like, four hours, and it's better to have, like, automation tools to reduce this time when incident happens and still we do some incident mitigation. And that's the challenge. And for example, in some cases, like in cases of targeted attacks, like we discussed the case with Roac. Roac is, like, what, yeah. It's a switch military supplier and they had a breach where they lost information for over two years. And when they come up with it, two years. I mean, that is why we have zero days bug. They are living long time today because the zero days are very well hidden. We have the, we can say heartbeat or whatever the bug you want to know, but they are used and the people who have them, they use them. In this case, they use some of those to tap information from Roac, military supplier, during two years and we had surveillance over what happened during six months and then they pulled the plug and disappeared from the internet. And that was the case. That was a long case how to make incidents looking for what has happened, recreate the story. Yes, so we can see like the time between incident happens and it was discovered and mitigated can be like about two years. And it's not the worst example because if we take a look at the history of target attacks, we can see that some APT's footprint, we can find traces of these APT's like during the last like five, seven years. So these APT's were active during such a big amount of time, they were discovered. So the idea is to make this time shorter and we need to provide both like discovery methodology and the second one is how to collect incident information, incident data, cloud digital evidence and to add also to Kim's speech. In this first project in forensic open stack project which was unfortunately, it's not supported anymore. In this project, we had proof of concept which collects logs from the nodes, from compute nodes and these logs, they were hashed and frost build the hash tree. So they, but the thing with this hash tree, with the logs exiltrated from the nodes that were filtered according to four specific tenant. And as we figure it out, it's not, it's not, it can be used like a digital evidence because the temporary logs, we don't get the raw logs, but we get only some of the information from the log and it's not digital evidence. So both technical and legal issues we see in this aspect. Yes, and then we have these issues with privacy that you can demand an expectation of being forgotten. If you're no longer a customer with us, we will erase the tenant. And if that comes back two years later, then we have really little chance of helping in that case because if they erase the tenant, all laws state that I have to properly erase the information as well, so. Yeah, and just one more thing to add to you, to finding digital evidence like we had a case like last year with the Cryptolocker attack to Telecom company. And so we find out the malware which was used to attack the data storage wasn't encrypted. Cryptolocker infected desktop computer and using Active Directory user account which had access, read write access to the file storage. Cryptolocker encrypted all the data on storage. So we try to help this company to recover files and to decrypt the files. And the thing that some of the keys that can be used to decrypt files, they were sent in a special chicken request encrypted chicken request to the remote server. And when we started investigating, if we can get like this URL history from the internet provider, we couldn't get it because by default, internet provider just, this option was not enabled. And this just destroyed our efforts to help this company. So it's essential to have login enabled, to have traffic cache, URL request. So this will help dramatically to investigate the case and to collect digital evidence to actually to find traces. And about the key that I understood about RUAC, the thing was that the experts from SIRT, from Citron and SIRT, SIRT is a computer management team they're responsible to analyze government attacks. So they find out that the RUAC network was infected at least since 2014, like two years ago. But the thing that they limited with the lifetime of LOX. Yeah, they had just, if for example this company store LOX for a bigger amount of time. So they could find out the exact time when the network was compromised. And based on this, they can know what information was leaked. So since what exact time this information was recorded and sent to the attacker. So it's also essential to keep in mind, and like for how long are you going to keep your LOX? Of course, the longer the better, I think. But in recommendation, I found out it's like around two days or three days, like recommended log-log rotation time frame. Yeah, this depends a lot. But from these logs, we're talking about the pure infrastructure logs. We have a bunch of different legal requirements, all that differ from each other. You have the privacy demands, it says that you have the right to be forgotten. You have commercial demands, it's that credit card transactions has to be logged in a specific way in a specific amount of time. And it all depends a bit on the customer requirements as well. If you have medical records, you need to have a transactional log for them for a long time. We have insurance companies who sell insurance. These logs are as long as insurance is valid. So if it's a pension, your log, and you start when you're 18 to save pension, well, that's a log for 80 years then. So the tenant filtering is important, but that's where you get indirect conflict with the courts because it can't be filtered. And it's not a sensible commercial idea that one insane logging requirement for 80 years can rule all the rest of them. That's why we, I think, need more direct approaches when an intrusion happens. It's hard to go back long in time, you can't expect it. I can if I get a good report of our immediate intrusion, freeze the logs and save them, but not for a long time back. That's then there will be filtered. Just add a little bit about what you said about the ransomware. We tried to track those ransomware and what they had was that they have information page on hacked web servers, and if we could get the information, the logs from those web servers, in this case it was a, help me, it was a go-kart driver in Spain, I think. Yeah, it was like someone's personal website which was hijacked by criminals and they put the command control server on personal website. Yeah, it was running on a hosting system, but the thing was the web servers were totally looking correct. The owner of it didn't know anything. He had only a couple of PHP files on his web server, but could we achieve information from that web server, then we could know a little bit more about our trace to trace it back because we had some IP addresses, but most of it was hidden behind the Torah servers. Yeah, and again we face legal problems like we cannot hack this website and to sinkhole it, just to investigate the case because this is just someone's personal website, and by the way I usually show to my students a movie called Untraceable, it's by 2008, it's about FBI agents, and they investigate the case with some maniac who kills like a cat and then people online and the people die faster depending on the number of visitors. So the more visitors the website has, the faster the victim dies. And when FBI agents start investigating, they find out that it uses some flux, fast looks, dynamic rule of proxy boards to hide the real backend server with the web page. So when you, for example, ping the website address, every time you ping the website, you get a new IP address. And then they say, okay, this is, we see like the most of computers, the most of proxy computers, they allocate it in Russia, like, and they cannot do, like they can do anything, nothing with another country data sets. So that's another problem. Can you please, like in the end, do some recommendation, how to enforce insert investigation process and forensic process, what best practices you would recommend? Yes, actually, I would always recommend the same as with the ops process. You should treat the security process the same as you do an ops. So if your machine breaks down, you will get some sort of alert to the ops team that your service down. They will come, try and fix that, try and get it working again because you will lose a lot of services. You should have the same with security events. An authorized login has happened. Why? It should immediately go trigger an alert, wake someone in the security team and the security escalation team up to go and check what has happened. So anomalies, you need to have a system that looks for anomalies. That's the really important part. We are extremely predictable in some sense. So when something strange happens, that's when you shouldn't trigger the security response team to investigate. And it should be 24-7 watch for this. SMS alerts waking someone up or something like that. You all have it in the ops section. So you have on-call teams there. So there's nothing strange with that. You should treat the security events the same way. I would say even with higher priority. Yeah, I can only add that you have to be prepared and think about what information do you have. Maybe you think that the information you have is not good, but the bad guys out there, they are not one or two. For instance, the malware we worked on last year, we think about it as it is at least developed by a 20 crew professional developers who developed this ransomware. And they developed it from January. We followed it back. So it started in January with a rather, it was breakable for us. But in September, it was unbreakable, hidden by the tour servers and everything. And it was extremely obfuscating code where the code was mixing between executing in data area encryption. So be prepared, think about what you or your customers has for information. And when somebody knocking on the door, it don't have to be the police. It can be, as I said, someone who's suing you because they lost the information or someone made blackmailing on your customers because they got the data out there. We had all of those websites lost information, dating sites and, yeah. Yeah, and from my personal experience, I would say that we had great presentation about holistic approach to security in security track. And I would say that firewall with laser beams will not protect your organization even if you have like inside security system, outside security system. So in 99% of cases that I investigated, the thing was the attackers, they penetrated social engineering techniques. They do not need to use zero-day vulnerabilities. They do not need to hack your web server or your firewall, bypass your firewall. They just send email, efficient email to your employees to like accounting office or to HR office who are not like security aware. And so, and they just put in attachment PDF file. And that's all, just employee opens the attachment PDF. PDF has explored JavaScript and the infection is done. So that happens like in 99% so this is the big chain is stuff. So I would recommend to do some trainings like some drills, for example, to test how many send some like fake efficient email with like some attachment and to check how many of your employees open this attachment from this fake email. And Anders mentioned like we need to prepare for the attack. It's not matter if but when you will be attacked and to be prepared, you need to develop infrastructure like tools to have tools to have security experts to have emergency response team. And this will like make you stronger in case of attack. And every organization, even military organizations, government, White House, US department, they were attacked last year. Successful attack using APTs and with social engineering technique like fishing, spare fishing and watering hole attacks. So there is no like 100% protection. There is no firewall with other beams who will protect you completely. So you need to train your staff and you need to prepare for the incident when the incident happens. I just mentioned what you said. I think 90% of you would click on an email where you got a new email about this conference. From OpenStack, yeah. From OpenStack conference, everything looking exactly the same. But the thing was the PDF, that PDF are fixed. It's included something. And all of you are gonna click on it. Yeah, and the thanks, you're welcome to ask your questions and I would be happy if for someone, I don't know, we'll start initiative to continue this OpenStack project. Even though it's not legal, the data collected cannot be used as digital evidence, but it can be used for internal investigation. Yeah, please, Kim. And this is actually why we should all go to the cloud services because the thing is you need to have your systems isolated. They need to be interoperable, otherwise it would be useless. But you still need to put them up in isolated environments more and more because they are going to be hacked. But that should only affect the system itself. Thanks, your question please. We have set seven minutes more. Yeah, please, no idea. No idea, I didn't check what this tool is about. Yes, this is the problem of course with the cloud systems. We can only be totally responsible for the logs of the infrastructure and what the virtual objects give us in logging system. We cannot do logging on the virtual machines themselves. That's the tricky part. So most of the evidence has to come from there and that would be the sort of golden ticket for us. This is why we need a quick response because I can freeze images from virtual machines through the cloud infrastructure. I can isolate and I can put them in a separate network containing with port mirroring but I need to know about it quickly to do that before they erase it. So you as a customer needs to report this quickly for all anomalies and this is a part of what I think it's not mentioned enough when you talk about cloud, public clouds especially. We have a security response team. I bet you that not most, lot of you have a 24-7 security response team. How many of you have a in-house security response team on call 24-7? Yeah, good. That's excellent but that's expensive. We have one that you can utilize. This I think is what public clouds can contribute a lot with. You don't talk that much about it. Like security as a service. Yeah. More questions? Logs that are used to identify anomalies not from an operational standpoint but from a behavioral standpoint. API users, we do know how the services are supposed to talk to each other. And you can quickly see for instance if the rabbit MQ has in some way been compromised and I want to use the rabbit MQQ to send specific instructions that shouldn't come from that, that shouldn't. So we can isolate because we can determine logs that behave naturally in a predictable manner and these things we need to look for is these anomalies. And then you can have a separate alert system, logging system for storing anomalies. You could even have with the automation possible today a freeze function that makes snapshots of all affected tenants with anomalies happening. And we need not to forget to calculate hashes for snapshots when you make it. So we need to keep a hash value in the cloud and then you can calculate the hash of the data received on forensic storage, for example. This will provide integrity of the data. More questions? Thanks. More questions? Thank you very much. Thank you experts. Thanks. Thank you.