 Hey, Sonja. Hey, Janos. You know what I really find fascinating? No. Monkeys. Monkeys. Monkeys. Okay, tell me more. Bananas. No, no, this time it's not going to be bananas. There was an experiment. Okay, it's probably not true. As far as I could find on the internet, it's probably a thought experiment where they put a bunch of monkeys in a cage and they put a banana on a ladder and every time a monkey would go up, then to get the banana, then there would be a really, really cold shower. So after a while, apparently, according to this myth, the monkeys started beating each other when one wanted to go up and get the banana. And even when you started switching out the monkeys, the new monkey would then also be beaten when they tried to get the banana. So after a while, the none of the original monkeys were there and they beat each other when they tried to get the banana, but nobody knew why. Okay. What does that have to do with honeypots? This is similar to, so this was actually in a psychology book and there's no cited reference, but it's kind of like a humanized thought experiment because that's how humans behave. So I'm thinking that this is similar to how we treat SSH because everybody says, oh, you should do this and that to secure your SSH, but nobody really knows why, right? Great analogy. No, seriously, you have all these rules about SSH, but you don't know what happens when you don't obey them. You don't know what happens when your SSH gets hacked. And of course, on the one side, you know that really bad things happen because people gain access to your system and they steal all your data, but even when that happens, you don't have enough forensic data to analyze what really happened. You can just pick up the pieces and try and figure out what's going on. This is how we get to the honeypot. This is how we get to the honeypot. Yes. Okay. So SSH, like normal honeypots, you try and put up a web server or whatever and try to analyze the attacks, but SSH is really not all that. Well, I haven't really found many SSH honeypot projects. So yeah, one of the key pieces of this project is actually SSH. So let's put the pieces together. Because you basically created a whole SSH server. Don't beat me to the point. Okay, sorry. So one of the parts is SSH. So you SSH into a server with a username. And when you build a honeypot, you usually want to allow any password to succeed. So you let everybody in and then you take a look what happens. And then people can type commands like echo hello world. But usually that's not what an attacker would type. They would type something like this. So to kill the Apache web server or whatever they want to do. And we'll talk about the incentives to hack an SSH server a little later. The other part of the puzzle is containers. So you know that you can run a container, that's where you can type commands into. So why don't we put the two together? That was the basic idea behind creating an SSH server. But that's what you do anyway, right? You can SSH into a server, you can run your containers. So what's new about this? Well, so let's take a look how you would enforce that you have to be in a container. Because when you're in a container, that's a sort of a, it works like the classic jails from the from the UNIX world that it doesn't let you access anything. It just lets you access the things that are inside of a container. So the idea is that the user SSH is into the SSH server. And then we pick a container back in like Docker podman or Kubernetes that we run the shell that the user is running, we run that inside the container. And there is a there is a way to do this with your classic SSH server. And there is a reason why there is a do not try this at home sign here. It's a bad idea. So the classic SSH server, there are a couple of SSH servers. One of them, the most popular one probably is open SSH. And open SSH has a feature called force command. So when you log in, then it runs this command that you specify regardless of what the connecting user is asking to do. And you can set force command to do a Docker run Ubuntu so that the user lands in a container and they can't do anything about it. And this works. And you can even put, you know, the whole topic is we want to see what the user is doing. So you can even use Askinema to record the session. And then you can replay it and let hilarity ensue of somebody trying to do something in a container. And this is how you do it. So if you want to try this yourself, good luck. Don't try this at home because it's dangerous. I want to know what it looks like. It looks like. This is how it looks like. SSH into this server. You type your password for the user that exists on the host machine. And then you can see the Askinema output that it's starting to record whatever you're doing. And then you can do an Askinema play for the file that's been recorded. And it will replay you whatever the user did on that session. So this is the basic idea behind Askinema. You're just combining things. I want to see what it looks like. You will get there. So there's another feature in SSH where you can type hello world. No, where you can run commands on the SSH server. So that's different to launching a shell. Launching a shell is basically asking a server, hey, launch me the default shell. But it's also run me this command. And that's a bit tricky because of that part, that's then going to be executed. And we get an error because we haven't prepared our little force command to run this command that's been passed. And it's also not an interactive terminal. So when we start to fix this, we try and pass this SSH original command variable, which contains the thing that the user said they wanted to run to Docker run. However, we can do shenanigans. This specifically is redirecting the output of this echo to a file. And this looks like, okay, this should run inside the container, except it doesn't. Because of how we constructed this, this is going to run the redirection is going to run outside the container. So this means that the echo hello world is running inside the container, and then the redirection happens outside. So we can write into a file on the host system. You can prevent this if you're really, really good at shell scripting. But you see why it's a bad idea. It's not. I see everyone who knows Pearl really shine now. Yes, probably. So you can do a Pearl script that takes care of this. And, and then hopefully you will not have a leak. The problem is that the force command command that the program that you're executing is running on the host machine. And if you mess it up, then the anything that you're running is running on the host machine. So there is, you have to make one mistake to expose your whole host machine. And also you need to create the users that you want to let in on your host machine. Yeah, we don't want that usually, usually not, not for a honeypot. Yeah. So that's, that's typically something where you would start thinking, okay, how can I do this? And of course you can go to classic Linux way and do a bunch of shell scripting and hope that nothing goes wrong. But generally, we have a better way. Let's get to it. Let's get let's get to it. So there's this thing that we started a couple of months ago. It was early summer last year. And it's called container SSH. And what it does, it's an SSH server that launches containers the same way that open SSH would without the whole shell scripting part. So same idea, user connects to container SSH. And that talks directly to the API of Docker or Kubernetes or what have you. We had a couple of other things though. One of the things is the authentication server. That's a web hook that container SSH sends to a component that you can run. And this web hook will send a username in the password of the user or the public key or any other credentials. So right now, one of the features that we're working on is the keyboard interactive authentication. And your server can say if we should let the user in or not. And basically, I mean, if you check out the website, there is an example implementation for an authentication server. Yes. But everyone is free to implement their own authentication server. So this can be used for enterprises as well as for small companies. And yes, that's the general idea is that the we are thinking about providing an authentication server for basic use cases like here's a MySQL database, read the passwords from that or right now the keyboard interactive is really interesting because we're looking into implementing OAuth, so basically single sign on. And the authors of various SSH clients have been very helpful. They have helped us make this use case happen by making the links clickable and the message, et cetera. But the basic idea is that this is pluggable so you can just come up with your own authentication server with whatever and still use container SSH in a useful way. Yes. And the protocol is really simple. So we'll show that a little later. The other thing that you can do with it is a configuration server. And the configuration server can be used to dynamically configure what kind of containers to launch. So that's again a web hook and you can create an application that gives back a per user configuration what kind of container image to run or what kind of volumes to mount, et cetera. So to do this, you create a YAML file because every configuration nowadays has to be in YAML and you specify that you want to do audit logging. And audit logging is a new feature in the container SSH 04 release. And for hilarity, you can specify the Askinema format and tell it which directory to store it in. Yeah, but don't do that. Unless you want Askinema videos to show. Yeah, so it's not something that you would want to use for enterprise grade audit logging, but let's see why not. You specify a host key. You can take this from open SSH if you want, or you can generate it yourself. We'll show in a minute how to do that. And you specify the authentication server. And again, we're supplying a demo authentication server with container SSH that you can use for this purpose. But generally, the idea is that you should bring your own authentication server. You can generate the keys with SSH key jam. And then you can launch container SSH. And then this is going to happen. When somebody logs in, then container SSH will send a post request to slash password on your authentication server with a JSON in it, username, password in base 64, because you know, there might be special characters. Do you know anyone who's using those in their passwords and regularly runs into systems? Yes, you. Yes, and you as well. So whenever we type special characters, random websites just break. So in this case, we base 64 the password. So you can check the best is when you type a password and they just omit the special characters. So you think you typed your password correctly, but they just don't take it. And then you forget the special characters and suddenly the password works. That's the best one. Anyway, so with container SSH, you get your password in base 64 and you can do whatever the hell you want with it. You get the IP address of the connecting user, always useful for logging. And you get a connection ID. The connection ID is an ID we're generating. And it is it is the same connection ID that you get in your audit log in your authentication server and configuration server. So you can use that to match up various logs. When are we showing the Askinema replay? In a minute. So what you have to send back is this. You send back success true to let the user in, success false to deny entry. There we go. There we go. Here's the Askinema part. So about Askinema, let's take a look what happens when you log into container SSH. So here you can see me typing something into a console and this is already the replay. So this is an Askinema file, an ASCII cast file that we're replaying here. And this is how it looks. The format is rather simple. So there is an ASCII cast V1 format, which you shouldn't use. And an ASCII cast V2 format, which is the one that you should use. So the ASCII cast V2 format starts with, the ASCII cast V2 format is a new line delimited JSON. So it's not a JSON file. It's several JSONs and one JSON is one line. So you basically, you have one JSON in each line and this is the header. For each, wait, so what each command that the user types in is its own JSON? Yes, basically that's what it is. So what you have here is the header. You specify how big the terminal is for the playback. You specify the starting timestamp so that Askinema knows when to type the commands. So each, the typing that you saw before is timestamped and that's how Askinema knows how to replay the whole session. And then the consecutive lines are just JSON arrays. And you can see the first entry in this array is a timestamp. So that's the offset to the starting time. And the second one is if it's input or output. So you can record input as well. So hidden passwords, etc. But normally you in an ASCII cast file, you only have output. And then you have the terminal encoded output that happened on a terminal. So for example, in this case, a colored hello world. I believe this text should be read from the color codes that are there. And this is what Askinema replays. Okay, cool. So we got our Askinema. Let's watch our hackers, right? Yeah, let's get to the point. Let's watch some hackers. So we deployed an early version of this. And yeah, so this happened. Hmm. So they just tried to hack into the username. Yeah, they tried to change the password, etc. But you can see this doesn't look like a terminal, right? No. The problem is that whenever most hacking attempts nowadays are not somebody going there and typing at your console, they're bots. So they will not request an interactive terminal. And Askinema is made exclusively for interactive terminals where people are typing something. So the format doesn't really log the things that we want to look at. This is not, it's not an audit login format. And you can't really use it to record things that bots are doing. And at the end, we'll show a couple of examples that we found that are really, really astonishing of what the bots are actually doing and why Askinema is not suitable for this. So once we realized this, we came up with a second login format, the binary format that just logs everything. Well, that was the actual plan. And we just came up with Askinema, I think, because someone asked for it jokingly. So you were like, okay, I'll do it. Yes, that's true. But I was, for a while, I was seriously considering Askinema as a way to log things because it, I mean, it's a format that already exists. I mean, the new line, the limited JSON is a good way to. That's a really good way, yeah. Because the problem with logging everything in a single JSON is that you need to buffer everything in memory. Yeah, exactly. New line, the limited JSON is much better because you just need to buffer one line. So the Askinema player would have been really nice to show what the hackers are doing. But as we put it online, we realized that we can basically not use the Askinema replay for anything really useful. So binary logging is different. You can change the audit log format to binary. And that's going to log in a rather complex binary format, which we provide a decoder for. And again, you can log into a directory and then do the rest of the configuration file. So when you log in binary, this is end you decoded afterwards, because this is already the new line, the limited JSON decoded format. You see this, what this is, is first of all, all the commands that have been typed or any data that has been streamed is base 64 encoded. And we will see in a moment why. And second, we record the things that are not visible in Askinema. For example, when the terminal is resized, or when non-interactive sessions are happening. So the other thing that we can do and that we did for audit logging is uploading the data. But let's take a look at the data format. So you have an SSH connection and you assume that it's one connection, you type into it and that's it. But actually, there are multiple channels. So for example, you could open one channel to type something interactively. And on the second channel, start an SFTP upload, etc. So those are the things that we are logging. Let's take a look at a couple of interesting bits. This is fairly simple. It's a program execution. So basically, the connecting user requested to run this specific program is this cat process. So basically, the attacker was taking a look at what kind of a CPU we're running or I don't, I don't know what the purpose behind this is. The second thing is that we found a lot is SFTP uploads. So it's no longer the case that they're basically just typing the malware that they're trying to run. They want to upload the malware directly? Yeah, they want to upload the malware directly. And then it's a tar file, so they untar it and then they remove the tar file so there are no traces left of what's going on. So it's really, really involved and they're doing this in a single SSH connection with multiple sessions. So without container SSH, how does audit logging work normally? Usually use force command to run something that's doing the audit logging. Okay, so there are other systems? There are other systems that can do this. There are audit logging programs. But as we saw with the force command thing, you have to be really conscious of potential security holes. So the developers of these tools have to pay a lot of attention and it is not necessarily, they cannot necessarily correlate connecting users that are running multiple shells. They cannot necessarily correlate these connections together. But so basically, if I as a standard single user just have my server up and don't secure my SSH, anyone can come in and I don't even know what they're doing. Maybe fail to ban, bans them if they connect too many times or whatever, but that's basically it. I don't have any overview of what they're doing. Well, I mean, if you have a weak password, then yes, that is the case. So I've seen recently, I've seen a case where somebody set username test user and the password test. And it took about, it took about 20 minutes for that server to be hacked. It was a Hanipot, right? It wasn't. I don't want to talk about that. So anyway, yes. So it's really quick. Your SSH servers get hacked real quick. And then we ended up having to perform an analysis on the virtual machine image to figure out what the attacker did because most of the stuff was gone already. So the approach of actually uploading things and then untouring them and deleting the tarp would work if there was no audit logging. That would potentially work, yes. Or you'd have to painstakingly go through the audit log, but if the attacker gained root privileges, then I might be able to delete the audit log. So I'm not, I have looked at some of these facilities, but I haven't looked into their capabilities. So you can produce a fully viable audit log using force command, but you have to be really, really, really careful to configure your SSH server appropriately so that you don't have a security hole. And if you just don't allow people to SSH other than into containers, this is not an issue. Containers are not 100% secure. You shouldn't run untrusted workloads in containers. But one of the things that container SSH can do, for example, it can connect to a remote container backend. So you can run the SSH server on a different server, different physical machine, if you will, then where your container backend is running and you can drop your users into a network disconnected container. No, that's interesting. Yeah. So because we are interacting with a container API, it is basically a standard input output thing. So it's a program that's running, but the program can be in a network disconnected environment. So SFTP would still work. But when they try to attack somebody else, that would not. You can do the same in Linux, again, you can use firewalls, you can do all kinds of stuff, but it is going to be very tricky. So I found one more thing that was really, really interesting. Which was? I want to guess what kind of an attack it was. Someone tried to send you an email and spam you with free ebooks. Spamming? Yes, email no. It was this. So what the attacker was looking for is a mobile device. So a mobile phone. Basically what this is, they're trying to send SMS, spam SMS. And the reason why they're doing this is because getting a new mobile number is expensive and the network providers that allow spam SMS usually get blocked really quickly. So what they're trying to do is try to find mobile phones that are connected to servers, which has been a popular way of doing alerting with Nagios or whatever, what have you to just send a sysadmins and a short text message that, hey, your service is down. And since SMS service used to be really expensive, or you want a really reliable way, what you do is you just put a mobile phone on top of your server and you just plug it in using USB. And this is what this malware was trying to use. And this brings us into the motivation. Why would you hack a server? Well, there has to be some sort of an incentive because you're going through a lot of effort to writing these malware, etc. So there has to be a commercial incentive. And one of the commercial incentives is sending spam SMS. So what you do, you build a botnet that is just spamming out text messages from monitoring phones. And you would probably not even realize this as a sysadmin, because there's something running if you're not looking at it, if you're not looking at the audit logs. And your phone bill is paid by the company. I mean, who the hell is looking at their phone bill nowadays? You just see maybe, okay, then they're gonna tell you, hey, why is your phone bill $200? But I mean, that takes a while, right? So this is what happens when you let people in your SSH server. So SSH attacks have become really sophisticated. It's not like in the good old days where an attacker would upload a pro script and then just run that pro script. These are really trying to make use of only the basic tools and they're trying to upload their own payload. They're static. Some of the payloads are statically linked. So they're maybe written in Go or they're written in C, but they're statically linked so they can run without any additional dependencies. They're using multiple channels within a single SSH session to make sure that the connection doesn't break or they don't end up somewhere else. They're trying to really keep a low profile when attacking SSH servers. So that brings us to security. So the basic rule, of course, when you run an SSH server. Yes. Change your SSH port. It's been the same 15 years ago. It's the same now. It's the same now, but a lot of people don't do it. And when you change your SSH port to anything but 22 or 2222, you will receive only a fraction of the attacks or you could just not use passwords. Old rule. Old rule. Use SSH keys. You can disable password authentication and open SSH. Really easy. Do it. Use SSH keys. Don't use passwords. Use strong keys. You will probably not get hacked if you use a weak key, but if you have a strong key, the chances are much less. So the other thing is just don't use usernames that hackers are using. Root, test, admin, etc. Those are usernames that an attack is going to try. And if you have a weak password for whatever reason, then they're going to get in. So just simply not using usernames like that is going to lower your attack surface. The other thing is, and this goes into the direction of targeted attacks when somebody wants to hack your company, is when you want it to attack a, let's say a web shop, how would you do it? Would you go for the servers where everybody's paying attention? Yes. Yeah, of course. It's like everybody is running head-on into the firewall, right? No. You would probably look at trying to hack a developer laptop. But those are unhackable. Everyone takes care of those and pays attention. And everyone's laptop is encrypted. What are you talking? Yes. And nobody is running untrusted and npm modules on it, etc. And containers off the internet because somebody said it's good, right? Yes. And since this doesn't happen, what are you suggesting? Yes. The thing is that when you're a developer, you are more or less running untrusted code. When you're downloading a library, there might be a script included that's running automatically when you're running containers. Yes, it's running in a container, but maybe not as secure as they should be, etc. And one of the low-hanging fruits is trying to look for your SSH private keys. And if you don't use key passwords, then your key is essentially unprotected. So a payload that's hidden inside of a popular library could steal your SSH key. So use a password like test. Yeah. The funny thing is, even if you use a password like test on your SSH key, it's going to be better than not using a password at all. But ideally, you would use some long sentence for your SSH key. And then you can use an SSH agent to not have to type the password all the time. Because the attackers are unlikely to run attacks directly from your laptop because that's going to raise suspicion. They're probably going to take the key and try and run it from somewhere else. If you want to be even more secure, you could just use hardware tokens like a YubiKey. And you can put your SSH key on there. And then every time you want to log in, your YubiKey starts blinking and you have to tap the YubiKey physically to use the key. And the last one, which nobody ever does, running security updates. The thing is... See, that's the nice thing about automatic updates. Yes, automatic updates are great. Especially on immutable operating systems. That's your topic, isn't it? Yes, so running security updates is really, really important. And if you can enable security updates, my experience is that you really need to run them weekly. There is something coming up almost every week. If it's your SSH server or some other library, you really don't want to be left in a position where not having applied your security updates is going to cause a problem on your system. And yes, it's painful. So as a SSH admin, that is really a hassle. And you need to think about how you're going to do that. And as Sanya mentioned, immutable operating systems, the concept of immutable infrastructure really helps with this. So apart from audit logging, which is really fun to watch, and then you left with gigabytes of data and hours and hours and days and months to analyze what your honeypot did, what can you do with this? Any thoughts? Teach about how to secure systems by showing audit logs? That's one option, of course. The other option that was the original reason why what I wrote the prototype of this years ago. It was written in Java and it wasn't all that great. So there's a Java library that implements SSH and it's called Apache Mina. And Apache Mina implements an SSH server. The problem was that I used Docker at the time and the Docker libraries available for Java did not work well with the Apache Mina. There's an async.io library in Apache Mina and there's an async.io library in the Docker client and the two were incompatible. So it worked, but when a customer wanted to download larger months of data, then it would get buffered in memory and the SSH server would run out of memory. Sounds good. Yeah, so not all that great. Internal DDoS. Internal DDoS, kind of, yes. And then customers complained, oh, SSH server doesn't work. Oops. So the basic problem of web hosting typically is that when you want to have an advanced permission matrix, so we have a bunch of websites and then different users should be able to access different websites because the Linux permission system is rather basic and of course you have ACLs, but setting those is problematic and there are various tools that then forget to set them or actively unset them. So then user A creates something and then user B can't access it, et cetera, et cetera. So using containers to work around that problem where every user has the same user ID, so the same numeric user ID, but the folders are mounted into containers dynamically is one of the use cases that was the earliest for this prototype. And this is one of the use cases that appeared in people who are using container SSH to give people access to websites and give them tools like, for example, with in one of the cases, it's a Drupal website. So there's this utility called Drush, which is the Drupal shell and it's already there in the container and they can use it and it's dynamically provisioned for the website that the user is trying to access. The other option, of course, is Linux learning environments because you create the containers dynamically and when the user disconnects, then the container is removed as well. So it's not like you then need to clean up after your students. You can just simply let them in there and it will run as long as the user needs them. You can mount volumes for persistent data but generally the container is deleted when the user disconnects. That's pretty cool. Yeah, and later on we plan to add features that let a user keep the container or stop the container etc. So there are many things that we want to add to this but right now the containers are removed when the user disconnects. And the third thing is a high security environment. As we mentioned before, the audit logs can be automatically uploaded to an S3 compatible object storage so they're not stored locally. Which means no attacker can delete them and they can't be modified or anything else. With an asterisk, S3 does not allow streaming data because you have to finalize the output. So you upload it in bulk. So if in that time someone deleted it, it's... Yes, but the upload is... So it uploads in 5 megabytes chunks and you need to finalize the upload to finish it. But do you finalize the upload? So when the user disconnects then we need to tell the S3 object storage that the upload is complete and it can actually create a file. And from that point on the object storage, there is a feature that some S3 compatible object storage is support which is object locking so that nobody, not even you, can delete that file. You can only delete the whole bucket. No, not even that. If you set a time, this is called a worm storage. It's typically used for when you have to ensure that data is retained for a certain period of time. Then you use a worm storage. It's called write once read many. And then you can upload a file and it cannot be deleted for the time that is set. For example, three years or one year or six months or whatever. And then no attacker can delete it. And this is usually done for compliance reasons. You can use this for invoices, you can use this for audit logs, whatever you need to do to stay compliant with whatever rules you need to stay compliant with. And then of course the container backend can run on a completely different server which is an additional security measure. So even when the container gets breached, your audit logs and your SSH server stay where they are and you have the ability to analyze what happened. And one of the use cases that we saw in one company using this is they did something where they needed to provide access for their people to customer environments. And the way that they did this is they dynamically provisioned containers and of course enable whatever logging they enabled. I don't have the details. But the idea was that in the username, the person who needed to connect could specify which environment they needed to connect to. So for example, they could do their user at customer name in the username. And then that's how they connected to that environment. And everything was logged and they know that this person connected that environment. So and they locked it down. So only a specific command could be run, etc. So it's done in a way that's easily traceable and they cannot use SFTP to transport things out, etc, etc. So there are many things there that have been added to provide a high security environment. So if you want to learn more about container SSH, you can go to containerssh.io or... Debugged it. Debugged it, yes. So if you want to read more from us, various topics, then you can go to debugged.it. So with that, thank you very much for your attention and we are hoping for your questions. Thank you, bye.