 actually use a lot of ESXi servers. You would think that after using these things for 10 years, I would know how to speak, but I do not. We use these for virtualizing machines. Some of these actually run sandboxes or run dubious software on it. So we really do want to prevent these processes from jumping from the virtual environment to the hypervisor environment. We have today, we have F1-YYY. He wants to be known by F1-YYY, so I'm respecting that. And he's from Triton Security Labs. And he's going to show us how the exploits that he discovered in the, I think it was the last Chinese geek pon capture the flag. And he's going to show us how these things work. And with that, I would like to ask you to help me welcome F1-YYY onto the stage. Is it? Hello? Hello? Hello? Hello? Hello? Thanks for the instruction. Good evening, everybody. I'm F1-YYY, a senior security researcher at Chanton Technology. I'm going to present the great escape of ESXi, breaking out of a sandbox virtual machine. We have demonstrated this full explode chain at geek point 2018. I will introduce our experience of keeping the sandbox on the ESXi. I will also introduce the work we have done about the sandbox on the ESXi. Now, let's start it. We come from the Chanton Security Research Lab. We have researched many practical targets in recent years, including PS4 jailbreak, Android routine, IoT offensive research, and so on. Some of us also play CTF with Team Bloop and T-Delivers. We recently won the championship at the HATECON final. We are also the organizer of the real world CTF. We created some very hard challenges this year. So if you are interested in it, we welcome you to participate in our CTF game. Now, before we start our journey to escaping the virtual machine, we need to figure out what virtual machine escape. I'd like to ask some of you that, did anyone use the virtualization software? If you have used the virtualization software, like VMware Workstation, Hyper-V, Virtual Box, and so on, please use your hands. OK, OK, OK. Thanks, thanks, thanks. Many. So if you are a software engineer or a security researcher, you probably have used the virtualization software. But if anyone has heard the word virtual machine escape, if you have heard that, please use your hand again. Oh, oh, surprised. Thanks. It surprised me that all of you know about that, but I have to introduce that again. What's virtual machine escape? In normal circumstances, the host OIS runs on the hypervisor, and the hypervisor will handle some sensitive instructions is queued by guest OIS. Host OIS emulates virtual hardware and handles RPC requests from the guest OIS. That's the architecture of normal virtualization software. And the guest OIS is isolated from each other and cannot affect the host OIS. However, if there are some bugs or if there are some vulnerabilities existing in the host OIS, it's possible for the guest OIS to escape from the virtualization environment. They can exploit these vulnerabilities. And finally, they can execute arbitrary codes on the host. So this is the virtual machine escape. Why we choose ESSI as our target? The first reason is we know that more and more companies are using or plan to use private cloud to store its private data, including these companies. And the vSphere is an enterprise solution offered by VMware. It's popular between companies. If you are a net manager of a company, you may know about VMware vSphere. And the ESSI is the hypervisor for VMware vSphere. So it's widely used in private cloud. That's the first reason. The second one is that it's a challenging target for us. There are several explorations of VMware workstation in recent years. HACARS escape from the VMware workstation by exploiting some vulnerabilities. These vulnerabilities existing in graphic cards, network cards, and USB devices, and so on. But there has been no public escape of ESSI before. So it's a challenging target for us, and we love challenge. Then why is ESSI so challenging? The first reason, I think, is that there are little documents about its architecture. The only thing we can find is a white paper offered by VMware. The white paper only includes some definitions and pictures without details. So let's take a brief look at the architecture of ESSI first. ESSI is an enterprise bare metal hypervisor, and it includes two parts. The kernel, it uses VM kernel developed by VMware. And the other part, the user word. The VM kernel is a politics-like operating system, and it uses an in-memory field system. It means that all fields stored in this field system are not persistent. And the VM kernel also manages hardware and schedules results for ESSI. VM kernel also includes VM VM drivers, L-stacks, and some user-words API offered to the user-words. And the word user-word is used by VMware to refer the processes running in VM kernel operating system. And the word user-word means that a group of these processes. These processes can only use a limited proc directory and limited signals. And they can just use some of the politics API. For example, there are some user-words processes like hostD, SSHD, VMX, and so on. Then that's the architecture of ESSI. I'd like to give you an example to show how a virtual machine works on ESSI. The VMS process in the user-words can communicate with the VMM by using some undocumented customized system call. And the VMM will initialize the environment for the guest OS. When guest OS executes some sensitive instructions, it will cause a VM exit and return to VMM. The VMM process also emulates virtual hardware and handles RPC requests from the guest. That's how a virtual machine works on ESSI. Then how can we escape from the virtual machine on ESSI? If there is a vulnerability in the virtual hardware of the VMMX, we can write a driver or write an exploit to escape from it. The driver will communicate with the virtual hardware, and it can exploit the vulnerability. And finally, we can execute shell code in the VMM process. So it means that we successfully escape from the virtual machine on the ESSI. So the second reason about why ESSI is so challenging is that you're the world API. The VMM uses many undocumented and customized system calls. And if you want to reverse some code of VMMX, it's hard for you to understand which API the VMM is using. But luckily, we find two system call tables after uncomprising the K-Pone B-00 field. There are two system call tables we found with symbols. So this field will be useful if you want to reverse some code of the VMMX. This is the second reason. Suddenly, there are some security mitigations here, including ASLR and NX. It means that we may need to link some address information before we start our exploit to break the randomized of the address space. Furthermore, after testing, we found that there is another mitigation on the ESSI. There is a sandbox. There is a sandbox that isolates the VMM process. So even you can execute some shell code in the VMM process, you cannot execute any commands. You cannot read any sensitive fields unless you escape from the sendables either. And finally, we think that the VMM of ESSI has a smaller task surface. After comparison of the VMM binary between the workstation and the ESSI, we found that there are some functions has been removed from the VMMX in the user world to the VM kernel. For example, the packet transmission function in E1000 net card has been moved from the VMMX to the VM kernel. And if you have read some security adversaries published by VMware recently, you can notice that there are many vulnerabilities existing in the packet transmission part of E1000 net card. And all these vulnerabilities only affect workstation. So we think that the VMMX of ESSI has a smaller task surface. Now, let's start the journey of escaping from the ESSI. Let's overview the entire exploit chain first. We use two memory corruption vulnerabilities in our exploit. The first one is an initialized stack usage vulnerability, which CVE number is CVE-2018-6981. And the second is an initialized stack read vulnerability. And the CVE number is CVE-2018-6982. And we can do arbitrary address free by using the first vulnerability. And we can get information linkage from the second one. After combining of these two vulnerabilities, we can do arbitrary shell code execution in VMMX process. And finally, we use a logic vulnerability to escape the samples of VMMX and reverse a root shell from the ESSI. So that's the entire exploit chain we use. Now, let's start the first one. The first vulnerability is an initialized stack usage vulnerability. It exists in VMMX Nest 3 net card. When VMMX Nest 3 net card tries to execute command update Mac filters, it will use a structure on the stack, the physics memory page structure. This structure is used to represent the memory mapping between the guest and the host. And it's also be used to transport data between the guest and the host. Then the VMMX Nest will call function dma create to initialize the structure on the stack first. Then it will use this structure to execute this command. And finally, it uses physics memory release to destroy the structure, the face memory page structure. So it seems that there are no problems here. But if we look at the function dma memory create, we can notice that there is a check before the initialization of the physics memory page structure. It will check the argument address and the argument size. And if the check passes, then it will initialize the structure. But if the check feels, it will never initialize the structure on the stack. And finally, we found that we can control the address argument by writing a volume to one of the registers of VMMX Nest 3. What's worse is that in function physics memory release, there is no check about if the physics memory page structure has been initialized. And it's just a free opponent of this structure. So let's think about it. If we can add the data on the stack, it's possible for us to do arbitrary address free. We can add a fake physics memory page structure on the stack and then make the check feels in the function dma memory create. And finally, when it comes to the physics memory release, it will free a pointer of our physics memory page structure. So we just tried to find a function to add the data on the stack. There is a design pattern in software development. We will store the data into the stack if the size is small when we allocate some memory. And otherwise, we will put it into the heap. And we find a function that fits this pattern. This function will be used when our guest OS executes the instruction outsp. It will check the size. If the size is smaller than 0x8000, it will use the stack to store the data. And finally, it will copy the data we send from the guest into the stack. So we can use this function to add the data on the stack. Then how do we combine this to do arbitrary address free? We can use outsp instruction in guest OS first to add the data on the stack. This data should contain a fake physics memory page structure. And the page count of this fixed structure should be 0. And the page error of this fake physics memory page structure should be the address we want to free. Then we set some registers of the VMS net3 to make the check fields in the function dma memory create. And finally, we order the VMS net3 net card, execute the command update MAC filters. And then in the VMS, it will use the physics memory release to destroy the structure we've had before. This structure is a fake structure we've had in the first step. And it will check the page count. If it's 0, it will free the page error of this fake structure. So we can do arbitrary address free now by using the first initialized stack usage vulnerability. Here comes the next one. The second vulnerability also exists in the VMS net3 net card. The VMS net3 net card tries to execute command get policy. It will first get a line from the guest. And the lines must be 16. Then it initializes the first 8-bit of a structure on the stack. But it just forgets to initialize the next 8-bit of this structure and just write this structure back to our guest OS. So we can link 8-bit initialized data on the stack from the host to our guest. And after debugging the VMS process, we realize that there are fixed offsides between the images. So it's possible for us to get all the information about the address space by using this vulnerability. Now what do we have now? We can do arbitrary address free by using the first one. And we can get all information about the address space by using the second one. What do we want to do? We want to do arbitrary shellcode execution in the VMS. So how do we combine these two vulnerabilities to achieve our target? It's hard for us to do arbitrary shellcode execution by using arbitrary address free. But it's easy for us to do arbitrary shellcode execution by using arbitrary address right. So our target changes into how to do arbitrary address right by using arbitrary address free. Then we realize that we need a stretcher. And this stretcher should include pointers we can write and the size. So once we can overwrite this stretcher, we can do arbitrary address right easily. And when we first tried to exploit this vulnerability, we used some stretchers in the heap. But we found that we cannot manipulate the heaps layout stably because the VMS frequently allocates and releases memory. So we cannot use the stretchers in the heap. And after reversing some code of VMS, we found a stretcher. The stretcher's name is channel. And it's used in VMware RPC-EI. What's VMware RPC-EI? VMware has a series of RPC-Max NISM to support communication between the guest and the host. And it has an interesting name, backdoor. RPC-EI is one of them. And the other words we may be familiar with is the VMware Toys. I'd like to ask again, if anyone has installed VMware Toys in your guest OS, please use your hands again, not as much as before. So if you use a VMware Workstation, you probably have installed the VMware Toys in your guest. Because once you install it, you can use some convenient functions, such as copy and past data and fields between the guest and the host, drag and drop fields, and create a shell folder, and so on. VMware Toys are implemented by using some RPC-EI commands. And here are some examples about some RPC-EI commands. For example, we can use InfoSight guest InfoSight to set some information about our guest. And we can use InfoGet to retrieve this information back. What happens when we execute this RPC-EI command in our guest? For example, if we execute this RPC-EI command InfoSight guest InfoPoint A123 in our guest OS, what happens in VMX? It will cause a VMX hit first. And finally, it will return to the RPC-EI handler of VMX. Then the RPC-EI handler will choose a sub command to use by checking the volume of the registers of our guest OS. The RPC-EI toy in our guest OS will use the sub command open first to open a channel and initialize it. Then it will use the sub command sendline to set the size of our channel and allocate a heap memory to store the data of our RPC command. And suddenly, it will use the send data sub command to add the data of the memory we allocated before. And once the lines of the data we send from the guest equals to the size we set from the sendline sub command, the VMX will use a corresponding RPC-EI command handler function after a string combination. And finally, it will use a close sub command to destroy the channel stretcher, including set the size to 0 and free the data pointer. So that's what happens when we execute this RPC-EI command in our guest. Furthermore, there is a channel stretcher area in the data segment we can use. So this is a perfect stretcher for our exploit. Now, we've got all the things we want. We've got two vulnerabilities. And we've got the stretcher we want. How do we combine this? We notice that the VMX uses PDMalloc of GLab C to manage its heap. So we just choose to use the fastbin attack. What's the fastbin attack? Fastbin attack is a method to exploit heap vulnerabilities of PDMalloc by using the singly linked list. And it's the easiest exploit method to exploit the PTMalloc, I think. It's also the first method to exploit the PDMalloc when I just started to learn how to exploit it. Then after considering the check existing in the GLab C, we decided to free the address at the reply index of channel N. Because by doing that, the GLab C will treat this address as a fake track. And the GLab C will check the current chunk size. And after doing that, the size of the fake track is also the size of the channel N. So we can set a valid volume to the size of the channel N and to bypass the check. So we can bypass the check. Once we free this address, this fake track will be put into the fastbin linked list first. Then we can reallocate this fake track by using another channel N plus 2. Now we have a data pointer pointed at the reply index of channel N. And we can easily overwrite the channel N plus 1 by using channel N plus 2. We can send the data to channel N plus 2. And finally, it will overwrite some parts of the channel N plus 1. So it's easy now for us to do arbitrary address write by thinking some parts of the channel structure. Then do you remember our target? Our target is to do a bitrate shellcode execution in VMS. And we can do arbitrary address write now. There are many ways to do arbitrary shellcode execution by using arbitrary address write. We choose to use the job. We can overwrite the gotPLT segment. We can fake the channel N plus 1 structure first. Overwrite the data pointer of channel N plus 1 to the address of gotPLT segment. Then we can overwrite a function pointer on the gotPLT segment. So once the VMS uses this function we overwrite, it will jump to our job gadget. So it's also easy for us to do arbitrary shellcode execution by using wrap. So now we can do arbitrary shellcode execution in the VMS process. We think that we have escaped from the virtual machine of the USSSS540. We tried to execute some command by using a system called SA, but it fails. We try to open and read some sensitive fields just like password. It fails again. Then we realize that there is a sandbox. We cannot execute any commands unless we escape the sandbox either. So the next part comes to the how we analyze and escaping the sandbox. After realizing that there is a sandbox on the USSSI, we wrote some code of the VM kernel, and we found a kernel module named as VM kernel size control system. And this module implements the fine-grained checks through a system call. And it seems that this sandbox is a rule-based sandbox. So we just tried to find the configure field of this sandbox. We finally found it at this directory, etc-vmware-sec-policy-domains. And it seems that there are many different sandboxes offered by VMware to the different processes in the user world, like app, plugin, and the global VM dome is a fuel for our VMS process and for our VM. After reading that, it's obvious for us that the Vorgon directory is the only directory we have read and write permissions. Then we look at the fuels existing in this directory. We got a lot of PID fuels, just like cloud PID, DCI PID, and so on. And it's also obvious that the i9-dconf fuel is the only configure fuel we can write. Then we just, what's i9-d? What's i9-d? i9-d is open source software, and it's a super server domain that provides internet services. Then we just analyze the content of the i9-dconf. The content of the i9-dconf is here on the ESSI. We can find that it defends two services, SSH and the OSD. And some of it defends which binary will be used by different services. For example, the iSPIN OSD will be used by the OSD services. Then also, after some testing, we realized that the OSD service is always enabled, where SSH service is not. So this is the only configure fuel we can write. Then we got an idea. How about overwrite this configure fuel? Or we can overwrite the binary part for OSD. Like that, we can overwrite the iSPIN OSD to BNSH. So once we can restart the i9-d process, we can bend the show as the part the OSD is using. Then we just find a way to restart the i9-d process. We analyze the configure fuel of the sendables again, and we found that the queue system call we can use in the VMS process. Then we just use the queue hab to restart the i9-d process. Once the i9-d process restarts, we can execute any commands by sending them to the part the OSD is using. So that's the method we use to escape from the sendables. And here's a demo. Oh, sorry. Oh, it seems that I cannot play this video, but it's OK. You can find it on the YouTube, and we created this demo after the gig point 2018. We get a reverse show after executing the exploits in our guest OS. So that's all. And if you want to get more details about our exploits chain, please check our paper here. And that's all. Thanks. So I don't think I'm actually worthy to share the stage with F1, Y, Y, Y. That was awesome. If you have questions, we have microphones. You need to come up to the microphone, line up behind them, and we'll take your question. Meanwhile, does the signal agent have anything? No questions yet? Do we not have questions from the audience? There's one. Can I have number six, please? OK. Do you talk to VMware for this little heck? We have reported all these vulnerabilities to VMware after the gig point 2018. And it has been one year after it repaired it. OK, thanks. That's definitely a relief. Number one, please? First of all, thanks for the great talk. I just want to know if there is any meaningful thing a system administrator can do to lock down the sandbox further so that we can have some preventative, basically tasks for our ESXi setups, or if there is nothing we can do, except patching, of course. I'm afraid, can you repeat your question? It's too fast for me. Sorry about that. Basically, is there anything you can do as an administrator to lock down the sandbox even more so that this is impossible, or that it is harder than what you showed? OK, that was the first question. You can set the sandbox down by executing a command on the ESXi shell. I didn't put the command here. I found the command to set the sandbox down. You can find it by searching the documents about the ESXi. I found it just by myself by using the command offered on the ESXi shell. It's not documented by the VMware. OK, I will share this command on my Twitter later. Sorry about that. I didn't put this command into my slides. But would this have prevented the attack? Prevented? Would it, by doing that change, by doing that command, would it be possible to prevent the attack that you just showed? The sandbox is used to protect the VMS process. So if you update your ESXi, I think that it will be safe. Yeah. OK, great. We have a question from the internet. Yes. Does this exploit also work on non-AMDv VTX enabled VMS using binary translation? So is it more universal than just the AMD? Was it VX? Can you repeat that again? I just hear the OK. Does it also work on non-AMDv or VTX enabled VMS using binary translation? Yes. Because all these vulnerabilities exist in the virtual hardware. Yeah, you will need to use a virtual hardware in your virtual machine. So any further questions? I'm not seeing anybody on the microphones. Any further questions from the internet? That's it. Then could everybody help me in thanking F1-Y-Y-Y for this fantastic talk?