 Hi, we are at Hiroshima from Entity and Yan Du Boa from Suze. In this session, we'll talk about how to run container D and KSOS or macOS machines. So, why do we need to run containers on macOS? 2022 is the year of the Linux desktop, so we could just run containers on Linux as well as on Linux servers. But some ordinary developers still need macOS or Windows laptops mostly for running some proprietary software. So, it's useful to run containers on macOS or Windows. Windows already has WSL2, so you can just run containers there. But so far, macOS didn't have a similar functionality, and it was hard to use a Mac for running containers. The motivation to run containers on macOS is just for the deep environments and the test environments. I don't think macOS is the best fit for running production servers. So far, the most popular solution to run containers on macOS was Docker desktop. It's capable of exposing the host file system to the containers running in Linux VM, and it's also capable of exposing the local host ports of the containers to the local host of the macOS host. But Docker desktop is proprietary, so we could just install Docker and Kubernetes inside plain old Linux VM, perhaps bare mini-cube. But this isn't really straightforward. Some of VMs such as VMware and Parallels are proprietary. VirtualBox is free and open-source, but VirtualBox is specific to the Intel architecture, and VirtualBox will not support Apple's M1 chip, which is based on ARM architecture. QMU is free, open-source, and supports M1, but still not easy to expose the host file system to the guest, which would run Docker and Kubernetes. And it's not easy to access the container ports from the host, so we made a new solution. It's called LIMA. LIMA is similar to Windows WSL2, but LIMA is made for macOS hosts, especially made for running container D or macOS, but other applications can be used as well. As in WSL2, LIMA is capable of exposing the host file system to the guest and running the ports automatically. And LIMA comes with built-in integration for container D and nano-CTL, which is CLI for container D. LIMA is an acronym of Linux machine, but we originally designed LIMA as a container D machine to mimic experience of former Docker machine for container D. But we exchanged the scope immediately to cover other use cases as well, so we chose the current name. But LIMA still focuses on container D and K3S. Let's take a look at what is container D and how to run container D with LIMA. Container D is a default standard container runtime that is used by most of distributions and managing services of Kubernetes. So far, Docker was more popular runtime, but Kubernetes 1.24 dropped support for Docker, so adoption of container D is growing significantly. Some people think that container D was solely made for Kubernetes, but it's not to be used. Container D even provides Docker-compatible CLI. It's called nano-CTL, container-CTL. The usage of nano-CTL is just same as Docker. You can use nano-CTL build to build an image and nano-CTL run to run a container. You can also use nano-CTL compose app to run containers from docker-compose.dml. And it comes with lots of calling edge features that are not yet available in other container engines, such as on-demand image distribution technique called Resifling, which enables running a container a head of completion of Pring's image. It also supports PTOB image distribution using IPFS and encryption of images using OCI-Crypto. Nano-CTL also provides optimized networking stack for root-less containers. You can't run container D and nano-CTL directly on macOS, but Lima provides built-in support for running container D and nano-CTL on macOS using a Linux VM. You can just use Lima nano-CTL build to build an image from Docker file that is located on the macOS's home directory. And you can use Lima nano-CTL run to run a container. And you can access container port just as HTTP colon slash slash localhost from a web browser running on macOS. I'll show the demo. So this is macOS. And inside the home directory, I have Docker file. And I can build an image using Lima nano-CTL build like this. And I can run container using Lima nano-CTL run like this. And I can access port using HTTP colon slash slash localhost from macOS like this. And you can even run AMD64 container on an ARM64 machine such as M1 with Lima nano-CTL run dash dash platform equal AMD64. And vice versa, you can even run an ARM64 container on an AMD64 machine. And you can also build a dual platform image that works for both AMD and ARM with Lima nano-CTL build dash dash platform. So next topic is K3S. K3S is a lightweight keyboardless distribution hosted at CNCF as a sandbox project. K3S adopts container D as the CRR runtime. And of course, K3S works with Lima 2 with these commands shown in the slide. So Lima CTL starts template colon slash slash K3S to create Lima instance for K3S. And Lima CTL shell K3S cat etc launcher K3S K3S dot yaml to create cube config. And you can just use cube CTL to connect to the K3S cluster running inside the Lima. Lima was originally designed only to support container D but now we expanded the scope so you can even run Docker inside Lima too. By Docker, I mean the open source version of Docker engine. It lacks proprietary GUI but the fire system mounting and port flooding are supplied by Lima. So basically it's very similar to Docker desktop except that it lacks GUI and even Portman like this. Let's take a look at how it actually works. The hypervisor part is just a vanilla QMU. We chose QMU because it's free open source, covers both Intel and ARM and has a lot of features. We avoided to use up both virtualization framework because it's proprietary and lacks some of functionalities such as UEFI and real estate bios. This is not a huge deal when you already have VM linux kernel and init rd but we wanted to just use UEFI and grub so up both virtualization framework didn't satisfy our requirement. For fire system sharing, we have reverse SSHFS and BrutIO9PE PCI. Lima B4 1.0 uses reverse SSHFS. Reverse means that macOS works as an SSH client but it also works as an SFTP server using the reverse SSH connection but Lima 1.0 uses BrutIO9PE PCI which is also known as BrutFS to replace the weird reverse SSHFS. Let's also answer BrutIO5 system technology called BrutIOFS. BrutIOFS uses BrutIO2 but BrutFS is different from BrutIOFS. We actually wanted to use BrutIOFS because it's faster than BrutFS but BrutIOFS is still not available for the macOS version of QDMU. We could use upload implementation instead but it's proprietary and lacks other functionalities so we still stick to QDMU. The next topic is port forwarding. Lima transparently makes the guest ports accessible as a local host. This is accomplished by watching events in the guest and running SSH-L to write SSH for the ports. For regular TCP ports we just watch slash proc slash net slash TCP in the guest but for CNI ports we also have to watch the IP tables rules because the CNI ports do not appear in the slash proc slash net slash TCP. That's all of my part. In the next part Jan Dubowa will talk about further details of networking stack and the community. Thank you. QDMU provides DNS to the virtual machine by randomly choosing a single nameserver from resolve.conf on the host. This works well as long as it picks a working nameserver. I was surprised to learn how many Lima users actually had a broken nameserver configured along their working ones resulting in random DNS failures in their virtual machines. This approach is also insufficient in many enterprise environments where you have to configure one or more virtual private network connections which in turn may provide additional name servers that resolve names for internal domains. QDMU DNS also cannot support MDNS because it does not listen on an external interface. MDNS is very useful for ad hoc collaboration between machines on the local network. And finally QDMU DNS does not provide the names from Etsy hosts on the host so those aliases are also unavailable. We have addressed these limitations by creating a host resolver inside the Lima host agent. This is compiled with C bindings so it has access to the macOS resolver running on the host itself. So it gets access to the default nameservers, it has access to the VPN nameservers, it uses the mdns responder and it uses the aliases defined in Etsy host just like any other application running on the host as well. This host resolver is then exposed on random UDP and TCP ports using the DNS protocol and it's only exposed on local host. When host resolver is enabled, which is this by default, we use IP tables rules to map the default QMU DNS server through the host gateway to our host resolver. This makes sure that the QMU DNS server is completely hidden inside and we know that container D or K3S or anything else will only talk to our host resolver. The host resolver also allows us to define additional static names and aliases. For example, we define host.lima.external as an alias for the host gateway address. So this can always be used to access the host from inside containers. Any changes in the networking setup of the host is automatically reflected by the host resolver. So if you connect to an additional VPN network, that automatically becomes available inside the Lima VM without having to restart it. Proxy servers are another frequent occurrence in enterprise environments. We try to set up proxy environment variables for the VM automatically. We start by looking at the system network settings and look for the network interface with the highest priority that actually has an IPv4 address assigned. This is normally the Wi-Fi or the Ethernet adapter. So if the proxy requires a password, the password has to be included in the URL. We will not retrieve the password from the keychain and then expose it in clear text in an environment variable. This has to be decided by the user how they want to handle this. The common way to do this is to run a local proxy on local host that provides authentication internally and can be used without username and password from local applications. The system settings can be overwritten by the lima.yaml configuration file for the virtual machine, because sometimes a user wants to use different proxy settings for the VM than for the rest of the host. And finally, we use the environment variables from the process environment when the instance is being started. This results in most of the time the proxy being configured automatically without any user intervention. But if there are special considerations, we have the mechanisms to override this. So once we have the final settings here, we change any references to local host or 127.0.0.1 to the gateway host address, because otherwise we would not be able to access a local proxy running on the host. There's no standard of using uppercase or lowercase variants of the proxy variables. So we create matching uppercase and lowercase names if one variant does not exist, and we make sure they have matching values. Finally, we store those values in Etsy environment inside the VM. So any shell session, any SSH session, any program started from those sessions will automatically inherit these proxy settings. Port forwarding is working very well, but there are a few limitations that are worth keeping in mind. One is port forwarding relies on polling, both PROCNET TCP data as well as IP tables rules. So it may take up to three seconds from when a port is opened in a container until the same port is opened on the host. Another problem could be that the port is already in use on the host. For example, you may be running a web server on the host. So the default ingress addresses that would be used by your K3S cluster cannot be forwarded to the host and the cluster is inaccessible, at least through the default ports. Another issue is that obviously the guest IP and the host IP are different. So any load balancer or node IP services inside a Kubernetes cluster will show the guest IP as the external IP address, but that address is not routable from the host, which means any applications parsing the service configuration would not work on the host. And finally, we have not implemented UDP tunneling over SSH port forwarding. It should be possible to do this, but nobody has written the code. In case port forwarding is not enough, you can add additional interfaces to a Lima VM using VDE VMnet Demons. They must run as route to be able to use Apple's VMnet framework for virtual machines. The connection between a Lima VM and the VMnet Demon is through a virtual switch. Multiple machines can connect to the same switch. Lima will manage the lifetime of all these demons. It will start them when the first VM connects to a specific network and it will stop it when the last VM connected to that switch stops. There are three network types. There are host networks that are purely local to the host. There are shared networks that have an additional NAT gateway to the network adapter on the host. And there are bridge networks that are bridged to the external network. So are being routable from other local machines. The configuration on this slide shows Lima VM one running K3S, which is using both port forwarding as well as a shared network and a bridge network. The VM number two running container D is only connected to the shared network. So it's accessible from both the host and VM one, but not from the external network. Lima has been extremely popular. When it was announced on Hacker News, it jumped to over 2,000 GitHub stars overnight and it has continuously grown since then to about 8,000 by now. In that time we merged over 400 pull requests from 45 contributors. We did releases approximately every two weeks. It's a very active project with wide support. There are several third-party projects that extend the functionality beyond the scope of Lima itself, like providing a graphical user interface or adding additional client support for specific technologies. For example, Lima includes over 20 VM templates, both for popular operating systems like Alpine, Debian, Fedora, OpenSousa or Ubuntu, or for many projects like Docker, Function as a Service, Nomad or Portman. The Lima GUI project provides a graphical user interface to select one of these templates. It allows you to edit the template and then start an instance of it, just like an app store. CoLima is a command line utility that specializes in the container runtime aspects of Lima. And Rancher Desktop is a GUI that also focuses on container and Kubernetes management for developers. The Rancher Desktop GUI supports container management using either container D or Mobi as well as Kubernetes using K3S. It also includes the Rancher dashboard for Kubernetes, which we borrowed from the full Rancher project, but narrowed it down in scope to just the local cluster. It allows you to drill down into any aspects of your Kubernetes cluster to look at the events, inspect the logs or SSH into a container. Rancher Desktop also makes it easy to change between Kubernetes versions. For example, you can install, let's say, 1.19.6, then deploy your apps on top of it and then upgrade Kubernetes to 1.20.3 and then see how your app behaves. That allows you to simulate a production upgrade and see if you can expect any problems from the version change or not. We've also integrated image vulnerability scanning using the Trivi service. So I don't have a screenshot for it, but there's a list of all the images in your virtual machine that you can inspect. You can click on the image that you want to check and then invoke the Trivi scanner and see the scan results. Rancher Desktop works also on Linux, also using Lima and on Windows using Windows subsystem for Linux version 2. Rancher Desktop is free and open source just like all projects from Rancher Labs and SUSE. In summary, Lima makes it easy to use container D and K3S on macOS. The automatic file system sharing and port forwarding provide a seamless experience almost as if everything is running on the host itself. You see here four simple steps to install Lima, start the VM, deploy an engine X container, and then inspect the endpoint of engine X. Please let us know how you're using Lima and if you're running into any limitations. Share your thoughts with us in GitHub discussions, file your enhancement requests of bug reports and GitHub issues or contribute with a pull request if you can. We also have a Lima channel at the Rancher's user slack co-located with the Rancher Desktop channel. This is purely for convenience because there is a lot of shared interest in the same topics between both user groups. But otherwise Lima is a vendor-neutral project and not related to Rancher or SUSE, of course beyond being a major component of Rancher Desktop on Linux and macOS. Thank you very much. This is the end of the recording. We'll hopefully switch to live questions and answers now.