 Hello, this is the talk Linux Gaming in 2020, preparing the Linux kernel to emulate modern Windows games. My name is Gabriel, I work for Collabora, and I'm part of a larger project to improve the experience of gaming on Linux. Since game is quite a broad topic and it gathers a lot of attention, I should make clear that I'm giving my personal opinion on some topics, and I will be exposing some ongoing kernel work that we're doing to improve gaming in Linux. But first of all, let's look a bit at the picture of what games are in Linux today, how they look like. We have basically two sorts of games running around, we have the open source games and the proprietary, and from a development perspective, from an operating systems engineer perspective, open source games are the easy kind. We can just rebuild them, fix big and adapt them, and develop them in a programming model that really fits the way the Linux is taught, the way that we usually develop software for Linux. But then there is the proprietary realm where there are games where the studios, they case some resources to running them on Linux, and the other scenario, most common and very famous games where the studio simply doesn't care about Linux at all, they write their game for a specific platform, in most cases Windows, sometimes for Mac OS, and they just publish it for them, and we are left behind. So this is the scenario we have today, and this talk will be obviously focusing more on this specific Windows-only games and the work we are doing to bring them to Linux. Basically games are in a special category of software, in the way that it's hard to make open source games, and most of the games famous today, they are proprietary applications with very little support from gaming studios to bring them to Linux. So recompiling them and modifying them to fit a Linux development perspective is a bit out of the picture. They are also exceptional in the sense that they carry a lot of locks, both for DRM, which is more common in other kind of application, and anti-cheating mechanisms, which are mechanisms to prevent you from just cheating on the game and making it not fun for everyone else. So they implement a lot of features to prevent anti-cheating that sometimes get in the way of our reverse engineering or our work to bring them to another platform. As I mentioned, they are mostly coded for other platforms, in particular for Windows, who always reigned over this specific area of the market. And with the long goal of bringing gaming to Linux, it's kind of a chicken and egg problem. If we had people gaming on Linux and huge chunks of the market gaming on Linux, maybe you could be able to attract more attention to Linux, as a platform for gaming development and game playing. But we need to increase developers' interest, we need to make the platform more attractive for them, and by making it more attractive, it includes also bringing users. So we have a way to circumvent this problem and bring all this gaming ecosystem from Windows into Linux through emulation, through compatibility layers, and sometimes through some kind of reverse engineering. So the approach that Valve in particular took back in 2018 was to start this abstraction layer called Proton. That was basically a compatibility tool. And a big piece of it is Wine. It's not a new technology, Wine has been there for a long time, but Proton brought together Wine with several other technologies to make this, to make running a game over Wine more effectively effective and easier for the end user. And this Proton has been maintained since 2018 by Valve in an open source fashion with support from developers all around the world. Basically Proton looks like this. It includes a bunch of libraries, Wine in particular, and it's the primary mechanism to start a game through the Steam client. It's going to use Proton to launch a Windows-only game and run it over Linux under this compatibility layers. It's going to configure a lot of stuff, set up the Wine environment and launch the game through Wine. One detail is that the game and Wine, Wine then is going to execute the game in this compatibility mode, in this sandbox-like environment, not exactly a sandbox, but Wine then performs as a compatibility layer. And as I mentioned, Wine is the big piece in that. It's not a security sandbox, a Windows application runs in exactly the same protective space as Wine itself. One important detail there is that Wine doesn't do any kind of virtualization. There is no, it's not a mean, it's not a matter of virtual machines here. Wine and the Windows application, they run on the same process as part, different parts of the same process. And what Wine does is quite simple. It exposes the APIs and the ABI that the Windows application is using. Why is ABI an API? Well, it exposes the ABI for the program, but it does that by implementing the API that Windows applications expect to use. And that API, it reflects a lot on how a program is developed for a specific interface. We're going to be discussing more on that later. Just a small comment is that Wine is also community-developed and it has major contributions by a Valve and Code Weavers and I would like to send it to those guys because they really help us on the development that we are doing that I'm going to be discussing here today. My point about the difference in the ecosystem is that when you run a Windows program, it's talking mostly to the Wine API and the way that you develop the Windows program is based on the assumptions made by that API on what it provides you. Same thing when you're developing something for Linux. When we say that Linux is a POSIX-like operating system, it means that applications running over Linux have a set of expectations of how the operating system behaves. And it's not a matter of system calls, it's more a matter of libraries, what is exposed to you. So the way that you program, the way that you think a problem for Linux is a bit different than the way that you program, that you solve a problem in Windows. And that is mostly what the API exposes to you. So when you write a program for Windows, you use the Win API. And when you run that over Wine, Wine is basically a mechanism to translate that semantics into Linux terms, into Linux libraries and the Linux kernel. So it's going to implement that functionality of the Windows API into something that can be executed by Linux. And one important detail there is the more different the Windows API, API is from the Linux API, the bigger the work that Wine needs to do. If they were very similar, Wine would be just a quick translator that could execute that. But when you have very different interfaces, Wine needs to do a lot of heavy lifting to emulate those interfaces and that results in increased overhead. So the effort that we had to collaborate in my team and the kernel team have been doing with respect to gaming in Linux is mostly identifying pain points for Wine to emulate Windows interfaces with the current kernel. Basically we look at Wine and see where the overhead is and what interfaces Wine would benefit from if they were supported in the kernel. This doesn't mean that we are implementing the Win API or any Windows is calling the kernel. On the contrary, it means that we are trying to solve in the kernel things that are harder to solve in user space, always adding them to a Linux design. So we take specific problems that are not resolvable in user space or in a compatibility layer and try to solve them in the kernel space. And the reason they might not be solvable in kernel space is usually performance. Or sometimes, as in the file system example I'm going to give, correctness. It's not possible. The Linux interface doesn't expose a mechanism to do a specific thing, raise free, or it doesn't export that functionality at all. And this is where we try to extend Linux. The first pain point that I would like to discuss is something that arose in file systems and how Windows and Linux file systems diverge in some semantics. So the issue at hand is that historically Windows exposed their file systems as case-insensitive, while Windows does them in a case-sensitive fashion. And case-ins sensitivity basically means that any of the files that you can see in the grey box in the right, they are resolved to the same file no matter the case. So it basically is a file system in a manner where case of words doesn't make a difference. That is a very interesting approach for human beings, because we don't usually think in a matter of case, but it's more complex for applications, for operating systems and file systems in particular. Since now there are different ways, bite-wise, to refer to a specific file. Linux has never dealt with that problem because most of native, in fact both native Unix file systems before Linux, they always dealt with file systems, sorry, they always dealt with file names as if they were an apart bite sequence, basically just a sequence of bites that doesn't have any lexical meaning. But when we are bringing a game to run over a compatibility mode, that game usually relies on this behavior, which means that the game might try to access a file with many possible cases and when it is operating over a Linux file system on a compatibility mode, that file with a different case will not exist and the game will eventually crash. So what we've done until today is Wine had to emulate, capture every access to a file coming from the windows side of the program and emulate it through to figure out what exactly the case was in Linux and then open that specific file. That kind of emulation is costly in particular for large directories because it requires Wine to cache a lot of information and directories change, files get added and remove it and that overhead started to count on our performance measurement of games, which means that at some point games were running slower because Wine had to do all this work without any assistance from the file system. And it's not just a matter of saying, okay, for people running games let's use XFAT or some other file system coming from the windows world that is case insensitive by default because Proton doesn't choose your file system, usually your distribution does or you choose yourself and you don't change it afterwards. So when we are dealing with this kind of problem my first approach is let's not change the kernel, let's see if we can solve that user space. Took the Wine implementation and redid it in a different way, we tried a lot of caching we used notification mechanisms provided by Linux kernel to track all this information and we came to a future user space implementation of this it was not Fuse dependent, it had some interception mechanism, but it has inherent performance problems, there is no way to solve that unless you can write metadata information to the disk that allows you to retrieve a file independent of the case and that requires modifying the file system. In addition it has some unsolvable race conditions for instance when you are accessing looking for a file and someone else removes the file that really needs to be the kernel to figure out and that at some point started to crash games so the only right solution for this is doing it in the kernel which might seem unfortunate for some people but is the only way forward so we started developing in 2017-2018 a native case incentive support for selected Linux file systems and by selected I mean the file systems that are most common in distributions, we started with EXT4 now there is F2FS which is a file system very important for Android and we have current work, we have ongoing work for ButterFS for EXT4FS and other file systems and that implementation it solved all the pain points that we had it's much more performant because we can modify the disk to store that kind of information it's also correct because we have the same semantics of VFS operations in Linux so one interesting thing about this work is that it received some small backslash from the development community which was expected since Linux always did things the other way in a case sensitive fashion but we found out that there were more users out there and when we started developing it the Android folks came and said hey we have the same problem with Android and Android also publishes also exposes parts of its file system in a case sensitive manner so this work that we started for a gaming interest to solve gaming specific issues got applied to other parts of the Linux ecosystem and this is what this talk is all about is how we develop these features on games and eventually they are useful for other people I said that it had a small backslash from the development community but it had a much bigger backslash from user community people screaming don't break my system and for that reason I prepared a very small FAQ actually an FAQA frequent questions answered regarding this feature which is quite controversial in some circles case sensitive is supported natively 64 since Linux 5.4 and it's not a feature that is either always on or always off it's an optional feature that can be enabled on a per directory basis it will never be a default setting it's not something that you want to configure on the root of your file system and the reason for that is even if you try to do that we made sure that it doesn't work because just looking at a generic Linux file system you can find stuff in slash user slash leap that collide in file name deferring only by case so this is something that should never be used on the root of your file system if you don't have a very very good reason to do so and you know what you're doing it's something that you should use on a very on a specific directory inside of your home where your games are set where you really have a use case for that on the case of Steam I believe Steam would configure that directory for a game when it downloads it or something like that and it's safe it only applies to those specific directories the rest of your system continues to be to work as in a case sensitive fashion the reason for that is that all the other programs in your system expect your file system to be case sensitive and we care a lot about not breaking stuff so we make sure it will work in a way that is safe for people but it requires some distril support to be enabled it requires the kernel to support the feature at build time it requires a specific flag to be set on your disk declaring the disk supports that functionality and it requires an up-to-date version of E2FS Probes if you're using EXT4 or the corresponding programs in another file system and the problem is many distros still don't enable config unicode and they also don't enable encoding installation and I just want to make sure that people know that enabling that configuration and enabling the feature disk-wide doesn't make your system go crazy it still requires per directory configuration and actually I expect in the future that more and more distros are going to be adopting that configuration by default because we make a lot of effort to make sure it works and it would be a shame for it to go unused and it can go unused for instance why do we always have the fallback mode where it tries to do case-insensitive in user space either for older kernels or for any other reason and there is a performance hit on that but there is still this compatibility mode so in the case-insensitive feature there is some things that are desired wanted features and that my team is still working on the first one of that is improving FSCK support so we can guarantee that any problem that arises in a case-insensitive disk whether or not it's related to case-insensitivity it's fixable this is not something specific to case-insensitive every file system can be corrupted at any time and we always have features to fix that we are pushing more distros to adopt it such that people installing steam on any distro can benefit from those things and like I said we have work on going to support case-insensitivity more file systems with the same semantics such that other users can also benefit from this feature without relying without falling back into the wine emulation, user space, case-insensitivity the second pain point that was brought to us by the wine developers is related to thread synchronization and how it's done in Linux it's quite a complex problem but an easy way to understand it is looking at this aesthetic example which is not necessarily a real use case but in this scenario we have multiple producers and multiple consumers but there is a relation where for each producer there is only a subset of the consumers that can consume the data from that producer so for each producer for each consumer we want to maximize its usage, its ability so it needs to be able to consume something from the producer as soon as it's done we don't want it to be waiting for a producer where a different producer is ready but since not all producers can be consumed by all waiters and the same producer can be consumed by different waiters there is a challenging model in it we observe that there are ways to model it in user space or model it using existing interfaces in the Linux kernel but wine developers and game developers started to observe that that was also becoming a performance issue for games running over wine so they asked us and started to look for an approach and basically that problem arrived specifically in a game scenario because games are obviously very thread heavy and there is a lot of communication between threads and the fact is Windows has a very convenient API to handle with that kind of things which is called wait for multiple objects it's designed to allow a program to specify a list of scenarios where that thread is going to be awakened it can be awakened by a mutex changing state it can be awakened by a signal, by an event by a different thread, by a condition variable by anything and it can specify a list of these things in Linux what we can do is we can pull we can pull on a pipe on a socket and wait for data coming from that socket we can use EventFD for that and pull on several EventFDs one for each thread or we can try to write a dispatcher in user space that waits on a single condition and every time that condition is triggered for any thread we can dispatch it to the right thread we experimented with all of these things and we observed issues with each and every one of them for instance the isync implementation is based nowadays by proton users it's based on pulling on the EventFD interface and the fact is EventFD fits the problem quite well and pulling on it is a good solution but there are a few problems there the first is that file descriptors is a limited resource and we heat it very easily with games and multiple objects and the second is the backup of pulling that started to become a bottleneck and appear on our benchmarks and that in particular we were able to observe heavy CPU usage for threads that were just slipping on it and EventFD also has a problem that it requires us to always go in the kernel so unlike few taxes it doesn't have a fast path that let us avoid going to the kernel for us we experimented with optimizing EventFD but we learned it was quite a challenge and then there is obviously the other solution I mentioned that is bring all of these and do a dispatcher in user space and we can model this in a very simple way put all your events under a single condition variable and whenever that variable is any producer that has read triggers that thread, triggers that variable that is going to run the dispatcher thread that selects the correct waiter to wake up that is a common model for parallel programmers but it has some limitations in the sense that it has a lot of spurious wakes we are going to be waking dispatchers to wake the actual job worker and that also showed on our benchmarks the fact is when we are dealing with with applications like games every performance change is observed in frame per seconds and very visible to users to gamers and we try to look for a better solution on that to make this problem really go away and the solution came as a suggestion by Zebeda Figuero who is a developer at WINE and he proposed a new few-text operation to model exactly this kind of program it doesn't completely emulate the wait for multiple objects in the kernel because wait for multiple objects can wait on several kinds of objects and here we are proposing that we pass a list of few-texts and wait on that list and many of those few-texts will trigger the sleeping thread that makes all the emulation on user space much easier because now we can treat every different object as a different thread assign a few-text to each of them and then the problem can be well resolved in kernel space and then we are reusing all the genius behind few-texts and then we are reusing them and it proved to show very good performance it's implemented by the kernel support that we published on the kernel main release and on a test kernel for gamers it's a Ubuntu-based kernel that has the patches and we published a version of Proton that has a module called fsync that is capable of using that new interface and well the interface itself it received very positive feedback from the kernel community the use case that we were presenting is very well understood and it's known to be a current issue in the kernel there were suggestions in the past to solve this issue the fact is that when we submitted those patches we uncovered a bit of a lot of old issues in particular with the few-text kernel code which had maintainability issues, complexity issues and a lot of people were trying to merge new features on that kernel on that feature and it wasn't well received by the maintainers so the community decided that it was a moment to rewrite the few-text interface and then we took this effort and started to design a new interface with the help of the community and as a result the result of that work is a few-text too which I think there is another talk on this conference discussing that specifically like case incentive there were developer feedback and user feedback and user feedback in this example they were very positive one of the Valve developers published a request on github for people to share the results using fsync with the test kernels that we distributed for Ubuntu for Lycorix and for several other test kernels and the results were incredible we have people reporting from 5 to 300% improvement in frames per second in different games and I can say as a personal example that I have at least one game that was really unplayable in my system that now runs really well I don't know if I can say the name of the game but what does that mean? Does that mean that Linux is worse for threading than Linux? Absolutely not. It means that the programming model that was used to design that game doesn't fit the windows, the Linux programming ecosystem the Linux APIs so we are extending it to better supported just a few comments on the new interface that we are proposing. FuelTechs 2 doesn't solve only the vectored version of FuelTechs the multiple object version of FuelTechs it solves several long-standing points on FuelTechs for the kernel community it brings all the new features that a lot of people were expecting a very valuable contribution to the kernel itself it brings new malwareness, variable size of FuelTechs all the stuff that people were really rooting to have in the FuelTechs code we are trying to bring that now to the kernel and it also has, since we are designing it from scratch it also allows us to convert the weight multiple code into a better design that we are working on and since it's a new implementation we can just design it in something that will benefit from vectored FuelTechs from the start and I still don't have numbers but from what I hear we have very promising results and my expectation is that we might have better performance numbers for games than we have with FuelTechs weight multiple I'm hoping for that but I still don't have the numbers to do back it whether that will show in benchmarks or not is something that still needs to be seen finally the third pain point that I want to discuss today is Cisco emulation which is the topic of my previous talk that I think went yesterday at the time of the conference and this is a problem of correctness for games running over Linux through wine basically the problem is that now modern game DRM and NT cheating technologies they are issuing system calls directly from the Windows game code and that bypasses wine because wine is not a sandbox wine is not a virtual machine, it's simply a library that gets linked to your program so wine is not able to capture those system calls and once that system call coming from the Windows code reaches the kernel the system call has parameters and arguments as it was Windows kernel running the game is definitely going to crash once again like we always did we tried first to find a user space solution because that is simpler it avoids tinkering with the kernel and as well it's definitely simpler and the solutions that we attempt which involved SIG Comp and dynamically rewriting system calls they had their own issues with NT cheating software we cannot modify a game binary without triggering all the codes to prevent it from being modified for piracy and cheating so we cannot touch that part and they also affected performance like I mentioned any attempt to emulate will introduce a performance overhead in your gaming experience so the right solution once again was let's go back into the kernel and then we developed this whole new mechanism designed specifically for emulation of gaming system calls that are integrated into the Linux kernel it's called a syscall user dispatch it integrates very well with SIG Comp and other modes that already exist in the kernel and it's designed for emulation so it's really fast for syscalls that are dispatched natively so it's very fast for a hybrid environment there are a few syscalls coming from Wine and emulated syscalls from the windows application this is still a work in progress like the few text stuff but it's expected to land really soon I was hoping to have it in 5.10 but most likely it will land only on 5.11 and the option for it is going to be very simple you just have your distro enabling the config option distros are enabling that because it's very it's a very isolated feature that doesn't affect the rest of the system so there is no reason to not do it and from then you just use a proton version new enough I'm not going to go deep into this item because I just gave a talk about it yesterday or maybe tomorrow I'm confused with the schedule but yeah basically I'm not going to go deep into this feature just watch the other talk there are a few other pain points that we've been working on with our customer to improve gaming experience in Linux we work it on device bring up we work it on solving some file system issues improving performance in other areas of the system we have a team looking at some scheduling stuff that we maybe can optimize in future corner releases so it's a very long ongoing work that is already bringing very good results for gamers using Linux as their platform of choice a few takeaways of this talk is that we are not in any way trying to bring the Windows world into Linux we are trying to ease emulation and only extend kernel and user space libraries when really necessary we try to make our designs that are useful for other people on the Linux ecosystem in particular the few-text approach is something that's going to benefit a lot of people and well we are seeing a lot of good results with all this work and hopefully they will reflect in a better gaming experience measured in frames per second as a final metric in several cases for end users and that's what we'll keep doing we will continue to make Linux support more and more games both natively and emulated as a personal wish I would love to see more open source games who don't rely on most of this these workarounds these emulating efforts but this is in the future hopefully I'd like to acknowledge a few companies and people who have been sponsoring this work so in particular Valve and one of their employees at Yardoop who have been sponsoring a work and supporting it and all sorts of support for this work also thank the wine developer Zibadiah and Paul for their valuable design ideas and a lot of technical input in all of the works that I presented here today thank you I'm asking to mention that if you'd like to work on making the Linux ecosystem better for gaming and a lot of different interesting projects collaboration is always hiring and I'll probably be taking questions during the talk so thank you