 Welcome to a talk about portability from Unix Windows using Gento. My name is Michael Haubenwalner. Just call me Hauby. And I'm working at SSE Schaefer IT Solutions, a company that merged last year from Salomon Automation with SSE Schaefer PIM. And, yeah, this one. What are going to talk about today? Of course, there is a challenge behind. Then there is a difference in the requirements between application and the developer. Another thing is the difference between Unix and Windows. Fortunately, there is cycling to fill the gap in between. But still, Cygwin has a problem with package managers. As a solution for portability problems, there is change to prefix. Another helper utility is change to parity. And then I'm talking about a few next steps. I already know that have to be done. And finally, I'll try to show a demo video. Initially I planned to do the live demo, but this would rely on a stable internet connection. For the challenge, SSE Schaefer is creating a warehouse for a customer like this one or not like this one, but similar. There is a software necessary to run this warehouse. This software is called WAMAS, short for Warehouse Management System. This is a challenge by itself. I'm not talking about it, but another challenge is a warehouse usually does run for a long time. Up to 20 years, sometimes even more. This is true for the software as well. Because of being a software, it does need an operating system to run on, which of course needs a server. The challenge here is, which operating system would you choose where you can provide 20 years of support for? Well, actually, within 20 years of software, and I'm working for a little more than 20 years already, it is a Schaefer, I have seen lots of different operating systems. Some of you may remember a few things like HP Unix, AIX 4, AIX 5, AIX is still active with AIX 7. Of course, there is Linux, the different variants Redhead, and the most wrong one, the most foreign one is Windows, Windows Server actually. So how to overcome this changing operating systems within 20 years? Well, one could think of server virtualization, but this doesn't help to remove the dependency on the operating system. But fortunately, almost all of the software is necessary to build and run the application when using open source software actually do support all or almost all of those operating systems. So the solution here is to create a custom GNU distribution that does run on top of these operating systems. But first, let's have a look at the different requirements for the application and the developer. The application does need the operating system. An operating system that surfaces would provide a file system, some kind of networking, some kind of IPC, and of course, process control. The developer, on the other hand, it's not the developer that needs the operating system, it's the development environment. The developer expects from a development environment, of course, an editor for the source code, a build system to create the application, a debugger. Unfortunately, the developers don't have just to develop, but they also have to debug their developed code. And to be able to debug the application, the developer also needs the application runtime environment. And this is the reason why the developer also doesn't need an operating system. In a more specific detail, the application prefers to run on the vendor supported, on the vendor provided part of the operating system and prefers to be built using the vendor supported toolchain. That is, as a company selling software, it is quite crucial to be able to buy support from the operating system vendor. So, yeah, there is... If you use a toolchain, just a GNU toolchain on AIX or on Windows, you won't get support from the operating system vendor. Another point is when using the vendor toolchain, there are tools that provide integrity checking. Which one we'll see later. For the developer on the other hand, the developer prefers an operating system independent software to build the application and the libraries. Then operating system independent scripting languages. For most of the time, it's glue code to glue various build systems together. And for all those dependencies necessary for the application, it is quite crucial to have a package manager which ideally works the same time or provides the same user interface independent of the operating system. However, the debugging experience is more important to be smooth and working with the toolchain than to be operating system agnostic. So what's the choice? What are the choices for these requirements? For the application-specific part on Linux, of course it's quite easy. There is the GNU toolchain. For integrity checking, there are things like Vargrint. And debugging, of course, is done with GDB as the back-end, as the real debugger. And the developer can choose whatever GUI he prefers. For Unix, this is quite similar. Fortunately, the GNU compiler is able to use the vendor-supported Biniutils. This is mainly the assembler and the linker. Difference is in the integrity checking tools. They usually work when using at least the vendor Biniutils. Sometimes there are problems when using GCC as the base compiler, but usually they work. And fortunately, even GDB does work so far on these operating systems. But then there is Windows. The native toolchain here is the Visual Studio compiler available as a command-line compiler, cl.exe. For integrity checking, there is application verifier. And of course, the Visual Studio is a quite good debugging IDE. For the developer, the operating system independent choice, the software chosen here needs to be highly portable. I have found the GNU build environment, auto-tourists, makefiles, and of course GNU libTour as the most portable software found so far. The same applies to scripting languages like Bash, Python, Perl, and similar. And Janto Portage is the so-far portable package manager because it doesn't rely so much on the operating system, except for the file system, but of the build instructions. So the real important thing here is the build instructions to be working for any target operating system. But still, how does Windows apply here? Or the other way around, how to apply this choice to Windows and Visual Studio? Well, getting a step back, what's the problem with Windows actually? Looking at these build tours, GNU tours, they do highly rely on the POSIX API. So let's have a look on the POSIX API. And here is where Cygwin comes into the game, which is actually a wrapper for the POSIX API on top of the Win32 API. Looking at the requirements from earlier, Open is for the file system, and Cygwin is able to just map them to Win32 API. The same applies to networking and IPC. But this is only partially true for process control. Windows, to create new processes on Windows, there is just to create process, probably with an extended version. And it's only spawn, POSIX spawn in the Unix world that does quite the same as create process. But exec and fork, known from the POSIX API, there is no direct opponent in the Win32 API. So Cygwin does introduce a Cygwin process ID and need to work around missing functionality in Windows, but it still is able to provide a working fork so far. But then the second fork, as of now, does have problems with package managers. Because the prerequisites to fork work out on Windows is that the child process created as a normal Windows process requires the identical process memory layout than the parent process. Cygwin actually does start the child process from the original executables, does suspend the child process, copy the memory from the parent process to the child process, and resumes the child process. But this works only if the binaries are identical. On the other hand, the package manager, his job is to replace even those binaries, actually replacing whole packages, including the binaries, while still using fork to run maintenance programs. So you see the gap so far? But there is help. There is a workaround for this problem in that when Cygwin is to create a new child process and detects the original binaries are not there anymore. Cygwin now, or there are patches to actually do so, they are not released in Cygwin yet. To temporarily create an application directory, put the original executables and DLLs there, combined with a Windows-specific DLL redirection file. This is an empty file with the application executable name .local. So with this forces Windows to load the DLLs necessary by the application to look in this very directory where the .local file is residing. On a detail, this temporary application directory resides in the var run Cycfork directory, and the original binaries are provided there and TFS hardlinks. These patches are expected to be released as Cygwin 3.0, because the Cygwin developers find this is a new feature whereas bumping the major release number. Now for gentle prefix. Who already knows gentle prefix? Oh, quite a lot. I'm not going to talk in detail about gentle prefix, just that it is working because most of the GNU packages do support to install into whatever directory you want, and this is the configure argument, dash dash prefix, hence the name gentle prefix. I did a talk three years ago on gentle prefix and how to provide a long-term support in the sense of 20 years support using gentle prefix. But still the question for Windows and Visual Studio. Okay, for Windows, now we have Cygwin. But for Visual Studio, there is something really new. There is gentle parity. The gentle parity is a wrapper around the Visual Studio compiler command line. It does use the Visual Studio toolchain, the command line utilities, cl, link, lib, and the like, while providing a GNU-like command line to the build tours, the GNU build tours. So there is a command, basically parity GCC, which understands the shared, for example, to create a shared library. It does wrap this one to the Microsoft toolchain and does create a real Windows-only DLL, or same for the executable. Additionally, parity provides a loader library, which provides features like run path embedding, as well as preloading. We'll see it that later in the demo. While still parity is a Cygwin executable, it is available on GitHub in the gentle space. Recently I've put it there as gentle parity. So there is a new answer for the Visual Studio question. This is called parity. What are the next steps? For gentle parity, it is not yet there. It is not yet compatible with the MinGW toolchain, which is not the native Microsoft toolchain, but still creates Windows boundaries. And the loader library does not use the Microsoft support by link.exe to support a helper function for delay loading DLLs. Hopefully, when switching to the linker supported delay loading, this will simplify the porting to 64-bit. For gentle prefix, there is something different. There is prefix chaining. A short note here is you may have realized that Windows does not provide the fork except without Cygwin. But when creating native processes using the native compiler, the target binaries are not Cygwin binaries. So the target binaries are not able to use fork. They just need to use the Windows API. Fortunately, there is prefix chaining, which is able to maintain another gentle prefix but which does not contain the package manager itself, gentle portage. Since that, there is a Cygwin prefix, which contains the package manager. And prefix chaining does this package manager to manage a different prefix, which holds only binaries created by the native toolchain. This is similar to cross-compiling, but actually it's not cross-compiling because the created binaries can be executed within the original build environment. It's kind of multi-lip because it's on the same architecture a different variant of binaries to execute. But still it's not multi-lip because it's in a different prefix, so it's not merged into the same prefix as the original Linux multi-lip does. And as of now, there are mean GW keywords in Chintu, but they refer to the real mean GW toolchain and not to parity for the moment. I am planning to use mean GW keywords with parity because the idea is basically to have Chintu parity as a plug-in replacement for the mean GW toolchain. But we are not there yet. Of course, there are lib-tool patches that do the board using parity. Actually, lib-tool does the board using the cl.exe toolchain already, the Microsoft toolchain. But it's different and quite, yeah, incomplete without wrapping the Microsoft toolchain. You cannot do a few features lib-tool usually relies on. So there is, of course, need to patch lib-tool to use the parity toolchain. These parity-lip patches are not integrated into Chintu prefix yet and are not yet submitted upstream because parity is still under development towards the mean GW toolchain. I can't promise, or I do not expect, actually, to reduce the complete need for patching lib-tool when parity provides the full mean GW toolchain. I don't expect this would work, but it could be. Yeah, no, I don't think it's possible. But finally, the goal is to build more libraries using the Visual Studio toolchain within Chintu. For a demo video, let's see how that works out. I have set up in a virtual box a Windows Server 2016 with standard Cygwin setup plus the Cygwin SSHD. On this Windows, I have installed all the Visual Studio versions from 2008 to 2017. I have set up a Chintu prefix as the 64-bit Cygwin prefix in-slash-forstum-Chintu. And I have set up a Chint prefix-Chint prefix using the Cygwin prefix from Forstum-Chintu to manage the Windows prefix in Forstum-Windows. Let's see how that works out. This is aging into the Cygwin or the Windows box. The Cygwin box, actually, here you can see this is Vanilla Cygwin 2.9. And I have started this Chintu prefix. Now I'm going to replace a DLL used by the currently running BESH. Currently running is the prefix BESH. The DLL I'm going to replace is Cygintl. And I have navigated already now to the location of this Cygwin Cygintl DLL. Is this readable so far? Nice. So the next step is to replace this DLL with the Cygwin provided DLL. It's a different file but provides the same API. And now we're using the Vanilla Cygwin DLL. You can see how it breaks. Why does it break? It does tell quite nice the different Cygwin... The current shell does fork or wants to fork to execute the LS process in the child. But Cygwin realizes that one of the DLLs necessary to start this Cygwin, this new child process from BESH, detects that this DLL while existing is loaded to a different address compared to the parent process. And this is the reason why Cygwin fork currently breaks. But now I'm going to restore the original DLL for now. Now I'm replacing the Cygwin DLL with a Cygwin DLL containing the fork patches. So this is running not so readable. Here I'm stopping the SSHD within Cygwin. Then there is... So there are no Cygwin processes running so I'm able to replace the Cygwin DLL. Then I'm replacing the Cygwin DLL with one that supports these so-called fork hubbers hard links and starting the SSHD again. Now navigating again to this very same... I'm doing this very same again. Navigating to this DLL to replace. Here you can see this is not vanilla Cygwin, but Cygwin with additional patches. I called them the gentle patches or actually I do need a different name. So here there is still the backup and the original DLL. And now LED dollar bash shows this is still to be loaded. Then replace this DLL with the Cygwin one. So it's actually a different one. And now LS bar bash is able to fork even with the replaced DLL it does need and execute LS in the child process. This is necessary for gentle portage or for a package manager running inside Cygwin to work out because as already said, package managers do replace DLLs and execute their presenter like. How does this work? Let's inspect this var run Cyg fork directory. There is a directory pair user and the directory pair application. This application directory does contain the bash.exe, the original executable, does contain the original DLL and the bash.exe.local file which can be emptied. This is the DLL redirection file. This is how the work around actually in Cygwin can work. The patches are basically accepted upstream and pending for the next major release. Now create some application. I'm going to create a simple application here, C++ application and going to compile it once with the Cygwin compiler and the second time with the Visual Studio compiler wrapped by parity. This is quite simple. The difference to see is between Cygwin and the Visual Studio executable. Cygwin does mangle the zero argument for the executable as Unix path while the Windows binary encounters the real Windows path. And the application I created just prints this zero argument on standard out. Compiling this application with Cygwin does yield the posix path because Cygwin does mangle this one. And unlike on Linux and Unix, it's not a.out but a.exe, the default output name. Now entering the train to prefix where parity is installed already. Parity is able to find the installed Visual Studio versions from the registry which is easily accessible through Cygwin using slash proc Cygwin. Here you can see there is Visual Studio, this is 2008, this is 2010, then there is 2012, 2013, Visual Studio 2013, 15 and 17. So their product title does include a year name while the compiler internally just has an increasing number except for the 13. They do like the airplanes where you don't have a seat number 13. Now I set up parity to use the newest Visual Studio by default. Again it is querying the registry for available Visual Studio and thus set up configuration files to use this Visual Studio. Compiling the application now looks quite similar to done with Cygwin. It's just a different host triplet called I686 PC Win&T and G++ which actually is just a command line provided like GNU compiler. The backhand called is Visual Studio compilers. Here we can see the zero argument is the full Windows path because it is a native Windows application. The Cygwin LDD for that Windows executable is able to find some DLLs, some native Windows DLLs but the Cygwin LDD is not able to read those manifest embedded DLLs. This is like the side-by-side application provided by Windows. Cygwin does not use this one. Okay, now for a shared library creating a simple library that does not so much do then returning 17 a header file and using this library function in the main application file. I'm going to show beyond the creation of a shared library also how that there is the parity loader library which is able to listen to the LD library path environment for example, beyond others. But first I'm going to compile this with the Cygwin compiler again with G++ the shared output file name .so, lib.so and the source file and finally again linking the final application just specifying this library. Executing this work so far now I'm moving around the DLL into some sub-directory actually this is the .so file is the real DLL for Cygwin after moving the DLL away Cygwin is not able to find this DLL again well actually it's Windows not able to find the DLL but because of Windows using the path environment variable pointing to the new directory does find the DLL again however with Cygwin it does not work to use an LD library path to find the DLL let's compare this with parity again the command line is pretty much the same except for the host triplet the creation created files there is the .so file liblib.so as the output file this actually is the input library and additional creation is the real DLL this is called liblib.so.dll again the same command to create the final application does work as expected now moving this native Windows library around actually I have to move the DLL away and now as expected the application is not able to load this DLL of course because of being Windows using the path environment variable to point to the new DLL location does work but also you can see it is even the library path environment variable which is interpreted by this loader library provided by parity that does find the library again let's have a look at the files created so far again the lib.so actually is the input library this is quite similar to an archive file in Cygwin this is the Cygwin file command which just knows about archive files while the DLL file actually is the Windows DLL so now I'm going to merge setlib because it is a smaller library in the Windows prefix which is managed by the Cygwin installed portage I don't recalculate dependencies but this is portage doing its work for Windows the build system of setlib actually does not know anything about Windows it does know about MinGW but nothing about Windows and actually this build system here does the very same commands as it would do on Linux the only difference here is some uname but I'm not sure about how this actually is used parity still provides for i5-86 precise Cygwin triplet for WinNT6.1 this is actually the profile still used in change of brief as the WinNT profiles then the setlib build system actually setlib configure is asking the compiler if the compiler is able to produce a pre-processor error and the error message shown by parity or the visual studio actually you can see these are real Windows path and this is the full path to the source file provided to the visual studio by parity continuing testing if a few header files do exist they are provided by parity as well for things that do map to Windows actually and are available in Windows often with a leading underscore like Win32 provides the underscore open core which does basically the same as the POSIX open core back to the compiling the make file used with setlib just runs the compiler uses the original GCC command arguments including the GCC extension dash include dash include with GCC does include the specified argument specified file before processing the real source file parity actually does create a temporary source file put this dash include file setconf.h there first and second it puts the original source file there it is a temporary source file and the visual studio compiler does show this source file name on the standard out here you can see there are three jobs running in parallel for the linker line this is a real linux link line dash shared the original C-flex output file libset.so with the version number let's look at what has been installed now so probably the cars are not that readable first a real file there is libset.so.1.2.11 this is the input library then there is libset.so1.211.dll as the runtime library and there are simlinks libset.so referring to the input library same for libset.so.1 actually this feels somehow wrong and additionally there is a libset.dll.lim simlink for compatibility already with the minjw variant in libtool which does look out for linking shared libraries using the name scheme libset.dll.lim so that's it so far for a summary going back to the different requirements for application and developers the built environment to provide or using the posix api on unix and linux it's quite easy using the host libc and on windows to segwin.dll for the operating specific toolchain where which is needed to provide a GNU command line on unix and linux it's GNU-CC using native or GNU-bin.dll and on windows there is gentoparity wrapping visual studio compiler to provide the GNU-CCC command line questions? don't use the visual studio IDE to compile or build the command line then if you want to debug you have no solution for us no you can run you can run the application from within the visual studio IDE it just isn't used for compiling however I have done it in a different project you can configure visual studio as an end make the visual studio is an end make project and actually run the segwin make to run the segwin build environment using parity as the wrapping compiler providing the final windows binary and tell the visual studio project this is the output file they execute and debug normally in visual studio did you try running gen2 in the linux environment that the lower windows have? I have tried but the windows subsystem for linux has two major problems here first it is not able or it was not back in time we tried it at least to call windows binaries so you kind of run the windows compiler in the windows subsystem for linux this is the first problem the second problem is you may remember this is a windows server 2016 and as far as I am aware of the windows subsystem for linux only is provided for windows desktop systems when you are listing the compiler versions to use can you use e-select instead of parity setup instead? this is the idea maybe I should repeat the question whether the question was if we can use e-select instead of parity setup actually the idea is to use e-select question related to performance and how do you run the original pipeline instead of using the one provided by segmentary sets the final application runs on native speed because it is compiled as a native application oh repeating the question first probably the question was for the speed of the application when built with parity and the answer is it is a native application so when there is performance tuning necessary within the application running on windows it is performance optimizing as you will do on windows with any application and maybe I have a question can you use all the options in well for instance in the cc-corporate-make.com can you use all the for instance if you want to customize your own the made options and compile in parallel can you do that with under prefix? the question was if we are able in chat to prefix to use all those chat to specific options or chat to portage specific configuration in slash etc actually yes we can it's a chat to system it's chat to portage yeah it's a chat to system there's quite a few minutes left otherwise no more questions otherwise yeah one more the question is if it's only this one company using this system or if there are other companies actually there are no other users yet probably because this is something I create within this is a shaper and for example I am a chat to developer as well so I am able to develop chat to prefix including the parity wrapper within the chat to community and actually inside the company I am the first user for chat to prefix there are of course users but I don't know if they are company wise or just normal users but there are quite a few chat to prefix users not as far as Cygwin yet because this is quite new and especially not for the windows yet because this is even newer but yeah there are a few people interested in chat to prefix on Cygwin already when updating a Cygwin application like Bash that would be in a different folder so portage doesn't interfere with the package management of Cygwin itself or if I run both in one prefix if I run portage in the same prefix where Cygwin installs packages that would affect the two package managers messing around in the same process so the question is if there is a problem when chat to prefix does install binary applications into the same file system directories as the Cygwin package manager well the question here is the idea of chat to prefix is to not mess with any system directory but chat to prefix does use its own otherwise empty file system directory not touched by any system package manager and does install the chat to distribution chat to prefix distribution inside this one file system no one else should touch this one ok otherwise thank you very much