 Okay. Good afternoon everyone. My name is Konstantin, I work for IBM Germany, and I'm also a Red Hat Partner Engineer. For the last two and a half years, I was working on enabling OpenShift Service Mesh on S390X architecture, and in my presentation, I'm going to share my experience about that. So, first I will make a quick overview of Service Mesh architecture, then I will talk about the differences in implementation on X86 and S390X platform. I will also talk about the features that were previously were enabled only on X86, such as large-it and wasm filters. We'll talk about boringness library replacement, and we'll talk about future and possible enhancement. So, what is a Service Mesh? Service Mesh is a dedicated infrastructure layer in a microservice architecture that enable you to add capabilities like traffic management, observability, and security without adding them directly in your code. So, therefore, the microservice developer can focus on business logic only. The main component of Service Mesh is envoy proxy that runs in a sidecar container alongside each microservice, rather than within them, and those sidecar containers form a mesh network. So, currently, there are three implementation of envoy are available. First one is upstream envoy. It is used as a base for Istio Service Mesh, and based on Boringness Cell Library. The next one is Maestro Envoy. It is used as a base for OpenShift Service Mesh product by Red Hat, and the main and the biggest change in comparison with upstream envoy, it is a fork of upstream envoy, and the main biggest change is that Boringness Cell Library is replaced with OpenS Cell, and there are also changes that enable build on other than H6 platforms such as S390X and PowerPC, and the third one on envoy dash OpenSSL. It is also based on OpenSSL, but the Boringness Cell Library is replaced with OpenSSL using more advanced technical solution. I will mention it in the next slides, but currently, the status of this project is walk-in progress, and there are no walk-in versions for any platforms. So, envoy is written in C++, and Basel is used as a build system. It has approximately 60 external dependencies, and each of them is a separate OpenSource repository. Some of them are pretty big and well-known, such as Google V8 JavaScript engine. So, from the security perspective, the envoy build is being done in an offline environment, and all code, including envoy itself on all of these dependencies, are being built from source without using any pre-built libraries or binaries. And so, the build takes more than one hour, depends on the hardware available on the build machine. So, what is S390X platform? S390X is a Linux operating system that is compiled to run on IBM mainframe computers. And there are situations in which the code that is compiled on X866, and it just walks, and if we are trying to come to this code and run S390X, it crashes. So, the most frequent reason of such crashes is different in engines, which is the order of bytes in computer memory, so S390X is big engine, X866 is little engine. Another type of problems that we have on S390X is C compiler errors specific to S390X. So, the build is specifically configured in that way so that most of warnings are considered as errors. And here, you can see the example, one real example of such errors. And the root cause is that character types may be different on different platforms, signed or unsigned. And the last type of class of problems is Bazel build errors, and they happen because of some of Bazel tool chain do not support S390X or don't have binaries available for S390X. So, by design, all communications between the microservices are encrypted and Borens cell is used in upstream Envoy STLS library. So, Borens cell is a fork of Google's fork of open SSL and the problem is it is not supported on S390X and its community doesn't accept any S390X changes and Envoy community doesn't want to support open SSL in addition to Borens cell. So, on S390X, we have to use either this one or this one implementation. And the last one, it is not ready. So, currently Maestra Envoy is being, the difference between them is how OpenSSL is replaced. So, in Maestra Envoy, all calls to Borens cell methods and all headers and everything is being replaced manually by Red Hat Envoy team and this is a huge amount of work. And to reduce this amount of work for each release, currently this project Envoy OpenSSL is being developed. So, the main idea is to have so-called compatibility layer for Borens cell to OpenSSL. So, it provides an implementation of Borens cell APIs sufficient enough to build Envoy against it. So, there is a mapping between Borens cell and OpenSSL method and when Envoy calls some Borens cell method, then corresponding OpenSSL method is loading dynamically from OpenSSL shared library. So, using DL Open C function and others, you can read more details by clicking this link. So, initially, GCC was used as a main build tool in Envoy, but about two years ago the community changed the build tools and they switched to LLVM, Clunk and LLVM as a linker. And this caused a lot of problems on S390X because LLVM is not supported on S390X, so we had to switch to LLVM.gold and there were more problems. For example, integrated assembler wasn't supported, we had to disable it on S390X, then we had to define manually missing types, numeric types, replace them with the existing double or float in Clunk. And we have to fix new S390X specific warnings by either changing code or disabling those warnings. So, here is the summary of Envoy dependencies that were changed or replaced to enable the build on S390X. So, first, BornSSL is replaced with OpenSSL. Another project is LuaJit is a compiler for Envoy filters on Lua language. It was replaced with LuaJit 2, which is OpenRest implementation fork of LuaJit and it supports S390X. Another problem was until our parser dependency, we have found a bug that when passing deliberately incorrect string to input as an input, Envoy crashed and parser crashed. So, by debugging points to some internal functions within lib.sdgc++ string, and I couldn't find the root cause, but I could find to only walk around to disable string length optimization during the build. And now if hacker passes incorrect input and Envoy won't crash on S390X. So, another change was to Google Kish and in its HTTP2 implementation by fixing Endian's problem by swapping bytes and the last change was to WebAssembly extensions. So, WebAssembly or Wasm is a portable binary format for executing code and it is run in a memory safe sandbox. So, a user can write filters and compile them and dynamic load into Envoy as a Wasm module without stopping Envoy or rebuilding. So, currently on S390X we support two Wasm runtimes, Google V8 JavaScript engine and Wasm time. And to support them, we had to change proxy Wasm Envoy dependency, let me tell a little bit more about that. So, initially when we tried to build Wasm modules on S390X we got first build problem because we could build using only extensions written in Rust language and we couldn't build extensions written in C++ because it requires AM script and toolkit which is not supported on S390X. But even if we build them on X86 and copy Wasm modules to S390X we experienced crash, Wasm VM crash. After investigation the problem was that Wasm module always by design keeps all instructions in little engine format on any platform and Wasm runtimes, for example, Google V8 supports this on S390X, but Envoy proxy Wasm didn't support it and I had to find all places that serialized this serialized data to Wasm module that reads or writes data from or to Wasm module between proxy Wasm extension and Wasm VM and proxy Wasm maintainers were S390X friendly after several code reviews and several pull requests they finally accepted all changes so that now Wasm extensions are supported in Envoy on S390X you can read more detail by clicking this link. So for now almost all features that are enabled on X86 are supported on S390X but we can improve something if we enable M-script on S390X then we could build and run all tests from on Master Envoy repository including building Wasm in any language and the next improvement could be providing real fix instead of walk around for NTR for parser and also contribute to Envoy open SSL implementation and monitor the development of Embed Mesh which is a new proxy in Istio organization and there is no working version currently but it is still under active development. Okay so that's all now almost all features are supported on S390X but the permanent walk is required because every day a new changes are being added to Envoy and to its dependencies and potentially any of this change could break S390X build. So thank you for listening. So if you have new questions you can find me later.