 Okay, so this is about the history, how Linux came to be used in the world of finance. And that's kind of a long story. And we're not talking about 780s here. This is 780s, of course, used in the finance industry, but these are specialized arrangements that are made to interface with exchanges. So in many ways, Linux is the dominant force today in high-performance computing. And that is actually a different area. And hand health. And so we have both the low end and the very high end of computing at mostly 100% here. But it also runs the world of finance these days. And so finance consists of the stock markets where they use Linux to match the trades. And there are special quants out there, investment banks that implement complex trading strategies and try to make money either of the customers, the investment banks, or of trading on the stock markets like the quants do. So then there's this classic stuff like customer relationship management, supply chain management, and all the other things. These are also all migrated to Linux these days. And so I'm going to talk a bit about the history, why these decisions were made, and how the industry evolved. And so we have a variety of participants in this area of finance, especially the stock exchanges. The stock exchanges get fees from the companies who trade on the stock market, and they want to provide an equal playing field for all the investors that they all have equal access and equal chance to buy all these symbols. So their interest is not so much in efficient or high-speed trading, but is in fair trading. Then we have the banks. The banks do trade for their investment portfolios and on behalf of customers. And their main interest is to get the best price. And typically the banks have a price differential between what the customer pays and what they pay on the stock market. And their interest is to have the most profit on that level. And today the corporate giants like Google, Facebook, and so on, they all have these days integrated finance operations that also directly interact with the exchanges. And mostly that is used for international trading as well as to preserve the value of their stocks and to manage the wealth that they are accumulating. And managing that with banks is something that they wouldn't be doing. They won't have their own departments that deal with this. Then we have small investors that have some software on the machines at home and they interact with the exchange and they think they have a special trading arrangement. Usually they have some kind of a niche knowledge where they can make money of these things. And their interest is to have fast access to the internet and be competitive with the big guys. And they are professional quants. Those people have a lot of money and they have sophisticated operations. Usually they have equipment directly located at the stock exchanges and they want extremely low, low latency and they want to have an edge about everybody else. And so these things, this is also an erase between all of these to get the best prices. And typically I think the professional quants win because that's where they make the money and typically the other ones try to come close to them but they are not willing to make the investments that the quants do to get that low latency. So where does this happen? The most important exchange is Chicago, New York, London, Frankfurt and Shanghai now. This gets bigger and bigger. So there's not a lot of professional operations starting up here and a lot of the classic entities from the rest of the world are trying to enter China now and also do these trades here. So how does this actually work? If you have a stock exchange, the stock exchange has a matching engine. There are the orders of the buyers and sellers float together and they are matched and when that is done a trade is occurring. And if a trade has occurred then the fact that the trade has occurred and how it has occurred as well as maybe the change of the value of a stock symbol is described by the market data distribution mechanism. This is usually a multicast group and so they use the hardware to replicate messages about this information and the information is sent out to various business entities that then may react to the event to the change of value or to a trade that they see occurring and they may send a trade back to the exchange in order to do something else maybe to make take a profit now because the value of the symbol has changed. So they have another connection which is usually TCP based where they contact as a system on the premises of the stock exchange and then that front end processor sends the trade of the entity to the matching engine and another trade may occur and the whole process starts over and over and over and this is usually going around all the day when trading is active and these entities are part of this trading cycle and they are mutually influencing one another and they drive the value of the symbol back and forth and they're trying to get an advantage out of the mechanisms that are active there. So and the technology sophistication of these various entities is different and so because there are different objectives here that they have and also the stock exchange as I said before since we have a small group if you don't want to if you don't understand something or have a question just speak up that may be the best if I talk too fast tell me that too. So what kind of technologies are being used here and you didn't have intelligence and speed for intelligence you can use something like Kubernetes or an HPC cluster to do data processing and then you need to add the exchange you if you're connected to the exchange and you're in this closed loop you want to react as fast as possible because you want to have an advantage and catch the chance of making a trade by being faster so we have powerful and fast processors people overclock these things sometimes game processors game rigs are used with water cool processors and very specialized designs they usually have very high capacities in a fast land and also some people correlate these symbols within multiple exchanges so they have high speed wide area connections between different cities where they can get information faster to the other point and therefore react faster if a symbol changes for example in New York and it affects will affect something in Chicago and so what they also do is they want to write very efficient code so typically cool is very much stripped down as bare metal it's very limited and just focused on this one purpose and it's extremely optimized and so there's a lot of Linux hot rolling going on in that area and a lot of kernel bypass direct interface with the hardware and so and so and such and such to increase the speed and gain an advantage on the other participants in this trading loop are you following me okay so how does this happen how do we get there so in the 70s we had this paper trading you had to file file a form and stuff like that and everything was going manually via hand symbols on the stock market in the 80s and 90s we had MS-DOS first the first PCs came about then Novel created the first local area network and then Sun and proprietary Unix has showed up and they were used by the stock exchanges as well as by the traders to do the initial trades which were very slow and so in the 90s also the 10m system were created this are specialized dual safe computing systems that are still being used today in some esoteric configurations because they're very secure the traditionally this was demonstrated at the exhibition by taking a shotgun and firing it into the machine and that the machine survived that and was continue able to continue to execute and with that they got most many of the stock exchanges to use the 10m systems in the 90s Microsoft developed new layers in the operating system to facilitate messaging like use in the stock markets for example they created a message API for this multicast distribution for the marketer distribution that's used by the stock markets and so Microsoft became a good platform to use for trading and in the around 2000 the Sun systems and other proprietary Unix this became very big in the trading area and they took over Linux was used at the fringes for some side processing but nobody really trusted that Linux could do the job and there was some fierce discussions and disagreements in the tech departments if you could discussing if a Windows system is better or that Linux and vice versa and there's histories of how tech departments being fired because then owners got mad at them and a little horror stories from that area but in 2007 2009 the wars were won by Linux and we saw from that time on we saw widespread adoption of Linux as the trading OS of choice first this was by the traders that had that were able to hot rod their unixes and they jumped on Linux because they could modify the source code and it could optimize the operating system as needed to be faster and they gradually the proprietary unixes where we're treating and around 2017 most of the proprietary unixes have been vanished and Windows is relegated to the desktop and from that point on nothing changed anymore and we have total domination of the stock markets by Linux so the individual traders went first and then the brokers came next because they couldn't compete otherwise the trading funds upgraded and the last one was the exchanges took probably five more years before the exchanges would move and that is mostly because they had to make sure they wanted to be fair to everybody and they were very afraid of doing any changes and so they gradually introduced the technology and they hardened it and they were very doubtful about this until one or two switched over and then finally the rest felt they couldn't avoid that and so finally everybody today is on Linux so what kind of technologies are we seeing in this area first of all the kernel optimizations a lot of the traders the quants they hacked the kernel and they do modifications to the various kernel layers that was going on initially then there are offload API in the kernel this is rdma API which allows direct memory to memory transfer between different systems in a network that was used to reduce the latencies and to avoid the high latency of the network stack then some people started using real-time kernels because they felt they were more efficient which caused some discussion because in fact actually they're slower and then the term real-time was really fine and now we have a different notion of real-time in the kernel that actually is faster but it's not really the academic definition of real-time anymore and then they found out to optimize the binaries as well so special compilers were being used hand crash assembly code in user space and then there was a discussion about the methods of what languages to use first there was java and use basic even in the earlier phases and the universal agreement in the industry after all these years is that c++ is the best one because you can load a code in this and most of the code bases that I know today are now on c++ in the industry there were numerous network enhancements being made there was a multicast layer added to the kernel and the multicast was very much improved because that's key to market data distribution that's used throughout the industry and everywhere and so today we have excellent multicast support in the linux kernel and what we have recently is the general skepticism regarding linux at all we don't want any software in between want to interact directly with the hardware because that's where we can be fastest and so there's a tendency these days to just write directly to the hardware and there are specialized development kits from intel as well as from the storage companies where you can bypass the kernel and can directly interface with the registers of a device and can bypass all the inefficiencies that you may perceive in the operating system one of these problems of the industry is the speed of light problem you want to be fastest but there is a physical limitation of how fast you can communicate information and so the problem is here that you cannot really go faster than the speed of light and so there's a limit to where you can go and there are various measures to get you closer to that limit and the closer you go the more specialized hardware you need the more uh sacrifice you may need to make in terms of cost so there's a this exponential curve as you go closer to it and so it's always depends on how much money you want to spend to reduce the latency it takes about hundreds of many seconds to send a signal around the earth that is for example the problem you cannot simultaneously update things globally and today the turning cycle of an exchange is much less than a millisecond so it takes hundreds of milliseconds to go around the globe so there's a mismatch now between what you can do locally in a very fast way and what how the signal propagates globally so there's a huge need of kind of fast communication between various continents because there is a huge time difference so if you can have an advantage prior to somebody else you can make a lot of money and so that's this led to various optimizations making the fiber straight not using fiber at all because fiber is slow slower than the signal over the air and there's rumors of people trying to use blimps to uh bone the signal off it some people have been trying to use the moon to bone the signal it's all sorts of weird things some people discussing uh neutrinos to shoot through the earth and detect them on the other side so there's a lot of adventurous thinking there but some of these esoteric things have never gotten anywhere so and we have a basically a transition here from manual automated trading manual takes hours to arrange and now we have milliseconds and stuff like that but this is probably um too much so um and too slow I have not have much time left so the how why was the linux used for this um there's a specific dependency whoever gets there first so you must be able to control your tech if you can control your tech and use third party tools and they make a change that will increase the latency then you suddenly your software broke and you can't use it anymore so there's a tendency to just have all the software in house open source is appreciated we can see in it we can modify it and we can make optimizations how can you be the other guy the other guy is doing the same thing and so you say okay we have more intelligent developers you have better developers we will hot rod this thing to the hilt and we will be faster than you because our guys are great um so that uh also and then we have to promise with the high latencies in the in the windows network stack that caused the move to uh linux because uh there was a lot of people analyzed the latencies of trading and they found that they had high latencies in the windows network stack they've also found high latencies in the linux network stack but these guys got active and contributed to the linux network stack and changed it so suddenly the linux network stack was better and uh windows didn't change so that kind of dynamic caused a pretty energetic move to linux and uh windows microsoft rarely updated that so after some time frame the differentiation became a factor of 10 also and it was not possible anymore for anybody a reasonable person to use that so the customer here of linux one out and the other thing is most internet services these days use linux so it's natural to use uh linux for the same purpose because you already have all the mechanisms for networking there in a sophisticated way and they are stable because otherwise the linux systems couldn't be used for web services or for routers for everything else that's out there and then we get this uh adoption of the uh exchange of linux and which that's also then caused a lot of uh the hang or left a lot of companies that were still hanging on to uh convert the systems to linux because they saw okay the exchanges are doing it and they feel it's safe and so I can now also use my linux systems and uh one of the last competitors out there was sun solaris and some of the exchanges had out to the very end for this and but after oracle was bought uh sun uh that was a sign for the okay they lost they are not profitable and they would be going away and at that point we saw the uh very fast adoption of linux for the rest of it and the sun systems were kicked out very fast because everybody felt they had a liability on their hand and they felt that their system was not stable and could be uh subverted by any change that oracle would do any minute now so the control issue was there with linux they had control of the source code they had a reliable operation there's nobody who could could take it over and mess around with it and there's still some minor uses of ax and some bsd's in the industry in some niche corners I think and that may not be going away but the the trust that people have of ibm is much higher than the have of sun and and bsd is also open source and people also have no control issues with there because he has a source code as well so this remains so but what's being used in the distance of distributions mostly red light enterprise linux sometimes in older releases because older releases have smaller binaries they perfect better and the caches of the CPUs they run faster so people are very hesitant to upgrade and in europe the european stock exchanges use susan and then there are various real-time versions of both that are used mainly by the exchanges some have been using gen 2 because with gen 2 you can recompile your binaries and you can make them smaller thus you have the beneficial caching effects and the binaries will run faster if you just remove all the features that you don't need and so that this was observed by various exchanges various companies and gen 2 is very is a favorite of the industry and then there's ubuntu and debion the reason for that is to evade the commercial entities like redhead and susie because at that point you interact with the large company and then you have the rooms of the large company right with ubuntu and debion you have a much wider availability of all source code and you have direct influence over the release of it if you can get hold of a maintainer or if you hire a maintainer so ubuntu and debion is prevalent especially in embedded systems debion is very prevalent that makes you buy buy a build a very small system that's dedicated to a certain function debion is mainly used so the challenge is for length and finance open source contributors are rare everybody thinks they have the secret source and it's very valuable and they're not giving it away so if they hack the Linux kernel they will first make sure that they can keep that advantage and they don't want to publish it the technique that i have developed is okay let's hold it but if you see the competitors trying to make a move to change it in this kernel on that level you dump your stuff in there and you show it's better and you kill the change that the competitor wants you to do because now he needs to change his software not i don't change your mind so the involvement in upstream development is pretty limited and most of the developers are constantly seeing regressions and various kernel components of course high latency and so people tend to stick to all kernels and use them as long as possible so that then resulted to the perception of the operating systems in the way so the recent development is that okay we are going to restrict the kernel to a certain amount of the system the rest of the system is going to be free of the operating system and we're going to write software that directly interacts with the hardware so we're using dpdk to interact directly with the networking and we use spdk to interact directly with the storage and that way we can remove and bypass all the operating system less and can directly go to the hardware and yes so often they think of the overhead the OS over it's impinging on the in an uncontrollable ways and like they don't know where certain latencies come from and if you have everything under your control you can avoid that and actually the the operating system has been modified these days that it can dedicate certain cores of the OS to trading logic and the OS will stay away from that and will not do anything on these cores so the cores can be fully used by some direct hardware stuff that you may want to use that is part of the standard kernel these days so i think you can't get away from linux anymore and then they're so wedded to the advantages that i don't think any of the classic other platforms will have a chance there's so much effort and so much brainpower accumulated in linux these days with the linux networking layer as well as a bypass mechanisms and everything else that it is very difficult for an alternate platform to become viable at all and i think what needs to be done is we need to find some ways to encourage contributions from the financial world to make this more easy for everybody to add their stuff and to actually get more enhancements to the north kernel that was it any questions any comments yes yes were they is that exchange successful in eliminating an arms race or do you know do you know what what happened next in that story like are they well it's just about the bad the book is factually incorrect he thinks that somebody front one around him no there are very intelligent systems on systems on the network that observe all trading participants and he traded every day at 10 o'clock and the system knew that after a couple of dimes that he always trades at 10 o'clock and it prepared the market for him so that he had the most most expensive thing at that point so he got paranoid about a neural network that observed him and he was suspecting that people had advantages of him but i don't think he got the the thing right it was an investment banker and typically investment banker think they can do what the traders what the quants do but the credit funds know much more than they do and so this was written by a banker and he had some limited knowledge on what's what's going on uh the but he sends out this exchange yes but um some people figured out that this can be actually be taken more advantage of than the stuff if you do the right thing because if you adapt to the intervals you can still pre-stack the order book in such a way that you get an advantage so the answer is still there you just have to modify it a bit any other question yeah hello you just mentioned that the financial company id people stick to the old uh kernel so i'm just wondering uh how they solve their uh security issues because i don't know most of uh latest kernel has the many important uh security patches so if they just stick to the old uh kernel and usually and do they backport the secret patches by themselves no they don't care about that because uh the uh the trading cycle that i showed here uh where is it down down down here in the beginning there this whole cycle is in a private world where the public cannot access this is highly regulated and these guys paid out of money and for this and none of this is accessible from the internet so the exposure to risk is very low and at that this boundary the exchange as well as the traders the protocol everything that's going on so if anything goes on towards they can figure it out and they can debug it but they will not impinge the performance of their systems okay okay thank you so anything else uh then thank you for coming and um if you want to talk with me afterwards i'm still here for the next 50 minutes or so okay