 Okay, good morning everyone again. So I'd like to introduce you our next speaker Miguel Sanchez de Leon Peque Who's a software engineer at OpenSystemS, a medium company based here in Madrid, and he's gonna be talking about automated trading Thank you. So welcome everybody to this presentation on Python for developing a real-time automated trading platform I'm glad to see so many of you here today First of all a little bit about us This project has been developed in OpenSystemS Which is a company based in Madrid and in particular in the research and development department or division which is led by Fernando Monera A little bit about me as well My name is Miguel Sanchez de Leon Peque I'm an industrial engineer with backgrounds in electronics, automatic control and computer science And I feel passion for programming data statistics and machine learning. I started programming with Python About two years ago exactly when I started working with OpenSystemS and since then I've been gradually falling in love with this programming language So what's the motivation behind this project? If you go on the internet and search for currently available platforms for real-time automatic trading You may find things like trade station, meta trader, ninja trader And in fact many many others each of them with their own little differences. However, they all share some caveats So for example, they are all proprietary. They are very heavily desktop oriented So many of them cannot even run if you don't have a desktop environment Do they provide Linux support? Well, no and most of them they do not provide non-windows support And the most important questions perhaps Can I use Python to create my strategies and the answer here is a bit no So what we wanted we wanted to have control over our own tool So we needed to have the source code to be able to modify it at will We wanted to have an optional graphical user interface And we wanted this tool to be multi-platform so that we were able to write in at least in the main operating systems and We wanted it to be programming Python, of course So is this possible? That's the question So allow me to introduce you to OS markets OS markets is a broker independent platform for real-time automatic trading with broker independent I mean it should be able to work with any broker available out there as long as of course the broker provides you with Convenient API to interact with It is implemented over OS brain. We'll see later what that means Okay, but for nothing it is sufficient to know that OS brain is what it's called a multi-agent system in which different agents or actors Run independently and communicate with each other using message passing. It is designed for real-time automatic trading So that means it must be fast which it is and it must be also robust This is another view of a system or architecture of OS markets as we can see the input of the system It's just the raw market data coming from the broker That means the updates on market data or market prices and the output of the system is of course the buy or sell Operations which are sent back to the broker We'll see What those little circles and theirs are the agents or the actors in the multi-agent system and The arrows in between represent the message passing The names inside the circles are just the roles that the agents are playing because normally agents are Specialized in a certain role that they play in the in the overall system So we'll start describing these agents. We'll start from the left. The first one is called a feeder Oops, I'm sorry. The first one is called a feeder It's only purpose is to get market data from the broker that means this agent cannot be independent From the from the platform cannot be platform independent So we must implement different feeders for different brokers. It is also multi-threaded agent We'll see later what that means but this particular agent useful needs to deal at least with One thread that is getting real-time market data dates and another thread that is able to attend requests from the agents in the in the system for for example historical data Just that they're on the right of feeder we can find router Router is finally a broker independent provider. So it's kind of like a feeder But in this case it's broker independent Its purpose is to store and distribute this market data among the agents in the system in OS markets So the rest of the agents can subscribe to new updates But that come from router It is also able to update the market data that it has available in memory with All the information that is coming from the feeders and also is able to resample data that is necessary Market data is implemented using non-py arrays Perhaps the main market data structures are the ticks which are simply the ask and bid prices at a given time stamp The bar which is usually an open high low close prices during a time span and Then and then the bar series which is simply a series of bars sorted of course by the time stamp As you may have noticed the bar series are updated in real time So in order to avoid full memory copies on each update what we do is Just use a view of an amp array a base array or a buffer Which is the one that we update with new bars So if a new bar comes we just update the base array and then which simply change the view Which is a much faster than performing a full memory copy On the right of router we can find the brains The brains are the most commonly common agents in the system Are able to subscribe to router in order to receive updates on market data and are able to subscribe also to other brains These agents run the actual algorithms for automated training. So they are perhaps the most important Agents in the system. So they must be able also to send orders like buy or sell and Well, of course today, I'm not talking about the algorithms that we have implemented in in open systems But I think it's important to note that thanks to the great ecosystem available out there for data analysis and machine learning in Python You can do almost anything you want with these brains You can make use of NAMPA of pandas sci-pi, scikit-learn, theano, whatever you want NAMPA So at the end you can do anything that you can do with Python, which usually means just anything One of the best things about this kind of architecture is that you can create some kind of abstraction In this way just creating a hierarchy in which the lowest level brains are those that are Calculating using the raw market data and then as we go up in the layers. We start treating concepts more which are more abstract or more Like how a human trader works so we can start talking about a marketing text intention or market the structure or detecting some kind of patterns And the last agent that we have here in the overview is the one on the right Which is the trader which is similar to the feeder. It means it is broker dependent, of course And it's only purposes to handle orders, okay So buy, sell orders coming from the brains and then it executes these orders against the broker So just to make it clear Here's an example Imagine that we have that system in which we don't have the bull brain in there and then at some point We spawn it we can do that in real time and then this brain requests some Historical data to router router may not have this data available So it will send the request to fear the feeder will get the data from the broker And then we'll send it back to router and from router back to the brain all in our sink in an asynchronous way Okay, so from that point the brain may start Subscribing to a real-time updates from the from the router and then performing some Calculations in order to generate orders that will send to trader and also it may start publishing Some information about the things that we have calculated within this brain and all the brains may need this information Or make use may may be able to use it There are other agents that we have mentioned such as logger to log error or warning messages the informer Which we can use to publish information about the accounts balance or the other mind Which is kind of a console to control the agents that are running so we can kill some or spawn new ones whenever we want We have also developed as I say a graphical user interface this graphical user interface is For real-time market data visualization and also for real-time visualization of indicators the indicators are just Kind of mathematical transformation of the raw market data This graphical user interface is really good in well integrated with OS markets Okay, and we'll see later how we do how we did this In order to implement this graphical user interface with used Qt perhaps Because it is the most widely used Also because it is really great. It has really great portability, but I think most importantly Is pie Qt graph pie Qt graph is much less known than Qt It's a great tool which is written pure Python and it's a library for fast real-time display And the best thing about this library is that it implements all the user interface all the user interaction interactions or that we need so panning or zooming in one axis or two axis is all implemented and working out of the box as well as updating in real-time the axis So here's another view of the main window in which we can see a list of The currently running brains in the network and from the main window We can create new tabs or new windows with different and independent charts Apart from that we can see how the chart is implemented. This is all pie Qt graph Okay, we can change from here the data that we are displaying from markets So in this case, we have just changed the period from one seconds period to two seconds periods And we have the option to post automatic update of the view and then as you can see we can zoom in zoom out We can see that everything works works pretty well We haven't written a single line of code for that. It's all pie Qt graph We also have this indicators menu in which we can select the indicators that we want to add to the chart We can add them as new plots as you can see so we can visualize indicators Below the market data and we can also add indicators that are plotted over the market data such as for example this simple mobile moving average and Of course, we're able to change the parameters of these indicators also in real-time And of course the graphics view gets updated Of course, you can remove the indicators and Everything all the zooming and panning still works pretty well so It is important, okay to know that this graphical user interface as I said it was optional So it is well integrated into OS brain. How did we do this? Simply because all the charts are agents in this multi-agent system Okay, so when we are visualizing this data, it's simply because The chart is subscribed to real-time updates from router and when we are visualizing those indicators It is also simply because we are subscribed to Brain that is publishing this information to us So at the end all the calculations are performed in the multi-agent system. Okay in OS markets So I have explained a little bit on how we have implemented the graphical user interface But I haven't explained how we implemented OS markets or this multi-agent system. So this is always brain Okay, and it's a general-purpose multi-agent system written in Python This general-purpose multi-agent system implements independent agents that run And communicate with each other using message passing and in particular with OS brain We have we were able to provide an easy configuration and deployment method And we'll see later how we did this When I say general-purpose I mean You can really use this for many other things. Okay, you can use it for not logistic Logistics transportation or even military and defense applications in our case We also use it for distributed computing. So we use these agents that are distributed among different machines and then we distribute Some groups are random selected backtests and then each of the workers perform An analysis and create a correlation matrix and calculate some kind of indicators and then we take into account to Generate portfolios that are kind of optimized So when I say that these agents are independent the question is how I are they implemented, right? So processes or threads. Well, if you have met the jail, then you already know the answer It's all of course processes and if you don't know the jail, then it's better for you because your life will be much simpler I also said that we implemented message passion. How did we do this? It was used in serum Q. There are many reasons But the first one is just because it's a great project. It's so cool So really if you don't have a project that needs it you should make one because it's really fun. I Totally recommend it So right now we have an idea of what a basic agent is We know that it is a system process that runs independently. It implements Methods for easy binding and connecting with different patterns that these patterns are implemented in serum Q So we have the classic push pull or request reply of public subscribe patterns It activates on incoming message This means use this means usually an agent is running on main thread main process which is just waiting or pulling on Input sockets for a message to arrive and that's when we start executing some code in the agent Multi-threading multi-threading may be used as well within the agent and you know that to do so what we do is to create Improved sockets. That's also serum Q implementation To communicate with the main thread. Okay, so we always try to avoid concurrency. So we always use message passing even within the same agent Configuration I said it was pretty simple. So how we know that the agents are independent But they man they must know the addresses of all the agents if we want to connect them Okay, so at the end there may be many many sockets So perhaps creating a configuration file would be to tedious would be too complex We are not interested on that and in fact for many agents or for most of them Assigning a random address is perhaps simpler and more convenient. We just don't care which address this agent is binding to So if only we could just an idea have something like a name server like an address book Where we can register all these agents have something like an overmind an entity that was able to Know the addresses that the agents are binded to in order to send them to other agents that are going to connect To the first one. So is this possible and here's where pyro 4 comes in which is another great piece of software and also Not much well known. I think Pyro 4 comes from Python remote objects and it allows us to treat remote objects as local This is done with an object that is called a proxy and we can use this proxy to call methods That are implemented in the remote agent and then these calls are serialized and sent to the remote agent Executed there and then the return is serialized back to us. So we can treat them just as if we were Dealing with local objects, but everything is being executed remotely. So it's great very very convenient And apart from that already has a name server implementation. So it's a win-win decision So now we know what a real agent is it is a system process it runs a pyro multiplexes server The server actually serves nice an agent object Okay, you see in pyro and then we have a main thread as I mentioned before that runs the main loop This is what is called a one-way call in pyro So as conclusion We were able to implement a general-purpose multi-agent system with Python in which agents are independent and communicate with each other using message passing and We have achieved an easy deployment and remote configuration process using pyro 4 So let's start with a little bit with some code samples just to understand how this works This is the most basic sample, which is a hello world and in this case as you can see In here, which we're just running a name server Okay, a random name server and then in here We're running an agent that we call example This is an alias and we tell them to register in this name server and what do we have here? Okay, this variable is actually a proxy. Okay, it is not the remote object This function will start a new process in the system But that by default with bind to local host and then with this With this proxy we can call methods that are implemented in the remote object So we can call for example the method log info, which is just an information login and say hello world Okay, when we do this this call log info hello world gets serialized gets sense It's been sent to the remote object the remote agent and the agent will execute it Okay, this is a second example, which is a simple one as well in which we are going to implement push pull pattern Okay, so in this case, we're doing just as before we're creating a name server And then we are creating two two agents a sender and a receiver. We call them alias and bob and then What we do is to bind the sender Using the push pattern, which is again a serum q communication pattern We give this socket an alias Okay, simply because it's easier to refer to it later that way And then what we got where we get in return is the address the actual address where? The sender binded because we didn't specify A port for example, so by default it binds to local host and then it will select random port because we just don't care So this call is executed in the remote agent, but then we get this variable here in this script Okay, and then what we do is again to serialize this call We're going to tell the receiver to connect to this address So this will be serialized and we are going to serialize the address Which is a local object and then we are going to serialize as well a function a function that will be used as a handler This handler is implemented in here and it's really simple. It's just as before What we are going to do is to get this message the message that we received and simply lock Some information and we will say received and the message So once we have done this We're going to start an infinite loop and then we're going to start sending message from the sender So what would what would we get in return? Okay in the console if we executed this we would start seeing messages saying received hello world received hello world received hello world Okay, this would be another example in this case of in this case of always markets In this case, we're going to define class Okay, that inherits from brain because we can also use object oriented programming if we like to So, uh, we're just telling this agent that on new bar. This is the behavior So if the close value right now is higher than before we're going to buy or what when it's going to sell And then we send the order. It's pretty simple and it's stupid and don't do it with your money, of course And here what we're going to do is to create an under architecture. This call is We'll simplify the creation of all the common agents like the feeder as well the router The trader they will all be created and they will all start working with a anda or against The anda broker so we just have to provide our account information And then we tell the system to start streaming real time updates on euro dollar pair And then we can add a brain to the system in which we specify the class So we serialize again this class and this class Will be used to create an object in the remote agent in this case in the remote brain and it will be served by pyro And that's it Then we can just call the subscribe method which will make This brain the brain example that we just created subscribe to Real-time updates from router on the euro dollar pair Based on one minute period And that's it as a final conclusion Python has been proved to be a great tool for implementing a real-time automatic training platform And one last thing I'm really happy To communicate that always brain has just been released You're the first to know in fact outside of the company So you can find this multi-agent system in pi pi and in git cap if you want to check the source code And of course you can find some documentation. It's a work in progress because We didn't have much time So that's it if you want to contact us you can send an email to me There you have my email address or per non moneta Do you have also some links if you want to visit our website or our blog or if you want to follow us on twitter? Also, we are hiring so if you like python or if you like trading Or if you don't like any of them, but you still like Data analysis or big data or database solicitation or web integration Or anything that you find in the world we do in our website Then do not hesitate to send us your your cv to our hh at opensystems.com So that's it. Thank you very much and if you have any questions comments or suggestions Thanks, miguel for this great talk. So we have the time for several questions and May I just ask you that we are recording everything so please wait for us to give you the microphone Um, hi. Thank you for the talk. Really interesting. Um, just How does pyro 4 deal with? Exceptions or errors like it's a trace back send back to the color or it's just silent in the Agent that it's being called. It's not easy to send a trace back Like the full trace back to serialize it. So no you just Can get the message with the trace back So you can use the trace back dot format x format exception and get that information if that is useful for you But uh, of course You usually have agents that work well because you have tests that validate their functionality Without the need to run it remotely, right? So you you write the tests locally you validate the The agent and then you run it remotely. But anyway, if it crashes, yes, you get some information Okay, but not everything of course Pyro uses by default well at least in always brain always brain uses by default Pickle for serialization and can use also deal If you are dealing with the proxies, okay, but still that's that's not easy, I think Hello, and what is the performance of this how many transactions can you do in a second or Because python it's considered to be slow So you use c python that is the standard interpreter. It's not we use c python Yeah, we use c python because uh for us it has been proved to be sufficiently fast for real time Of course, that depends on your application our performance is limited to the performance of py serum q which is Python bindings for serum q which is the message passing that is used between the agents I mean deal and pyro are Kind of slower, but we only use that for deployment and configuration after that All the message passing is used is done using pickle for serialization and serum q or py serum q for message passing so That will be our bottleneck. But anyway, uh, the bottleneck is always With the internet so against the broker It's always much slower to get the messages from the broker and then to send the messages to the broker through the internet Like by that could take I don't know 50 milliseconds But anyway for real time, it's More than enough Hi, so first, uh, thank you very much for the talk and I would have several questions namely, uh, historically This kind of problems Were handled with tools that are labeled as c e p complex event processing And a lot of companies have invested a lot of money To solve that problem you have storm from twitter, which was also open source you have the lambda architecture from linkedin samsa spark and gazillion tools And i'm just interested why did you feel that you have to invent something similar? and which is like Very similar to the existing tools. That would be the first part of the question Well, I may disagree with that. They are very similar This is completely written in python. That might be an advantage. It is an advantage to us and we like that And this project of course has evolved Okay, we didn't have this intention. We didn't want to implement this at first, right? We started simply using serum q and separated processes But at the end we find out that uh, these tools Could be glued together and work pretty well. So for example the use of pyro. I don't think you can Do things like that with those tools that you are mentioning and also while having a full python implementation Yeah, that's a great answer The problems you mentioned are of course Solved in a different way in different frameworks But okay. Yeah So the other part would be um, I didn't notice that you would talk about the edge cases which are like very important in um frameworks like this and Especially if you're playing with money. So I'm talking about things like, um, how are you guaranteeing that the events are processed exactly once and um, you know, um If you're connected to uh, several data streams How are you really sure that the events that are coming in are really ordered because it could happen that you get from Wanda Some really what sorry? I really Or ordered like one after another Because it could happen that you get something from Wanda at time t but because of uh latency issues or whatever you get from Dukas copy or whatever exchange an event which uh happened before that but later So traditionally people used something like Kafka which solves your locking problems in distributed systems Um, because of course, uh, traditionally you have a lot of smaller systems Which wants which wants to access the latest state of the system So you have the locking problems. That was a long question. But anyway, yeah, I agree. Thank you Um, yeah, the thing is that with serum q and that's what I like about this architecture. You are free to implement your Message passing. So you mean how can you be sure that you only receive the message one? Well, it depends on your application is if that is important for you, you can implement manually those patterns for communications For our case in fact, which is just publish subscribe because in real time We don't care even if we don't receive one tick once which is something that Doesn't happen, but we don't care We just have a flow of data and we want to process it as fast as possible if there is Little wrong data or some missing data. We don't care. We just want to keep Working. Okay. Normally that would that won't affect your strategy So you're not afraid that you would make Based on different data the same decision twice because your workers would process the same event twice or three times And you would then basically go the same But as night capital or something like that. Uh, can you repeat the question? Sure, so you're saying that you're not afraid that based on one event The workers would process this process it multiple times and no that that won't happen Sorry if you understood that no that won't happen because we are only sending Sending those that that data once okay the worst that can happen is that we miss a packet Okay, a message once we didn't care about that because usually you just have a tick So maybe your bar will be formed a little bit different I've lived a little bit different But uh, that's not really important. In fact, when you change brokers the data that you get from those brokers is also different, so It's not important if the data that you get is uh, one percent different which is which is not is We don't care. Okay, so that's the good thing about this that we can implement Flexible not very strict message passing such as publish subscribe that works pretty well and pretty fast We are more interested in speed Than in for example getting all the exact data that brokers is providing us because we don't trust that data anyway Not that much Okay, it's first. Thank you for this wonderful talk and my question is That in such multi agent systems, we have another problem is for example, that's one of the processes may die unexpectedly and so how to monitor this overall system and What is be the reaction if for example some agent dies Yeah, of course monitoring is also implemented By the developer the developer so that depends on your application We are we are monitoring our agents and our brains and one of the best things about this architecture is that exactly one brain For example dies crashes, which is something that should never happen and never happens Then uh, you can kill it and you can respond it and everything works as expected And why is that and that it's again thanks to serum q because uh, for example You can bind. Sorry. You can connect to a to an address that is not binded yet So there's no problem. There's a client that is connected or subscribed to a brain that is polishing information Then this brain can die We can kill it. We can create a new one and it will start receiving the same information It won't break anything around. So agents are completely independent Thank you. Um great talk um Numpy arrays uh, numpy arrays. Why why did you choose those? What features? Were you looking for um, that you couldn't find in uh, in basic types? Uh, that's that's one point and secondly I'd agree with the uh, the last two, uh, questioners about uh, such a complex system, uh, you've got an agent going down coming up Uh, the discontinuity Um worries me greatly about especially when it's handling finance Um, sorry. Can you repeat the question? Can you speak a little bit slower? Sorry, uh, the the question was why numpy arrays? ah, okay, and also, uh Reassure me about uh discontinuities nodes popping up popping down Um, you've talked about pickles. Uh, you've talked about pickles over the internet. This is scary Yeah, no, I didn't talk about okay Uh, I'll answer the second one first. I didn't talk about pickle over the internet It is true that is this system is completely unsafe by default So it is meant to be executed in a control or restricted environment for example in a in a cluster or just in your own computer If you want to communicate, uh between agents in remote machines and you have to pass over the internet Then it is up to you to implement the the security So you may want to use a vpm or an ssh tunnel, but of course this is not secure Yeah, it's only secure if you have programmed it and uh for the first question why numpy arrays? um The answer is that uh all the ecosystem available out there for data analysis not all but Most of the ecosystem seems to work well with numpy arrays. So Why wouldn't why wouldn't we use numpy arrays? Okay. Well, um, why wouldn't you because they're more expensive and uh I was wondering what features you're after but don't worry about it Uh, the the the idea that a brain though doesn't use your own dsl, but actually is just pure python code I don't see what your product offers besides, uh, a collection of already existing frameworks for Creating a large System, uh, I'm struggling to see I'm sorry. I can you ask again the question a little slower? I have difficulties for hearing you sorry um You don't have a dsl A domain specific language, which abstracts away all the bad things that you could do as a developer when writing a brain So what does your product offer in terms of stability? and predictability That the components will work together as a system because I don't see any yet No, what would you just implement test of the independent agents and then of the whole ecosystem and that's it I'm sorry. There are all the questions you you get out of kind why you are not using these on that technology But my question to you is about that My question is about twisted because it is python and I would expect to be good in this kind of situations and You are not using it. I guess you have a reason why So as I said, this has been evolving Uh in the in the past months So we didn't start with the idea of creating our own tool, but that just happened I mean we liked serum q very much I don't know if the tools that you are mentioning implement this flexible low level Communication patterns or not and also I don't know if they implement some things like pyro. I guess we could implement another tool Using that That framework for example and then using pyro as well for the configuration Yeah, maybe maybe that works as well higher again, so I'm very interested. How are you Thinking or how are you approaching? testing this framework because as you already mentioned it's a composition of separate agents and I mean, how are you approaching the testing and How are you testing The whole system. Are you just using end to end tests? What kind of tests are there? Any tools you could recommend so what's your experience with that because I think it's Very a very different problem Then what we are doing, uh, I mean most of us here probably Each day I mean we know we are aware of the tests for the websites and things like that and This kind of problem looks a bit different. So I would be interested in how are you testing this beast We are testing this unit using unit test and py test and That's it Anyway, I'll be happy to talk to you and I'm sure you will explain me better tools for doing this Hey, um one quick question In finance trading you Your brain must know a current state for example all all your open positions And I would like to know how you save this state and how you replay What needed to know the state and if for example A brain dies How you still know your open positions Well your open positions are usually and it was stored in the broker But of course we store it locally as well. That depends on your implementation. You can make trader store That information or you can make if you're going to perform to implement some kind of We are handling of the orders you can implement it in the brain Anyway, we've got as I mentioned, I think before Other agents such as the informer which are publishing information about for example the account state Or the rates are which you can trade your account's currency against other currencies. So In this in this sense is completely flexible. I mean it's it's up to you Great. Thank you to all of you for asking so many interesting questions And please join me in thanking all the speakers from this morning again