 So Let's say you read it in the title a mirrored trading. What mirror trading platform? So when I first joined that project I had no idea what that all was about and well that that's also mirrored trading probably on that on that picture but What this talk is about is actually well Doing financial transactions. So let's say you have a trader Who has understood how the markets work or who's lucky trading? stocks for example or foreign exchanges currencies and Well, he has a clue about all that stuff and he knows how to make his balance on his bank account higher over the time On the other hand you have people well who have money Spare money that they would like to invest. They have no clue how to do that and Now we got this fancy little black box the mirror trading software that can actually bring those both worlds together One guy or an algorithm or whatever that could be That knows how to trade the markets and on the other hand people who would like to invest and who would like to tap into those trading decisions and Apply them to their bank accounts And So hopefully after all When they tap into those signals their account balance will also rise over time and all those guys People are happy. All right First thought I had one well Thinking about financial software But I need to add we're not talking about high frequency trading which is the totally other league But it's still like latency is important And when we receive an order, it's important to make sure that it reads it receives all the clients In a timely fashion without any delay. So I also thought ruby performance May turn out tricky, but fortunately well, there's j ruby and As we will later see that works pretty well So you may remember that magic box with a question mark on one of the prior slides So what's actually in this magic box? we have well a broker that we would like to talk to connect to which is managing the clients accounts and those accounts May not be only on a single broker, but there are multiple brokers that we would like to talk to who just managed those depots and and accounts Um We connect to those brokers via a given Protocol, which is kind of an industry standard, but a little cumbersome to implement right, which is called fix It's not very complex in the wire format, but Well to get all the details right is tricky and you need to have many data models to get that right Unfortunately, there is a existing implementation in Java, which is called quick fix J And we tap into that implementation in order to avoid reinventing the wheel And Well those those boxes here, which we call connectors actually adapt The broker specific communication to what our core system which we will later see understands On the other side here, we have the signal providers that supply us with signals that actually will Are manifested trading decisions buying a share selling a share trading a currency pair and We also get those signals either via fix or via a rest API that actually goes through a real stack Well, we need some component to bring that stuff together, which I'd like to call the core This is a set of state machines that actually Yeah transform The signals receive into actions that we relay to the brokers on the right-hand side of the diagram so far we only have a set of isolated boxes here, but we need some means of communicating with all those components and We took the decision to use a rabbit MQ instance. So everything we have in place is message based so those state machines receive messages and Advanced the state based on that message content and Yeah, this is For the signals unidirectional messaging or bidirectional messaging for the rest so far we haven't yet seen a database and But there is one we currently use Maya skill for that But the important thing is that only the core Rides to the database So we can ramp up the concurrency the parallelism of the system Much better much more easily because we do not have a single hotspot By means of the database but everything runs in here Which all is the database which makes parallelism concurrency much much easier One important aspect and I think that only and that also shows how efficient J ruby works for us is Receive a lot of market quotes. So why are this fix? communication protocol we not just get The we not just send market orders and don't just get the result of those orders But we receive plenty of quotes Which means just the price for a given share or a given currency pair Each second so and for our experience that can amount to thousands of of messages per second It's not very viable to store that in a database, but instead we push this data to a reddice instance or more precisely to a reddice cluster and Given our current setup we process thousands of them each second in plain J ruby. So, um, I Briefly touched that so far already, but What actually made us choose J Ruby for that task first and most importantly, I think we love Ruby because it's so much more Yeah, nicer to work with than if we would have built that thing in in Java for example That we actually picked a Ruby was initially a little coincidence because well we thought For interacting with those trading platforms Should we build it on our own or should we use an existing solution to interface and there was this this quick fix J library that basically Provided us with all the basic communication mechanisms that we needed so we tapped into that one and Then the decision to use J Ruby was basically set Later on we we also noticed well, hmm if we use J Ruby It's much easier to to scale out the system in terms of utilizing all cores All CPU cores that we have available So that turned out to be an excellent decision in terms of Parallelism and performance too now running a J Ruby instance or a J Ruby System is at least in our experience a little different than if you run MRI processes We had a given Java background at that time, but we opted against using an app server But looked for other alternatives to make sure that the processes that we run run in a robust fashion Maybe one or another of you has already had the honor of seeing an out-of-memory error and if you have Had that luck you also know that once such a thing happens You well the virtual machine is in a state that you don't want to want to have so you need to take countermeasures And for those of you who haven't seen an out-of-memory so far Just a brief explanation Java virtual machine has a certain amount of memory that it has available for data and once this So called heap is filled up exception start to be raised and Well That can happen and at very inconvenient times So in terms of of robustness We we have an active passive setup in place where we have multiple instances of the back-and-running at a given point in time and only one is elected master using zookeeper that was already mentioned two talks ago and Effectively when one instance crashes the other one takes over But on a lower level We also would like to ensure a virtual machine health on Well per process level so virtual machine can also enter the state where it's busy garbage collecting but still alive and We found that the tenuki service wrapper, which is an open-source component Can effectively monitor for such situations So this is a little Java library a wrapper that can turn your Java process and as such also your jruby process into a background demon And it will monitor the Java virtual machine for responsiveness So it will so to say ping it in regular intervals and see if it's still responding or if it has become stuck for whatever reason and It will also monitor standard error and standard output for unexpected output matching certain patterns so that turned out to be a very efficient means to Verify and to ensure that the process that runs our back end is actually in a healthy state And if that tenuki service wrapper Detects that something has gone wrong It will kill the process first it will try to terminate it regularly and When that also fails it will forcefully terminate it Initially when when we tried Doing that integration it felt a little tricky because there was barely any documentation available at least not for that special component But eventually we figured it out the integration is well you would say a little cryptic But once it's in place it works very reliably So in the end this is well you don't need to understand all the details of that of that Config file. It's just an excerpt But you can define virtual machine parameters On one hand and you may need to make sure that the class path is correctly in place For example, so you need to put the j ruby jar on the class path you need to increase the stack size from the defaults and most importantly you need to provide it with an entry point which is actually Ruby main and That one again gets supplied with a ruby script That boots up our back-end service as a parameter There's one other thing that we needed to Implement which is well we called it service wrapper glue because it's just a little tiny glue component that we use to guarantee Well coordinated regular shutdown Because otherwise the service wrapper would would always forcefully terminate the virtual machine because the jruby Virtual machine the jruby instance inside the JVM wouldn't Shutdown in a coordinated fashion. So we also build a little a little Java class that when The service wrapper would like to terminate our code our back-end just Gets the global tree Ruby instance and then runs tear down on it to make sure that it actually shuts down and that all The shutdown hooks get notified and so on Yeah Next topic I would like to talk about that you also get effectively for free when it comes to the JVM is a jamex, which is a monitoring infrastructure for Java processes and Effectively as soon as you run a JVM whatever runs inside of it you get access to jamex metrics and the good thing about that is that maybe you have an existing monitoring infrastructure in your operations available and Most monitoring solutions do have jamex bindings So you can easily use that to tap into statistics about the virtual machine. How much heap is currently used? What's the CPU utilization? How is the garbage collector doing and so on? Furthermore, there are also libraries that you can hook into jamex So that for example the quick fix j communication that we're doing also Exposes its statistics and its health date as as jamex If you don't want to set up a full-blown monitoring infrastructure, which I could perfectly understand you can also Just have a peek into jamex monitoring if you bring up j console and there is this this ambience tab and Under that you have well, it's actually a tree of the different information categories that are available and Below that you have all the metrics exposed But some of those categories also offer operations for example for the broker connectivity We have we also have the possibility to use that jamex interface to reconnect a broker or to disconnect a broker or whatever There's also more benefits that we had or that we have from using j ruby Which aren't so closely back and related for our front-end processes. We use passenger with j ruby and Again, it turns down to well we get much more performance by utilizing parallelism Because we can run fusion passenger in threaded mode and have pure JVM's running Because we are actually well our memory footprint is quite quite large So we benefit a lot from being able to spawn a few JVM processes But get our parallelism through Multithreading inside of those processes Well, we also had a couple of issues Some of them are j ruby specific some of them are not some of them are already solved fortunately We we have a constant struggle with virtual machine startup times But we came up with a pragmatic solution that we have Commands such as just running plain rails migrations active record migrations for example that don't need j ruby So we just use MRI for those And have well an elaborate set of aliases that for each task that we would like to run Brings up the right Ruby interpreter memory usage is also well we could improve on that and Well debugging fortunately we tend to use Ruby mind for development and there there was a buck where you couldn't inspect The attributes of objects, but fortunately, I just noticed that it solved in one seven twenty We haven't yet had the time to upgrade to that version, but that is well Very relieving to be able to inspect your your processes date again From time to time we hit bucks where j ruby just did things differently than than see ruby such as big integer arithmetic or divisions where See ruby returned not a number and j ruby through null pointer exception or whatever But in our experience the the support by the j ruby team was really great and those bugs were were very quickly fixed in the past Sometimes when you get one of those native stack traces you're also confronted with a with a fact that they are well pretty pretty lengthy and You have well You need to decipher them to figure out what was really going on because you see well the all the AST that has been built as part of the stack trace Yeah fork I'm not sure whether it has been implemented in your versions, but one dot seven also doesn't come with the fork implementation so we need to fall back to to see ruby for that and That's not sure ruby related the last point But we had a very ugly bug in the zookeeper master election code where we all of a sudden had two masters Which is well, not what you would like to have But overall coming to a conclusion Through j ruby ruby really does scale for us and Contrary to what I would have expected prior to joining that project release scales and is a very very good choice for implementing a complex backend system Mainly due to the fact that well we get the concise and elegant syntax of ruby, but we also get the complete Java ecosystem Meaning we can reuse what's already there and what has been built don't need to reinvent the wheel and We also get the JVM with its just-in-time compilation and all that stuff and Well, while we don't do high-frequency trading performance still important for us, but I would like to remark that Don't be frightened by a micro benchmark performance figures in practice When you hit performance trouble, it's Because you're waiting for IO or whatever or your database. So it's Factor of two in the runtime isn't that important because well it It's left behind by by IO usually well So that would be my talk are there any questions