 Welcome to sports betting systems with Acre. Yes, Acre. Before I start, does anyone know who this gentleman is? Brad, shout it out. Johann Cruyff, one of, if not the best players of the game ever. Certainly one of the best, if not the best manager ever. Totally changed football. So if you ever want to learn more, look up Total Football. Ajax Barcelona, Dutch national legend. Obligatory introduction. I'm Trevor Burton McCready. You'll find me all over the internet, some form of think more stupid less. Often think more stupid, depending on how I'm feeling on the day. I work for Lunatech. We are around about 120 people from 30 or 40 different countries. Offices in Amsterdam, Rotterdam, Paris and Brussels. And we're really good. You should hire us. But this is not about Lunatech. This presentation is all about me and my personal exploration of sports betting or a small fraction of it. I was introduced to sports betting by an uncle of mine. When I was too young to legally go into betting shops, he'd take me in, shown me how to bet and going into this magical, illegal world of betting for a 13, 14-year-old kind of completely got me hooked. When I moved to university in London, I got a job working in a betting shop and found that actually the reality is very different. These are desperate, horrible places, full of despair, punctuated by the very odd moment of brightness when someone has a rare win. You'd see people going to pick up their employment benefit check from the post office, come two doors down to us and then spend most of that money immediately because they're just compulsive gamblers. Ruins lives and the bookmakers must be stopped. So I'm going to show you a little bit about how to do that. One of my colleagues called the shop the most divorced place on earth and that was pretty accurate. One of our values at Lunatech is humility. So I'm going to start this presentation with a biblical reference. Bookmaking, betting generally is all about this battle. It's a cheat versus gazelle arms race between the price makers and the price takers. The price makers for a long time have been the bookmakers and I usually have a section, bit of a background about how bookmaking works over rounds and making up a book and balancing things. But with a shorter time limit I've cut a lot of that out but I think it's important to give you a bit of an understanding about prices and odds. So here's an example of a market or a match from a couple of weeks ago. United Emirates versus Bangladesh, Clash of the Titans. I've chosen popular cricket because it's such a beautiful contradiction in terms. But the prices here are exactly the same. They're just different formats. There's decimal odds and there's fractional odds and they mean exactly the same thing. They mean two things, actually. They are the implied probability of something happening and they are also the money you will win if your bet wins. How do we calculate the implied probability? Well, it's a simple inverse relationship. The higher the probability, the lower the price. The lower the probability, the higher the price. If we look at decimal five, it's the same as four to one. If I bet £1 or one shilling or one shekel, I will get five back. I'll get my original one plus four in profit. So the fractional odds kind of show us that. I bet one, I get four. 1.17 is roughly equivalent to one to six. It's better than evens. We refer to it as odds on. If I bet six, I get one back. So it's not a great deal because it's very, very likely to happen. We add up the implied probabilities. We get more than 100%. Why do we get more than 100%? Because that's the bookmaker's profit. This is what's called the over-round. And if the bookmaker balances his book correctly, then he'll make 5.5% of all the stakes no matter what happens. So the bookmaker always wins unless they're stupid in which case they go out of business pretty quickly. And they can also... So this is an example of the different types of bookmaker. There are soft bookmakers and there are sharp bookmakers. Sharp bookmakers review their odds almost second by second. They are very up to date. They have low margins and they will take big bets. Pinnacle are a really good example. A company called Pinnacle. You can go to put 50 grand on a match on the nose with Pinnacle. No problem. Popular games you'll get even more on. Above that you have to go to the Asian syndicates and it's really frightening. Soft bookmakers won't take as much money. This is an example from a couple of weeks ago me trying to get 200 quid on Bristol City to get promoted to the premiership. They won't risk a payout of £3,400 so they limit me. So this is one of the ways in which bookmakers will balance their books. What's next? Up until around 2000 a bookmaker was the only way to get money on. They were the price makers. Around about 2000 just after we got into really good liquidity at Betfair Betfair is an exchange and Betfair does something slightly different. Betfair allows you to be the price maker as well. So if Brad, if you're a fan of the cherries you're going to get money on Bournemouth or you're going to back Bournemouth to happen, to win I will lay Bournemouth. I don't think Bournemouth are going to win. I think it's going to be a draw or West Ham will win so I lay that bet. So it's equivalent of shorting in the financial markets. You're betting against. This is kind of revolutionary and it's great business for Betfair because whatever happens they take 2% of the winnings. So they are rolling in cash. And they do really well out of this. So that's just another close-up of that. So how do we beat the bookmakers? How can we do this? There's two ways. First of all we can know more than the price maker so we have some information about the game that we know something's going to happen and it means that our estimation of the true probability is better than theirs and our estimation of the true probability says it's more likely to happen and so the bookmakers odds are higher so we've got what's called a value bet. There's value in this for us so we can put money on there. You are not likely to get a value bet on the NFL from Vegas lines. You are not likely to get money on a value bet on the premiership at William Hill. The famous big kind of events and tournaments and competitions are really well modeled. You will find an edge on the Dutch women's second division. You will find an edge at the moment on esports if you know and do your homework about the Dota 2 teams or CSGO teams. There are really badly modeled markets at the moment here and there that you can pick up on and you can make money so there's value betting there. There's something called match betting as well where you can take advantage of the offers that bookmakers make but they're likely to get you banned because they recognise that you're only betting on free bets and things. The other which we'll explore in the rest of this presentation is about exploiting price movements. In this case we don't care what we're betting on. We don't care what sport, we don't care about anything. We just care about these streams of numbers that are coming in so you can see where I'm going with ACA. We've got streams of numbers coming in and we want to compare things and find an advantage that way. The secret special option 3 here is that you can do a little bit of both. Tennis is a good example. If you've got the number one seed playing the number 40 seed every now and again maybe he's got a cold coming on or he's had an argument with somebody and he'll drop the first set or she will drop the first set and their price will move way out from 1.4. We've seen them go beyond 3 and you can back them there and they come back. The number one seed they come back and they storm the rest of the sets and they win the match and we've got value at 3. So there's a way in certain events on certain games that you can do both. But we're going to focus on that second part and a specific way of doing this called arbitrage which you may be familiar with. Let me run through very quickly traditional arbitrage. What we're doing is making a book from multiple bookmakers. So we've got different prices here from Bet365 and Pinnacle and if we put a £500 total stake or $500 total stake what we'll end up with is a guaranteed $6.09 win. Whatever happens or 6.10 if we're lucky. That extra cent matters, right? What's happening is we've got 21% probability on here and a 77% probability on here which adds up to 98, that's less than 100. This is an under round. This is a 98% book means it's value for us. An over round greater than 100% is better for the bookmaker. So if we can create our own book with an under round we can make some money out of this and we can lock it in. So this is the essence of it. I'm going to show you a slightly different way to do this. A hedging version where we can use Betfair's lay probability. Capability rather. So if we've got a price of three on Fedora, Pinnacle and we can lay at two on Betfair and we put 100 quid on at three we get 300 quid back. We get £200 plus our stake of 100. The important mathematical thing here is we take the return we divide it by the odds of the back and then we use that as the liability on our lay bet. That's it, that's the trick. We can guarantee we can lock in whatever happens here we make 50 quid. This is called a 50 pound green or a 50 pound green book or a green bet. The green meaning profit as opposed to red meaning deficit. A 50 pound green is like a white hole in cosmology, right? Theoretically possible but never really observed in the wild. Realistically, five, six pounds dollars you're doing really well. So this is an incremental kind of thing, right? And if we make any kind of mistake we've lost £100. So if we're, you know, even in this example we've got to make two successes to make up that failure. In reality, you have to make hundreds of winning bets to make up for one failure. So again, you can see where I'm going with this with ACA and reactive and handling failure. So it's a really good use case to this. So the process is we place the labour on bet fair first because there's a way to back out of that position on bet fair. We can back. So if our lay bet fails we can go place a back bet and we can get out of that position. If we're lucky we get matched then we just zero out and we've not lost anything. If the lay bet is successful then we place the back bet on pinnacle. And an important point is that we can do this automatically, right? We have APIs for this. The softer bookmakers won't have APIs for obvious reasons, right? They don't want people exploiting this stuff. There's ways around that. And if you pay me thousands of pounds I will do it for you and explain. But not on here. And if we fail on pinnacle we haven't put the money on but we've still got the lay bet so again we go back and back. And then if we're successful then, kitchen. So what we want to do is take the stream of pinnacle prices the stream of bet fair prices check to see if there's an arbitrage opportunity and if so give it to an actor and let it play out this scenario, this model. So, yeah. So, yeah, it's just another example of what I'm doing. If we need to know how to connect one thing to the other, right? When we get a bet fair price we need to be able to connect it to a pinnacle price. So we need to match the markets. And this is kind of an important thing that I'm leaving out here. Matching markets is one of the hardest things to do in this game. It's one of these things where it's really easy for a human to do to see that it's Manchester United here and it's Manu there but getting a computer to do this reliably is something that's eluded me so far and others too because again we cannot fail if we get Manu and we get some other team then we can find an arbitrage opportunity that doesn't actually exist and waste our money and send it to nowhere. So, and give it to the bookmakers who are evil who we want to defeat. So we need to be really careful. So it's usually a manual process but you only have to do it once then you've got that registered and you can match a bet fair idea to a pinnacle idea, get the latest bet fair prices when you get the latest pinnacle price and then do the arbitrage thing and if it's successful give it to the actor. So in code so I spent some time this week working this up a really kind of naive, trivial kind of example but this is what it looks like. So I'm assuming everyone can see this so we'll start our actor in this waiting state and in the waiting state all it can do the only message it can receive is this new opportunity message which contains the info the arbitrage info the opportunity somewhere there there it is so it's just this arbitrage info which is the selection and the price of the bet fair the selection for price for pinnacle so it's just like a wrapped up version of this. So once we've received that message we'll start the process with triggering the label and I've just marked this service up to sometimes it will succeed, sometimes it will fail so we can just mimic this and we can see how it looks like and then we move into this working state so we can use the actor's ability to model to be a state machine basically now it moves into this working state if we get the label failure then we back on bet fair if we get the label success then we back on pinnacle so it's the same model and it's just played out on here and I've added some custom metrics here so we can see some nice graffing so that's the actor that's relatively straight forward there's a lot of typing but it's relatively straight forward so we can see from our message handling exactly how that works now the trickier bit and Scala format has screwed up my really nice little diagram there of how this looks but essentially we've got a repository which contains the ID matches a repository which contains our latest prices typed keyed by an ID and I've used the ACA HTTP cache here which so this is ACA HTTP cache it's just a nice way of getting stuff in and being able to do things just a kind of cheap shortcut we can back that by a database as well I just didn't have time to do that and then we have our sources so this is our bet fair market source so we get the events from bet fair, the prices from bet fair and bet fair have a really nice stream based API socket where you can connect to and you can tell it I want to subscribe to these markets and it will send you the changes it will push them to you pinnacle you have to poll so you can make that into a stream yourself but it's relatively trivial to do bet fair API I have to say is just one of the wonders of a modern world absolutely fantastic I don't know about message driven but all the reactive principles are in play there I've used it for thousands of times a day for years and I've never seen a single bet fail it's like unearthly it's brilliant it's a fair play to them so we can create this source and then we can map getting the market book registering the market book so this is our latest bet fair prices stored in the cache then we can pull the pinnacle ID I've used cat's option tier to just save some complicated for stuff and then we can get the market book for that ID and so what we're returning is this tuple of bet fair and pinnacle market books same thing just the inverse for pinnacle so now I've got two two sources my bet fair source and my pinnacle source but they are both a tuple of bet fair and pinnacle so I can merge them right I can merge them together and then push them through this decider right the decider checks if his arbitrage arbitrage opportunities here and it does this with this cool thing here this is my percent difference operator I may not have used the mathematically correct symbols but I don't care it's my code and I'll do it how I like so this is the the maths here for checking the difference it's relatively straightforward again subtract one divide the rest off you go and then once we've merged all that in together we name it here because I'm going to show you this running in on a graph and we've used cinnamon for that but this is the important bit if if we have an arbitrage opportunity then it will push it to this arbitrage actor right so it will spawn a new actor spawn a new actor and then ask it to process opportunity it will send it and then it will need a response so that it can back pressure the rest of the flow so the speed at which we can spawn actors and get these things moving that will back pressure all the way back when I run this you'll see we're going to get a lot of arbitrage opportunities in reality again this is not something you see you'll get I don't know on a Saturday with all Premier League games going you might get a handful it's not a high volume game but it works and it's guaranteed money so there's that so let's run this if I can find the right thing this is where things always fail to what could possibly go wrong with the live demo oh no it's running right that's stage one done so now I can see it's hammering through these opportunities and if we refresh this or we wait for this to refresh in a moment I sweat for a few seconds and talk to stretch it out oh no look we've got a graph so we can see we're starting to get things moving through and if you're using Acre and you're not using Cinnamon or Acre Insights or whatever like Bender calling it this month then you'll find you need to have a word of yourself really because it's absolutely fantastic yes you can use command and it's open source and whatnot but for Acre streams especially the level of insight I've been able to sample things so you're not really treading on performance there's just nothing else out there like it's really good so if we go into the Acre streams and these are all just the default things apart from that custom dashboard we can see we're getting running streams we can see the throughput here we're getting 14 something thousand operations a second this mark is it's not a bad mark but it's the fans just coming on so we can get a lot of these things through and this is just a single Acre application six hours come on see when we fix that so it remembers so we've got half a million more than half a million opportunities being handled here so responsiveness is not a problem so let's look at resilience and have a think about that because we want to be reactive so I've added this little section in here this supervision strategy at the moment there's no supervision on this so if it was to fail if one of these actors fails for some reason then it's going to kill the whole stream and it's just going to stop processing things which might not be a bad thing we want the whole thing to shut down because we don't want to lose money but we also don't want to miss an arbitrage opportunity unnecessarily so if we go back to the actor here we can say pinnacle backpacks yes so if we stop the actor at this stage say stopped if I can spell correctly so if we stop the actor there and run this again so it's going to kill it it's going to die now because the actor failed and the privilege wasn't handled correctly because it's trying to send something to an actor that stopped and everything falls apart so we can handle that with our supervision here so if in this map async stage we can do these things so you can supervise the whole thing you can supervise particular stages Acastreams works really nicely for this being able to handle failure at particular parts of the stream I suppose I should really prove that that actually works just for completeness ok ignore that let's move on that was a last minute addition it's not my fault it does work so what we got left seven minutes right I'm going to jump through the next section because what I was going to say was the next section kind of introduces this problem that we have with that particular thing which is arbitrage opportunities are going to last like a second maybe two at the absolute most so we need to jump on them and we need to do something really quickly but what can happen is we can place the lay bet and then pinnacle can change their price or we can place the lay bet and then place the pinnacle bet but something can change right in the meantime so we want to be able to catch that because otherwise we're going to place the lay bet place the pinnacle bet which may get placed because pinnacle will allow us to get the bet placed when the prices have moved in our favour but then we're going to not have an arbitrage opportunity anymore so we can lose money if the wrong side wins and sod's law says that the wrong side will always win and we'll end up losing we never get lucky I ran this for nearly a year and not once did I get lucky with that so you need to cover your back so what we can do is change the process we have two flows and instead of trying to join things in the stream we can push the state where the logic is and put everything into the actor and we can feed the actor with a stream of bet fair prices and a stream of pinnacle prices and let the actor decide right so we can while it's processing the lay it can receive updates on pinnacle if now it's not an arbitrage opportunity anymore it can go back and back out of the bet fair position so we can add more compensating actions if the information it needs for the logic lives next to the logic how do we connect those two things the receptionist is a good example of doing this I'm not going to run this example because I'm running out of time but we can have oh sorry I've got my reading glasses on is that five in it so we can yeah we can run the same stream so we don't merge the streams now we have two streams of bet fair price a stream of bet fair stream of pinnacle and and then we can send them to this arbitrage actor right but how do we find the arbitrage actor we ask the receptionist and we can key it by the market ID of the market our universal ID that we've created that will map to both the pinnacle and the bet fair market and then that will give us a unique actor that particular selection on that particular market so for all the thousands of markets and the many selections that we're trying to arbitrage against so we can have an individual actor for each one and then just feed the data into that the receptionist APIs fairly straight forward as well we can just ask it for the listing for this key and we get back the list of all actors because you can register more than one actor for a particular key so here we can just pull the head and fold over it if it exists return it if it doesn't create one register it and return it of the registration is done in the arbitrage itself always go the wrong way so we can just call the receptionist and register ourselves when that actor is created again I'm running out of time so I'm not going to show an example the third example I wanted to show is this is just running on one machine so if that machine fails we're losing opportunities again so let's embrace Acre and go for it and run this on a cluster so I'm not just going to run it on the cluster because that would be too easy we're going to go mad I'm going to get the services running in a cluster singleton because I only want to connect to Betfair once so it makes sense to have a cluster singleton I only want to connect to Pinnacle once cluster singleton for that I've merged the two for simulation purposes and then I want to distribute the processing so ok thanks so we're going to have multiple fixed number of processes and those fixed number of processes are going to send a sync ref to the cluster singleton and that singleton is then going to connect to those syncs and shard the markets across them and then we've shard the arbitrage across the cluster as well so now whatever happens if a node goes down Acre is going to rebalance everything for us a new sharded daemon process comes up registers a sync ref and starts picking up the messages from that shard so sync refs really short time sync refs are really useful when you want to send things across the network absolutely brilliant and basically all you have to do is send that ref the messages sent down so in this case we're merging everything into a merge hub and then we're just broadcasting everything to the available syncs after doing our shard algorithm the last step I wanted to talk about was then introducing Kafka so for this and I've put this code up online so go to think more stupid less on github and you will find this all this code but we can use cloudflow which is a criminally underused piece of technology which allows us to create Acre streams applications and wire them to and from Kafka really nicely and it also gives us a nice configuration file which tells us how things are connected up as well because the Acre streams code can get horribly messy and spread out everywhere and you can't really tell what the flow is cloudflow goes some way towards solving that problem create the two streams put them onto two different Kafka topics and pull them and still have an arbitrage cluster still have a cluster within cloudflow as well that was Nolan's invention over there well done so you can create an Acre cluster and scale that as you scale pods in Kubernetes last thing is running the application don't use Kubernetes don't go on the cloud it's too expensive you're not going to make a huge amount of money from this so save your money and if you've got some old Raspberry Pi is knocking around go to Eric Lutz's thing another Lunatech an ex-Litebend employee created this way of running Acre clusters on a Pi cluster and you can save yourself a lot of money and feel like a real technology engineer with lights it's brilliant, it's fun and look at this and bang on time to say thank you very much any questions, sorry come on it's one question why am I doing this if you give me a million pounds I'll tell you go on Nolan it's already done before it gets to the merge I'm taking the market book for Manchester United so the home team the away team and the draw and then flat mapping that so we've got just three streams and it's already connected before it gets to the merge so we've got a tuple we've got the bet there selection and price the pinnacle selection and price that matches and that travels along and gets merged and gets sent along so it's not they're already paired before we get to the merge but you could do it differently this is just my blight naive too late thank you for a question cheers our two questions so cool