 First, I want to say I came up with the PPSTAC six months ago and I can prove it. Just kidding. So I'm kind of curious, how many people here have Elixir applications in production? OK, cool. So mostly this is going to be you guys laughing at me telling you the bad ways that I've done things in the past. For everybody else, it's going to be me warning you about the bad ways that I've done things in the past. But if you find that they're similar to the ways that you're doing things now, don't feel bad because I did them too. OK, so for me to tell you about this application, I kind of have to tell you about what our company is and then kind of where this fits into it. So Simplify works in the ad space. And oh, shoot, I got an ad. Oh, crap. So actually, these are all ads that I took from my browser when I was getting ready for this talk. And the basic idea of what we do is like when your page is loading, somebody who built the web page has decided that there's going to be an ad there and they don't really care what it is. So they go out to companies that act like auction houses and those companies send requests to companies like us and say, do you want to buy this ad space and how much are you willing to pay for it? And then it's up to us to make that decision. And it has to be in real time. So you can kind of see these are actually pretty good examples because they're for hotels in Austin. I was looking for a hotel. My wife and I recently bought a house. So obviously, I've been to Bed Bath and Beyond. And then mobile phones and cars are always being advertised. So just to sort of put that into picture form, we do everything below the dotted line. We decide in real time which of our ad campaigns we should put there, if any. And we actually don't always decide to bid because maybe we just decide it's not really worth anything. So that doesn't sound all that exciting. But from a developer's perspective, I think it's kind of fun. I said that has to happen in real time. We do it about a million times a second, actually over a million times a second. And all of the ad exchanges, which are like the auction houses, they want to hear back within, say, 75 milliseconds. But that has to count the flight time. So really, if we're not responding once it hits the server within like 10 milliseconds, we're not going to do well. And it turns out that actually it's been interesting to see a scale up to a million requests per second. I think when I started a little over three years ago, we were hitting maybe 200,000. But it's also been interesting to see how we've scaled in terms of the number of ad campaigns that we use. And that's actually kind of simplifies niche in the market. So there's a lot of big companies in this space that'll handle maybe a few hundred campaigns and they're for big brands that you've heard of. We handle, we've handled 30,000 campaigns in the month of September, which is a record for us. And we just keep going. So I get like I have the graph here over the last three years. That has scaled more than 10 times. And it's sort of interesting what parts of the system have to scale to handle that. So some other little things that make it sound like a sexy place to work. We serve about 60 million ads a day. We're distributed across the globe in five data centers. We are hiring, by the way. And we put up a thing on the job board out here. And I saw somebody reading it. And they're like, OK, you use all these different things. But no, we really do. And that's one thing that makes it a really cool place to work. We weren't using Erlang when I started three years ago. And I'll kind of talk a little bit about it here. But I just said, screw it. I think Erlang would be a good way to solve this problem. And it ended up working great and was well received. And it's a place where you can do that kind of stuff. So just basically what I'm going to talk about. I'll kind of explain what problem we're trying to solve just so you have some idea and why it's kind of an interesting and hard problem. I'll talk a little bit about how the solution has evolved over the last basically three years now. And then I'll say some things. And you'll laugh at me because I did them poorly. So this is a term that I made up, real time accounting. Basically, each one of the ad campaigns that we're trying to serve against has a budget. So somebody will say, each one of these ads is a fraction of a penny. So maybe we're bidding. We won't only spend $1 a day or $100 a day, which boils down to like 1,000 impressions or 10,000 impressions. And impressions just showing somebody an ad. We have to count that so that we don't accidentally serve too many ads. And that's bad not only because somebody's got to pay for that. And it kind of sometimes it's awesome. Sometimes it's the customer. But either way, the customer runs up pretty unhappy because a lot of the time our customers are actually people running ad campaigns for, say, some local tire shop or something. They don't want to worry about their advertising campaigns so they hire somebody else to manage it. And every step of the way where you're communicating that that somebody has a chance for somebody to get upset and then they take their business somewhere else. But on the other hand, we do want to spend as much of that money as possible. Because basically, we charge a little bit on top of what we charge our customers. So for every dollar that we spend, maybe we make $0.25 or $0.50. And then another interesting thing is that the advertising industry is entirely budget-based. I don't know if you ever hear stories about in government agencies where they get $1 million a year budget. And they actually have to work to spend that budget because if they go back to their superiors and say, well, we only spent $750,000, then next year they're only going to get $750,000. So all the time, everybody's trying to spend their entire budget. So this particular process that I'm going to talk about fits into that picture where when we serve an ad, we get to count whether it gets counted against the budget. So we have to figure out how much it costs, which campaign it goes to, all that fun stuff. And the thing that makes it even harder is that we don't know. We roughly win around 1% of the times that we bid. So when we say we want to buy this ad space for $1, we don't know that we got that until we actually serve the ad. So there's a delay inherent in the system. And because it's an auction, we don't know how much we're actually paying until they tell us. So getting ahead of myself, what makes this sort of challenging besides the delay? This is like super coupled to how much money our company makes. And we're a really transparent company, which is nice. But at the same time, like everybody in the company is watching the output of this all day, like you just walk around the office and it's up on people's screens. So if this goes down, like everybody knows right away. And then if the budgeting piece goes down and it's not coupled well to the bidding piece, we can have the worst situation, which is where the bidders can actually continue bidding on ads blindly. They don't know that they're actually spending money because nothing's telling them that they're spending money. And that can lead to really, really bad situations. So it actually tends to be better for us to just shut down if this process goes down. So it can be sort of high stress. It's also got to be fairly performant. So like in the middle of the night, tends to be our highest through rate, although lately we've also been serving a lot in the middle of the day. We're serving like, say, 5,000 ads a second. And that's in five data centers. So it's got to be able to process. That's not like an astronomical stream of data, but it's a pretty good sized stream of data. And then the thing, like I said earlier, that actually makes it even more challenging is for each one of those things, I've got to look at 30,000 different ad campaigns and decide, because they all have slightly different cost models. People make out deals like, we're going to charge you a margin of 10% for geo data until, and it's also time dependent. So we're going to do that for the first month, and then it's going to go up to 15%. So there's all this campaign data, this huge cache of data that we have to reference every time we do this. It's also a distributed problem. So I can be bidding on a user out of our Washington DC data center, and then I serve an ad to them on the West Coast. So I have to have this consistency across data centers. So how did we get to having an Elixir application that does this? So the original system, which I won't spend a whole lot of time on, was written in Ruby. We were always kind of a Ruby shop. It was just a script that ran. I'm pretty sure it was keeping track of all that campaign data by just marshaling it out to a file, and it just ate a ton of memory, and it was slow. So when I started, one of the first tasks that they gave me was like, well, C++ is fast, so write this in C++. I'll keep the story short. Basically, I spent two days trying to hook up a Postgres library in C++, got really frustrated, and said to my then boss, hey, we've heard about this language Erlang. Can I try this in Erlang? And I'm pretty sure I learned the language Erlang and built this application faster than it would have taken me with a medium amount of C++ experience to write the whole thing from scratch in C++. Our initial thing was we were using rebar, doing a little bit of testing with eUnit. I'll say a little bit later about eUnit versus xUnit. I think eUnit's got kind of a big learning curve. Our deployment was literally clone the thing and build it on the server. So this is the start of you guys laughing at me for bad choices. So this is the fun part. What was it like to get our first Erlang application in deployment? So processes, we're running all these processes and we're going really fast. This is great. This is so much faster than I even thought it was going to be. And then something happens. We crashed. But if you've had anything in production with Erlang or Elixir that's running at a reasonable volume, like handling a fair amount of traffic, you'll realize that when it crashes, it actually looks more like this. You don't have some catastrophic beautiful collision of vehicles. You have all of these processes just dumping out all of their contents and what the hell happened. Nobody knows. So we learned some things. Kind of the main point to me of this slide is 2013, 2014, we learned from our first batch of mistakes and rewrote a little bit of stuff. But the same piece of code's been running for almost three years. And we've scaled our number of campaigns by a factor of 10. The volume that we're processing by a factor of four or five. I won't lie, some of that ability is just buying better hardware. But I was pretty happy that the basic idea of what we put together was able to last for quite a while. And I do attribute a fair amount of that to the Erlang VM and the way that it lets you solve problems. So I think it was early this year. Elixirs really started to come around. I've been hearing about it for a while. And we did finally decide to start using it. So this is kind of the story about that. So this was me. Well, now this was my perception of everybody else. Like Elixir is the hipster language, whatever. And this is me. I'm like, I don't care. This syntax doesn't look that great. I don't need another layer of indirection. But I don't know, is Proctor in here? Maybe, maybe not. Oh, I see Proctor didn't come to the talk. We were at that time around the DFW Erlang group in DFW. And he managed to get Dave Thomas to come to the meetup and give a talk on Elixir. And he said, well, actually, Elixir has a lot more going on than just the syntax. It was a really, really great presentation. And this was me. So the next day, I sat down and started converting all of my dependencies to use MIX and building against MIX, converting tests to XUnit, converting our gen servers to just use gen servers when I have to write the same boiler of late code over and over again. Now we're at a point where we're deploying via Chef. We're using MIX and, which is one of the greatest features of the language as far as I'm concerned, even though it's really subtle. And pretty much everything new we're writing is in Elixir. So what were the dumb things that I did? So when you first learn Erlang, and I think this is a little bit better handled in the Elixir documentation, one of the things that I really love about Elixir is the documentation and the tools and everything. It's constantly nudging you in the right direction. But rewinds three years. And when I'm learning Erlang, it's like, oh, everything is a process. I'll make everything a process. It doesn't work very well. You really tempted to write something that looks like this. So for us, we're generally working off something where we have a stream of incoming messages and we're going to work on it. I have to read from a cache for every message. And then I'm going to output it somewhere. And I say, OK, well, I just build this little unit of scaling and then I just scale it up the number of times that I do something. And then I get all these, I'm too smart for my own good kind of ideas where, well, if I need to access a campaign and I don't have it cached, I'll just throw it back onto the message bus and move on to the next message. Which, by the way, if you do that, you'll just end up repeatedly saying I don't have this thing in cache. This is much better. Block on cache. Basically, the thing that I learned was that learn to realize what the bottlenecks are. I mean, you can just throw a concurrency at everything, but that's not really the way it works. What you end up doing is just building up more of a backlog faster. So if you have to wait for something, just wait for it. Identify the things that are embarrassingly parallelizable. So for us, that's parsing. And just once I have all this campaign data actually doing the calculations, that's all just straight up functions. That's my unit of scaling. Batch up messages to a database cache or something like that. Batch up all of my outputs. Basically, the hard thing for me was to learn to think in terms of processes. I wish I knew how to just explain that. I think I could probably make money. But once that clicks, you start to see these things. And it becomes much easier to solve problems. So just to make that a little more concrete, what are some of the processes that we have in what we call Budget Notifier, which is this piece of this application? Our scaling is the number of processes that we use to just basically work off the message queue. And that's more or less just limited by the number of CPUs that we have because it's going to be CPU limited. Our campaign cache is a whole bunch of processes. Basically, one campaign is one process, which is kind of cool. You think about, we have like 30,000 campaigns. And then each one knows about its own invalidation. So it'll say, OK, I've been sitting here for an hour, and nobody's asked me for any information. I'll just die quietly, and nobody worries about it. There's also, this is kind of, to me, a nice example of solving things using processes. Campaigns will have, they can be grouped. And they'll say, these three campaigns can spend $100 a day. That doesn't really fit in with this kind of campaign level cache model. So what I did was just throw up another process. Its whole job is just to repeatedly go through the list of campaign groups and say, do you have budget or not? Things that didn't work well, price calculation. So again, I think this is something that the elixir documentation may be steers you in a better direction. But I thought, just wrap everything in a gen server. That's what it's for, right? You end up with all of your various processes talking to the same gen server, and so that becomes your bottleneck. Cache state dictionary is a similar thing. So I have all these campaigns, but then I need somewhere to look up and say, do I have this campaign in cache? And if I do, what's its process ID? That didn't work well as a process. It was a similar problem. It's a star graph. Everybody's asking the same process for the same information. So it becomes a bottleneck. So if I was to try to abstract that out, I don't want to sound like I'm against processes. Processes are great. Launch processes for logical units of thought. ETS is a good thing to learn. So if you have something that looks kind of like a table, in my case, campaign ID, cache state, I'm just asking for a number or a process ID for a campaign. You can set up ETS not to have read locking. So that becomes significantly faster. And then you launch a process to manage that ETS table. It thinks in its own way. When a campaign goes out of cache, it messages the manager, and the manager removes it from that state dictionaries. And you end up finding better ways to handle some of the raised conditions that can happen. Lots of processes. It has no problem doing lots of processes. You just have to be careful about lots of processes talking to a small number of processes. Know the tools. The tools are great. Mix and Hex had a bit more about this. I decided to make it short. I think I'm going to actually end up being done really early. Mix and Hex can be a great way to transition. Like, if you have something legacy that's eroding, and you want to move it into Elixir, disclaimer for myself is I never really tried Rebar 3. But I didn't love Rebar. I don't know if I think I've talked to several people who said it was OK, but not the best. I like Mix a whole lot better. So it's actually pretty easy to replace your Rebar configuration with Mix configuration. And again, it sort of nodges you in the right direction. Mix end is great. So instead of we used to have our sys.config file, just had a bunch of clauses. If basically whenever we were looking up our config, we'd look it up and production environment would be one of the keys in the keyword list. And so it'd be like database test. And that would return the test database. Database production would return the production database. Whereas with Mix end, you just have three different config files. X unit, I like X unit a lot. It does some things for you that you might not realize. It starts apps for you, which unit doesn't quite do. I think it has a slightly gentler learning curve than a unit. It's got a good way to do set up and tear down of tests. It's really easy if you already have unit tests to convert them to X unit tests. Documentation is great. How many people here who have something in production in Elixir are not the only people working on it? It gets a bit more interesting when you have C++ developers coming to you who have to make modifications. And they're saying, what the heck does this do? So using module doc and doc and specifications. Specifications is a great way to lure in C and C++ developers, by the way. Can be a great way to convey what the code is supposed to do. And Dialyzer, I like Dialyzer a lot. I can't tell you how many times I've done Git commit. And the comment was, thanks, Dialyzer. It just catches, even if you test a lot, I try to test a lot. It catches things that you didn't think of before. OK, so I wanted to say something that was a little bit maybe controversial. I've seen a lot of people get lured into Erlang and Elixir and then get sort of turned off. Because it was sold to them as something that made concurrency and parallelism trivial. It's good at that. There's a lot of use cases that I've done that really well, like React and RabbitMQ. I'm not super familiar with Ejavert, but I think it does. It won't architect a solution for you. I'd say unless you're in a situation where it's really the right solution for you, you might be better off looking at using something like RabbitMQ, which is built on the Erlang VM and has kind of like Elixir. It nudges you into doing things the right way, because it's like, well, here's the primitive things that you can work with. And then you kind of get to solve your own problems, rather than trying to figure out how to build a distributed application, which is something that isn't probably what your business does. Unless it is, and then don't listen to me. OK, so I guess if I wanted to say these are the things that I wanted people to take away, just to harp on the point to me, the Erlang VM, so Erlang and Elixir, it's about processes and thinking about things in terms of processes, but it's not about trivializing distributed computing or parallelization. When you can find ways to think of things in terms of processes, it can make actually solving certain types of problems significantly easier. Elixir nudges you in the right direction a lot of times, and so if you feel like you're fighting Elixir or the way that it's trying to tell you to do something, step back and ask whether you're really trying to do it the right way. And then learn the tools. Mix and X unit are great, SPAC, dialyzer, documentation, all that stuff. And that's actually all I had to say. I was afraid that it was going to be way too long, so I cut a whole bunch of stuff. So does anybody have questions? All right, thank you very much.