 Hey, I'm Zach. I work for Factual, and we're going to talk about a little board game called Go. A quick show of hands. Who here has played Go? Oh, this makes it so much easier. Okay. Anyway, we're going to do a quick refresher. So Go is played on a board with intersections, typically anywhere between 9x9 to 19x19. Stones are played on the intersections, first black, and white. And each of these stones has liberties. Liberties are empty adjacent intersections. And when you play a stone in one of these liberties, this is extending the group. These contiguous blocks of stones of the same color are now one big group. And when a group loses all of its liberties, it is removed from the board. And the capturing player gets a point for each stone removed from the board, but also gets a point for the enclosed intersections. Each enclosed intersection is one point. And so if black here has enclosed the corner intersection, it has a point. And white cannot play in that intersection because it would have no liberties. This would be effectively suicide. However, if white completely encircles the black stone, for that one point, it can play in that point and thus remove the stones. Now, this may make it seem like Go is just a game of encirclement and re-encirclement and all this other sort of stuff. But if white plays here, even if it's fully encircled, black cannot actually remove this from the board because in each of these two liberties, black cannot play in either of them. It would be suicide for it to play there. So this white group is stable. It's safe. And that's because it has two eyes or two liberties that it's completely encircled. And so Go is not a game about capture so much as it is about trying to encircle territory in a stable way. And so it's less a tactical game and it's more of a strategic game. And so what you'll see is when the game develops, what it is, is it's a series of claims, right? By playing one space to the right, black here is sort of implicitly claiming the right side of the board. And white can have many responses to this. It can play to the left thus effectively sort of seeding that territory, though that may develop over the course of the game. It can play above claiming the top of the board and seeding the bottom. It could even play within the territory effectively being claimed by black, kind of stating that it can create a stable group before black can encircle it. And as the game develops, you'll see that a lot of the territory is encircled and not contested, right? This is the ending position of a professional 19 by 19 board. They haven't really gone and played out all these territories because it's not necessary, right? They kind of understand that they stand more to lose by sacrificing a bunch of pieces in those territory than trying to actually fight over all of it. And I don't recall who won here. I think it might have been white or something, but at any rate, we want to write a program that plays go, right? And I'm interested in this not because I think I have any unique contributions to make to this field, right? I enjoy Go. I think it's kind of a beautiful game, but I'm not particularly good at it. I studied this a little bit in school, but it's not something that, like, again, I have any real expertise. This is just something that no one's really done before, right? And when I say not done before, I don't mean just enclosure. I mean that people typically write this in very low-level languages, like C, C++. I think there are a couple written in D. But, like, that's as far as it goes. And, you know, maybe this is for good reason. You know, certainly there are some stumbling blocks here. But, you know, closure is a wonderful language for exploration, right? For experimentation. And it stands to reason that maybe this is something that, you know, would be helpful for us to have this sort of high-level language, to be able to kind of span both sort of the lower-level stuff that closure can do, but also the high-level stuff that closure kind of excels at. So, where do we start? Most AI is search, right? And so when we're talking about, you know, we're here at this particular position and we want to make a move. We want to make a smart move. The thing is that we want to search for a move that will lead to victory. And so, in Tic-Tac-Toe, we start with an empty board. And we say that this is the root of our game tree, of the game space that we're searching. And we say that the children are all of the first moves, all the initial moves that X can make. And then beneath that are all the responses that O can make. And that just kind of continues down until we get to the terminal position, so the end points of the game. And this is a space that we're searching. Now, if this were a cooperative game, if this were solid hair or if we were playing together and trying to be nice, then all we would need to do is search through for a path to victory, right? Just some sequence of moves that would lead to us winning. But we're competing, right? And so we have to assume that as much as we want to direct the game towards this ending position where we win, we have to assume that our opponent will be doing his damnedest to make sure that we don't, right? And so we have to optimize our chances for victory given this sort of adversarial quality to our search. And the approach for this is called Minimax. It was invented or discovered back in 1930 by John von Neumann. And the basic notion is that we search the entire tree. We go all the way to the bottom. And we say that each of these has a value according to its outcome. In this case, this is fairly simple, right? We say it's one at its victory, zero if it's a tie, negative one if it's a loss. And we assume that the player in the state preceding that will choose the best move for himself, right? He will direct it towards and it's there. And so we say that the value of the node just shy of the ending condition is equal to the best possible outcome for that player. And then we can go up one more level and we can say, well, the player will clearly direct it to the best possible for themselves. And from our perspective, right, we're constantly maximizing the outcome, and our opponent is constantly minimizing the outcome. And from that, we kind of go up the tree and discover what the optimal path of play is, assuming that both people are playing as best they can. And if we're able to traverse the entire game tree, this works. We can play a perfect game. So for Tic Tac Toe, we can do this pretty easily. There aren't really that many possible Tic Tac Toe games. This has been done for checkers as well. In both cases, it's a tie, by the way. But for more complicated games, this is not something that we can do. This is not something that's plausible. We can't just enumerate all the possible chess games or all the possible go games. And so we have to kind of compromise. And the way that we compromise is we say, well, you know, this is still a pretty sane way to talk about how we're searching, but we just can't get to the bottom. So instead, we go down as far as we can. And we look at this intermediate game state, and we say, how likely are we to win here? Right? And that's a difficult question to answer. And this requires not just an understanding of the mechanics of the game. This understands sort of the fundamental essence of the game. This is something where we require sort of meta information more than just the rules of this move leads to this move leads to this move. And this can be difficult for a number of reasons. One is that if we say have a chess game and we start counting pieces or something like that, then clearly we can optimize that we'll have a certain number of pieces at a certain depth of the game. But that isn't really a very accurate proxy for victory, right? There are pathological situations where that can direct us very much in the wrong direction. Similarly, we have this sort of horizon that we can't really see very far into the future. And this horizon is kind of inversely proportional to the cost of our evaluation function. We have something that's very rich, something that gives us a lot of information, has a lot of analysis that's done, but then we can only see so far into the future. So it's possible that a more naive evaluation function which lets us see further could be better. Of course, that's very difficult to quantify. And so this is sort of a trade off where we're trying to kind of twist this dial and not really perfectly understanding what the consequences of that are. And finally, you know, we as programmers, we write this code, we write the recursive search, we write the evaluation function. But understanding the local effects of this function is not the same thing as understanding the global effects of the repeated recursive application of this function, right? This thing that is pruning this tree and sending us in one direction at the expense of another. And this is a bit of a problem. I mean, this is actually something which is horribly frustrating because you'll see this movie made and you know that it's the wrong move, but you know that if you try to tweak your evaluation function to make it so that it doesn't make a wrong move in that case, it has untold consequences elsewhere, right? And this is sort of the essence of the problem. This is kind of wonderfully frustrating on some level. And this is, you know, why I am so interested in this whole problem. But, you know, all these things apply to go to a greater extent than pretty much any other major game. And the reason for this is that the tree is enormous, right? On a 19 by 19 board, which is sort of the standard, you know, size for a board, there are 361 possible moves. And even worse, the depth of play, the number of moves into the future that a strong human player can see is fairly large, right? And this is because the board remains fairly static. How many of you have ever played reverse eye or Othello? So this is sort of a variant, a simplification of go. And what this says is, if there are two stones of one color on either side of a line of stones of another color, they all flip. And what this means is that three moves out, the board can have completely changed configuration. And so it's very easy to write a strong Othello AI, not just because it's a simpler game, but because it's really hard for people to read very far ahead. Conversely, go remains largely static over time. And so it's fairly easy for people, right? Basically, we're uniquely qualified to be good go players. And so this has remained kind of a problem. And historically, a lot of people have tried to write go eyes, starting back in the 60s. There was a prize for a while that if someone could create a program that could beat a low level professional player by the year 2000, they would get $1 million. This is one of the millennium prizes. And no one won, right? A lot of people tried. A lot of people sort of threw themselves onto those rocks and kind of walked away defeated. And this is because it's a hard problem. And the computational power wasn't really there. And so what they would do is they would try to offset that by being very strong players themselves and trying to kind of encode their brains into the program. But that only works to the effect that they can perfectly capture their understanding of the game. And this is this is hard. This is harder than I think most people give it credit for. And so typically speaking, this has kind of been a bit of a backwater or sort of a failure, whereas, you know, chess is sort of taken off and kind of capture the imagination of people go has remained this sort of almost embarrassment. In 2006, though, someone tried something new. And they used something called Monte Carlo simulation. And to sort of demonstrate what this is, imagine that we have a quarter of a circle inside of a unit square. And we have a function which says, given the x and y coordinates, this is inside the circle. And we have a function which takes no arguments and simply says for a random point between 00 and 11, is it inside the circle? If we just call this a million times, and then we look at the ratio of trues to the number of samples that we had, that's pi over four, or converges on pi over four. And this may seem like sort of a trivial sort of clever math trick, but it's actually something kind of fundamental. What it is is it's saying, we understand the rules that govern a space. And now we can begin to understand its structure. And, you know, in this case, it's fairly easy. It's geometric. We can kind of say, you know, here's a space that we're trying to sample. But in real world cases, this is sort of less doable. And so it's instructive to think about what would happen if we didn't really know that it was 00 to 11. What if this was a little bit shaky, right? Depending on where we skew, this can start to change the value quite a bit, right? If we sort of skew down to the left, and all of a sudden pi starts, you know, ballooning out. And it can also start shrinking if we go in some other way. And so the thing is that it's not just that we do a bunch of simulations, we have a bunch of numbers, and it all kind of, you know, magically comes out in the end. We have to understand the space that we're sampling. We have to understand these sort of dimensions of what we're actually trying to search out. And so we can use Monte Carlo simulation as an evaluation function. We can say from this position, assuming that we simply just play naive moves, no real insight into the game, no attempt to make a good move, just a move. And we do this repeatedly, tens of thousands, hundreds of thousands of times. And we say, you know, on average, black wins 60% of the time. Therefore, this is a stronger position. This is kind of cool because what this means is that we've taken the rules of go, no understanding of go other than just the simple mechanical rules. And we have arrived at a determination of what is a better move, right? This is something which isn't really clearly encoded in the rules, except in some sort of various sort of amorphous way. But we're not trying to talk about the space of all moves that can be done, right? You know, if we leave ourselves open to capture and our opponent ignores that for the entire game, that's not a very likely outcome. We want to talk about plausible games. We want to talk about games that are likely to be played. And so if we go outside of this range of what is a plausible game into the range of possible games, this is noise. This is actually like sort of dirtying our understanding of the space. And so this is a problem, right? This is a balance we have to kind of do. Again, this is like the evaluation function. A heavier, more computationally expensive evaluation function is at the expense of being able to see further, being able to, in our case, have more samples. But there's an additional thing, which is that, you know, for doing this hundreds of thousands of times, each new game gives us more insight into the space that we're trying to simulate. And to capture that, to take advantage of that, we use something called a multi-armed bandit algorithm. And the multi-armed bandit basically says, we have a bank of slot machines. And we don't know what their payout is, but we presume that one of them is better than all the others. And the problem here is that we can go through and we can continue to, you know, pull them all one by one and try to build up a statistical model. But once we start to have a hypothesis as to which of these is the best, every move that we do, every lever that we pull, which is not the one that we suspect is the best, is that the possible expense of our expected return, right? We have this balance between trying to explore all the possibilities and exploiting the best known possibility. And there are a number of mathematical approaches to this. I'm not going to talk about them in any detail. The one that is commonly used for go is called the upper confidence interval or upper confidence bound method. But it's important to understand that if our probabilities aren't changing over time as we're sampling, we should expect that we will converge on the best solution or the subset of best solutions. So if we put these two things together, if we use Monte Carlo playouts and the multi-arm bandit, this is called Monte Carlo tree search. And it has basically four steps. We have a tree and this tree is only partially searched, only partially expanded out. And what we do is we go and we select a path and we use the multi-arm bandit at each game node, each board node, to determine the next move. Until we arrive at a leaf, at which point we select one more move. And then from there, we do a single random play out using the Monte Carlo sort of approach. And we then take that signal of loss or victory and we propagate it back up the tree to all the moves that led there. And since a positive signal will effectively give a greater weight to the path that we took, the assumption here is that we're going to have a very asymmetric expansion of the tree, which tends towards the better moves. And so for the rest of the talk, I'm only going to talk about the simulation part of this, because this is sort of the medius part of it and also from an implementation standpoint the most troubling. So let's look at what it actually takes to play go. For each move, we have to select a move. We have to check whether it's a suicidal move, whether it's a valid move. We have to update our representation of the board. We have to check whether or not we've captured anything. And then we just repeat that over and over again. And these two, the suicide check and the capture check, are similar problems and also are kind of a little bit expensive potentially. And consider this kind of pathological case here. If we look at the corner position, black cannot move there. That will be suicide. It would make the entire group disappear from the board. But to determine that, we can't just look at the neighbors. We have to walk that chain all the way to the top, examining at each point whether or not it has any neighbors going, continuing, continuing. They'll finally find, oh no, this has a liberty. It won't be a capture. Similarly, if white moves there, it has to walk the chain all the way up that entire chain of black stones, checking at any point whether there are any liberties. And so if we have a very naive representation of this board, which is just, you know, this stone is here, that stone is here, this point is empty. This becomes very expensive. This is something which we're doing a lot of duplicated effort. Each individual move to determine something that we may have determined at the preceding move. And so the way around this is that we want some sort of incremental state. We want to have something which is derivative, right? This is not a completely like normalized representation of it. But we want to minimize the amount of computation we have to do at each individual step. And so when we put down a stone, we can actually keep track of how many liberties it has, right? I mean, in this case, we put down a black stone and it has four liberties. We put down another black stone. And now we can say that this group, collectively, has six liberties. And rather than the previous stone keeping track of its own stuff, it just points to the parent. It points to the owner of the group. And we place down another stone. It'll point there. And it's notable that in this case, the highlighted intersection is being counted twice. And this is because we could try to keep some sort of unique set of points in order to have an actual unique count of liberties. But that means that we have to have a whole data structure to store this. This isn't just arithmetic at this point. And so this is not a liberty. It's what's called a pseudo-liberty. This is a fairly common approach in computer go. And we will double-count any intersection that we're multiply adjacent to. But if someone plays in that position, it will be double-decremented. And so this tells us whether or not we have zero liberties, right? It doesn't matter for double-counting. Once we're completely surrounded, we will have both zero liberties and zero pseudo-liberies. We also keep track of the sum of neighbors by assigning a numerical value to each intersection. And the sum of the squares of those neighbors. And the reason for this is that if the sum squared equals pseudo-liberies times the sum of squares, then we have one unique liberty. And I'm not going to go into the math. It's not that hard, but it's, you know, I'll leave it as a proof for the reader. So let's go back to these things that we're trying to do here, right? We have these five steps. And literature varies a little bit. But, you know, people can have talked about being able to do anywhere between 5,000 and 50,000 games a second at 9 by 9. So we have one second to do 10,000 games. We'll say that there are about a hundred moves a game. There are 81 positions on a 9 by 9 board, but we get captures. And so the same intersection will be played in multiple times. This is actually a little bit low. It tends to be closer to about 120. This means that we have one microsecond in which to make a move. Here's some things that take a microsecond. And, you know, I mean, it's worth looking into, like why is this, right? I mean, we have fast computers, right? You know, a giga Hertz processor means that it takes one nanosecond, one thousandth of a microsecond, to do a single instruction. Like why is it, I mean, yes, we have these abstractions that we've layered on top of this, but why? And here's some things from a fairly old article by Peter Norvig, which talks about how long it takes to do certain things on a CPU. It takes one nanosecond to do a typical instruction. It takes 25 nanoseconds to lock or unlock a mutex. The notable one here is the fetch from main memory. And notice that this is an article from 2001. This number hasn't changed since then. Now RAM has been getting faster, but that doesn't actually speak to its latency. What it says is that when we do a memory fetch, we pull down more memory in that contiguous block. And this means that if we're scanning over a contiguous block of memory, these reads are basically amortized. So we do less fetches, therefore it's faster, but we don't store things too contiguously in Clojure or in Java. We have a lot of references all over the place, and our caches are getting larger, and so this sort of minimizes the problem to a certain degree. But what this means is that every single time that we have to go off and search for something else, something that we haven't seen before. Every time we, you know, allocate some memory, which is, you know, something that we haven't touched in a little bit, we lose 100 nanoseconds. And so it's pretty easy to see how some of these things end up taking. They have, you know, 10 fetches from memory or something, you know, around there. And so what do we do, right? I mean, it's clear that on some level we have to take a step back from the standard approaches we might otherwise use. So I'm going to talk about some of the approaches that I used. And before I do that, I want to talk about why this is not generally applicable, right? Why you should ignore some of what I tell you here. So in my day job, I work with servers. And if I can get the response time under 10 milliseconds, that's pretty good, because that means that probably the latency is going to turn my response time into background noise, right? That's not true here. We're throughput bound. Also, we don't really have the option of using a smarter approach. We can't use a better algorithm because by virtue of the method that we're using, this Monte Carlo search, the only way we can be smarter is to do more of it. The only way that we can be smarter in the same amount of time is to be faster. And while maybe there isn't sort of diminishing return where once we get to a certain sample size, we don't like get that much smarter or anything like this, this is beyond our reach right now. No one has found a point where they're like it's okay, we can stop. Like the more computational power, the more simulations we can do, the better we will be and that will just kind of scale fairly linearly. So, we have to use mutable state. This is from the doc string for def type. This is, you know, Rich telling us, don't. And he's right. You know, this is not something that you should use lightly. This is not something that, you know, the fact that it's available in Java so freely, this is sort of the default, isn't an indication of its safety. This isn't like we're being put it in like, you know, little waders or something like that when you have to go into the pool. This is scary stuff. And you should read this book. Because if you read this book, you'll understand how scary it is. And you'll understand, you'll gain a whole newfound respect for the ways in which we are kept on the straight and narrow enclosure. And you know, it's terrifying too. And I mean, it's kind of a very interesting book. And if you want to go down the same path, you think that you need to do this, I strongly recommend that you read this, that you internalize this and that you spend a lot of time really thinking about how you can shoot yourself in the foot and not even realize it. But here's the difference between unsynchronized and volatile. If something is only going to be read and interactively on a single thread, you can use unsynchronized. In every other case, you should use volatile. Unsynchronized is marginally faster. So here's pretty much the simplest possible mutable type, mutable object that we can create. It has a single long counter. And because it's mutable, it's not actually accessible. We can't use the dot accessors or anything like that. So we have to write a getter and we have to write a setter. And we have to use set bang inside of the def type because it's not legal to be used elsewhere. This means that if we want to create some sort of function like thing that wraps over a complicated use of set bang, we have to write a macro. It's ugly, but that's what we have to do. Furthermore, if we wanted to say add a number to our counter, this is one fairly obvious way to do it. We get the value, we add to it, we set it. In my experience, this tends not to inline. And so you get a significant performance boost by bringing the read and the set within the def type. And so what this means is that if you are as sensitive to the performance as we are here sort of artificially, you need to do this. And your mutable data type tends to get a little bit big and has a lot of functions. And again, this is just sort of the price that you pay. So I said that it took a microsecond to associate five elements into a map. The reason I was able to say that with authority is because of a really excellent library by Hugo Duncan, one that I think is like relatively unknown, sort of criminally underused. And that's criteria. Criteria is a benchmarking library. And this allows you to start to understand how pieces of your software behave. And this does all the things it needs to do to control for the JVM. This is not the, oh, I'm going to wrap a time mac around a due times and I'm going to run it two or three times until the JIT seems to stabilize and boom, I know how fast my program is. No, you don't. You need, you should use this, right? And you know, there are some caveats. It tends to exaggerate the cash coherency, which means that the memory reads are going to be kind of minimized as maybe not a completely realistic real world sort of case. And the minimum resolution seems to be around 15 nanoseconds. If things are taking 15 nanoseconds, you may be able to do them 10 times and they'll still take 15 nanoseconds. And so you should, you know, be aware of that sort of lower boundary. But what this gives you is the ability to start talking about performance in the same way that you talk about anything that you use of the Ripple. You start to build it from the bottom up. You start to understand it from the bottom up. And this is a really important way to approach these sorts of problems. You need to take each individual piece of your code and you need to understand how fast it is. And then you need to take them in combination in some sort of common action. You need to understand how fast that is. And this is the only way that you're really going to be able to understand where the inefficiency, where the inefficiencies are being added, right? You need to build up an intuition for how fast things should be and you need to be very sensitive to where your intuition is wrong, right? You need to understand why you couldn't predict how fast a compound action was going to be. One of the reasons that it can be slower is because of the indirection that Clojure uses. There are lots of operators which or functions which under the covers go and do several different things depending on what it's handed. The single equals for instance goes down to, well it also goes down to dot equals but in, you know, the sort of simple case it'll go down to double equals for numbers identical for reference equality. Count goes to a whole bunch of stuff but if you're using an array as you probably will be if you're performing sensitive, the Java laying reflect array dot get length is significantly faster than count. And again I want to emphasize that you know when I say it's much faster, this is something that in any normal program would not make a difference but when we are sensitive to this when we're trying to minimize everything getting one of these things wrong, adding one of these things can double triple, you know, quintuple the amount of time it takes to do something fairly simple. And so you have to be very, very sensitive and you have to be very, very careful about each individual action that you're doing in this sort of interloop. Another form of indirection is primitive math and this is kind of a tricky one because it's really hard to know when your math is primitive. If you don't type int it, even if you use unchecked add and some other sort of stuff, if you don't properly give the compiler all the hints that it needs, it will default to the safest option and it won't tell you. And so when you are adding together two numbers and you want to know, you absolutely got to know that it's efficient, you basically have two options which is to attach a profiler like your kit or use gen class to compile it and decompile it. And neither of these are fun options, right? Neither of these are things that I want to be doing with my day. And again, this is not something I do during the day, but in sort of pursuit of this goal of let's make this as fast as you can, let's write C in closure, which may be an insane goal. This is the sort of lengths that you have to go to. So I wrote a test. It's a benchmark. It plays a random nine by nine game and I want to see how fast that is. And it's about 120 microseconds. And that means that we basically got to it, right? We're playing 10,000 games a second. We could go a little bit further. I haven't actually implemented this yet, but some things that I've been playing around with are we can emulate C style structs, heterogeneous data structures in byte buffers. This is significantly faster for the reason that we have this sort of contiguous memory and we take advantage of this architecture, which is built around the C style accesses. We can also go one step further. And one of the reasons that people don't do this in Java very often is that if we, for instance, want to get the nth element and we want to get a field and then a subfield, we have to create intermediate data structures that represent each of these intermediate steps. In C, that's not the case. In C, what it'll do is it'll look at it and it'll say, oh, you're going to this index and you're getting this field and all it'll do is it'll just do a couple multiplies, a few additions, and then it'll read from that position of memory. And this is unsafe for all the obvious reasons. But we can potentially do local analysis in closure, assuming that we have the right sort of information and everything like that. And we can do this safely as long as we have the fallback of all these intermediate data structures. And so this is actually kind of interesting for a number of reasons, one of which I'll talk about. This was alluded to in the last talk. Co-processors are coming back, right? We have GPUs. We have the PS3 cell processor. We have instructions on the x86 chips that we don't have access to, vector instructions and everything like that. And these are not things that we can really talk to very well. This is not something that's really within our purview. But what we can do is we can use something like OpenCL, which allows us to create a C like description of a highly parallel task. And it gets compiled at runtime. And it gets sent across the wire. And then we can send it data. And it will work on it. And this task, actually, in many ways, is a pure function. It takes data. It returns new data. Typically, it is, many of these are used in composition with each other. And whereas C, which is sort of what a lot of these APIs are written in, is a very good way to represent the tasks themselves, it makes for a very poor way to choreograph, to compose. And while closure is not uniquely qualified to do that sort of choreography, it's well positioned, right? This is sort of something that we do very well. We compose things. We talk about things that don't have side effects. We talk about many things being used in concert with each other. And since we're doing runtime compilation, we're not really limited by the performance per se. We're limited by our expressiveness. And so closure can be an expressive choreographer of other computing devices. And this is hypothetical. This is not something that has been realized. But I think it's an interesting sort of niche that closure could fill, which hasn't really seen a lot of attention. And so I think that this is interesting. So with that, I wrote a Go AI that's called Pushkin. It's kind of strong, not really. What it can do is it can play against GNUGO, and it can win sometimes. So it doesn't make stupid moves. It's not completely brain dead. But it doesn't play particularly deep moves either. It tends to get tripped up on particular life and death exchanges. I think that a human player realizing this vulnerability could probably never lose to it. But this is because I've only taken the sort of first step here. There are two spectrums upon which I can improve. Right now, it's playing very naive Monte Carlo play-outs. I can make it stronger. I can get it closer from possible to plausible. I could also take the multi-arm bandit mechanism and I could try to seed it with something that is my best guess as to the best move. Remember that we're converging on the best solution. So the faster that we can converge, the more we can exploit as opposed to explore. And so these are things that remain to be done. This is something that I think is a really fun sandbox for me to play in for other people to explore. And I want to emphasize that this is not a well-defined problem space or a well-explored one, rather. There have been a couple of people that have written things. Typically, they, as I did, stayed very close to the literature. And at the same time, you know, they write and see and they put all this engineering effort into it. There's not a lot of motivation for them to go wild. And there are performance constraints and, you know, on some level it is just more painful to write code this way. But I think that with the right libraries, with the right approaches, with the right sort of best practices, closure could be a really effective tool for exploring these sorts of problems. And, you know, if anyone is interested in this, I encourage you to come talk to me afterwards. And, you know, I will talk a lot more about this. I guarantee you. With that, I'll also say that, you know, as I said, I work for Factual. There are, I believe, eight or nine of us here at this conference. We do not do anything related to go. But we do handle some pretty interesting data analysis and systems engineering problems. Closure is heavily featured. And if you want to hear more about that, come talk to me. Come talk to any of us. And we will gladly also talk your ears off. And with that, I'll leave the floor open to questions.