 My name is Rob. I work at the Gemini Observatory. So I write software for two of these big machines. It's pretty cool. It's not as glamorous as you might think, but I enjoy it. This is the only fun slide I have, so enjoy it, because it's over now. Right, so what I want to do is talk about JDBC. Who likes JDBC? No, nobody likes JDBC. So I want to talk about the implementation of Doobie, which is a pure functional database layer that I wrote for Scala. So more than talking about all the awesome features of Doobie, I want to talk about the implementation and try and motivate this idea of treating programs as composable values, rather than intangible processes. And I think you get a lot of good benefits from that. So I'm going to talk about the implementation, and then at the end, I'll sort of backfill some of the information on Doobie itself, so you can kind of see what the end result is. So the takeaways really are that free monads over free functors are really cool. I'm going to explain what that means. As I just said, we can think of programs as composable values. And this technique that I used is something that you can apply generally. You can use it to make terrible APIs tolerable and turn something awful into something that you can use to build up programs that you can reason about. So let me describe the problem to you, and then we'll go a little further. So here's a little JDBC program. You give it a result set, and it reads a string from the first column and an int from the second column, and then it constructs a person and returns it. So what's wrong with this program? If you're a Java programmer, this is what you would write, more or less. But we can do a lot better than this. There are quite a few things wrong with it. So the first thing is this result set is a managed resource. So if you accidentally close over that and pass it to a future, then the result set may have been closed before you try and use it in the future, in which case the future will blow up, or you might pass it to another thread, in which case you get weird undefined behavior with two threads accessing the result set at the same time. So you don't want to pass something this valuable to user code. You want to kind of hide that. So that's a problem. So what else? We've got side effects. So get string and get int are not pure functions. They depend on the internal state of this result set. So that's no good. And then there's not an obvious way to compose this. There are not obvious little units of work here. You can build that up, and that's what we're going to spend the next few slides doing. So I'm going to sort of claim that the way you can improve this is by turning this program into a data structure, turning it into a value. And you're just going to have to trust me for the next few slides, because I'm going to go through some mechanics. And then hopefully I'll be able to justify to you why this exercise is worthwhile. So here's our game plan. First, we're going to talk about all these primitive operations we can perform on a result set. That's just going to be the data type that we're using right now. And we'll treat these as our smallest, meaningful programs. Then we're going to define some ways to combine these programs together, to make bigger programs. And we're still talking about just composing data now. So eventually we have to build an interpreter that can take this program value and actually do something useful with it, because at the end of the day, we really do want to talk to the database and get worked on. So what does this look like? So what we do is we define a data type that describes the operations we can perform, in this case, on a result set. So we have a constructor for every method on a result set. So there are, I don't know, 190 operations. And then this type is parameterized on the return type of that operation. And this just turns out to be, that's just kind of the way you do it to make this mechanism work. So this is great. So just given this description of the operations we can perform, we could write a little program. That's just a list of instructions that we want to do. And then we could write an interpreter that takes this list and a result set and gives us a list of answers back. That's not going to be very useful. You get a list any back, and there's no composition or anything. But you can kind of see where I'm going. We're describing things that we want to do, and then later on we're going to interpret it. So what we want to do is figure out how we can take these operations and compose them and build bigger, more interesting programs. So one of the things we might want to do is if we have an operation that produces some value of type A, another operation that produces a value of type B, we might want to combine them together using some function to produce another program that yields a value of type C. That seems useful. I'm going to claim that that's useful. Another thing we might want to do is just change the result of a program. So say we read an int, and what we really want is a float. We should just be able to apply dot to float to that result and get a new program that returns the value of the type that we want. So we just need a little operation. They can just go straight from one type to another. A more interesting problem, a more interesting program we might want to write is to take a program that produces a value of type D and look at that result. And based on that result, decide what to do next. We can go this way or we can go this way. So something like that, where we know we're going to produce something of type E, but we don't know which one we're going to use. It depends on what we produced before. So for instance, we might want to say if D is an operation that says, is there another record in the result set? We want to say, if there is, we'll read a person and return that into a sum. If not, we'll just return none. And a consequence of this is that we don't really have a way to construct a program that returns a none. So we need another operation that's just a way to just make a trivial program that computes any value we can think of. And that gives us some freedom to just return arbitrary things as part of this program we're putting together. So these are well-defined operations. So you may see what I'm getting at. So the purple one is just monadic unit. The orange one is flat map. And this might not be the way you think about flat map, but that's what flat map is. It takes some value from some computation and you can look at it and use that to determine the next computation. The one in the middle, of course, is just map. And the first one actually, it's called liftA2. And we'll see that later when we'll just use the Cinnabon operator instead of calling it liftA2. So the upshot is, if we want to combine this stuff together, these little primitive operations, what we really want is a monad. Yeah. Yeah, yeah, yeah, sorry. It turns out that you don't have to define all these operations. Like if you just have flat map and unit, define the others in terms of those two. So if you have a monad, you get all of these operations and others, and we'll talk about some of them. That's a good question. So if we had a monad instance for result setup, and I'm just assuming Scala's everywhere. So if we had a monad instance, then we could write this program below. So we say, OK, for a name produced by get string and an age produced by get int, we can yield a person constructed from those values. We have no idea what this would mean, because we haven't defined flat map or map or anything. But in any case, we can't do this, because we don't have a monad instance. So what if we just borrow a monad instance? So be seated. So I'm going to have to kind of hand wave through this. The content of this slide could have been a two-hour workshop yesterday. It turns out that you don't really have to internalize the way this stuff works just to go through the mechanics. So that's the part I'm going to show you. OK, so there's a data type called free. Free is a monad for any functor f. That doesn't help us. The result set up isn't a functor either. But there's this other thing called coionata. And coionata s is a functor for any s at all, any type constructed with a single hole in it. So that might be helpful. We can compose these. And so free of coionata of anything is a monad for anything, any type at all, any type constructed at all, that has a single hole in it. And Scala abbreviates that and calls it free c. So if we just set s equal to result set up, let's see what happens. OK, so here's what it looks like. So we've got some imports. So we're declaring a new type called result set io that is just the free monad over the free functor of result set up. So what do we do next? Well, we define a smart constructor for all of these methods on result set. And then we lift the operation associated with, we lift the result set up constructor into this new type using lift fc. So what we have now is a way to construct these values of type result set io, which is a monad. So result set io is the free monad. So now we can write this program. So we can say for name taken from get string, age taken from get int, yield a new person. And I claim that this is better than the program we saw on the second slide because there's no result set. There's nothing to leak. The user who's writing this can't observe the result set. All of these things are just values. All we're doing is constructing data. There are no side effects. These are all just pure values. OK, and now we're composing stuff. We've got little programs. We've got two little programs, and we're putting them together to make a third larger program. And notice that the service complexity of this is the same. It's just a result set io of something. We've composed two small things. We've got something back that has the same shape. And this is what we want from composition. So I haven't talked at all about what map and flat map mean in this context. All we're doing is just mechanically putting stuff together and hoping that it's going to mean something useful for us. And I'm going to explain how to do that. But first, what I want to do is talk about other things you can do if you have a monadic API like this beyond just using four comprehensions, because they're richer than that. So one of the things you get is the functor operations, because all monads are functors. So in this case, let's say we have a program that can get along at some index in the result set. And what we want is a date. Well, we can just use map and pass the result into the constructor for Java util date. Now we have a way to construct programs that read dates at particular offsets in the result set. And all we had to do was use map. There are other interesting operations, by the way, that you get from functor. You should look at these things in the ScalaZ source, because they're all really useful functions. They use them all the time. So what else? We've got applicative operations that come from this blue operation we talked about before, plus the purple one. You need that one as well. So what we can do is we can get person in a different way. We can say get string from the first index, oink, get int from the second index, and we map that to the person constructor. And that turns out to be the same program written using applicative style. So what else can you do with applicative? Well, Brendan talked about this earlier. You can discard one value and evaluate it only for its effect. So what we can do here is say, go to the next row in the result set, throw away the boolean it returns. We'll just be optimistic and assume there's another row, and then get a person. And this operator is called right shark. There's right shark and left shark. I just coined that just now. So that's kind of interesting. But we can do more than that. We can say, all right, I want to read 10 people. So I can say, get next person, I can replicate that effect 10 times. And what I get back is a result set containing a list of 10 people. So it'll go through 10 rows and read a person from each. Well, how does that work? So the first thing it does is it says, okay, we're just gonna make a list and we'll just fill it up with that effect. That's all we're doing. We have got that piece of data that describes how to read a person. We're just gonna make a list of it, right? And look at the type of that. The type of that is list of result set I owe a person. Okay, and now magically, as if by magic, we just call sequence and it flips those type constructors around. So now we have a result set of list of person which is what we want, okay? So sequence and traverse have been mentioned by several people today in Scala Talks. It is awesome. Sequence, awesome, thus. It's one of the coolest things in all of programming. You really have to get familiar with sequence and traverse. It will really save you, especially if you're writing monadic style programs because this gives you a way to sort of iterate over structures of things. And you get this because list is a traversable functor. It doesn't work on all structures, just some. Okay, okay, now the strongest operations we get are from the monad, so flat map and unit. So now we can write a program like this. And what this says is we'll look at the result of calling next. If it's true, we'll get a person and then use this map operation and turn it into an option. And if it's false, we'll return none. So this program may or may not give us a person back when we run it. Furthermore, we can iterate if we have a monad. So here what we can do is we can say, well, let's evaluate next. And as long as it's giving us true values, we'll run getPerson and we'll accumulate those results in a vector, right? Okay, and seriously, that's how you would write that in Doobie, and that totally works. And it's stack safe. It means this actually legitimately runs. So this is a program that reads everything in a result set into a vector of people, which is pretty cool. All right, so here's kind of the amazing part of all this. So far, we can write these programs, but they don't mean anything. We have no way to actually run this program and do anything useful with it. So let's talk about how we do that. So to run this program, we interpret it into some target monad. So we have result set IO that we've been working with, and we're gonna interpret it into something like task or IO, typically, as a target monad. And to do this, we need to provide some mapping that takes any result set up, and that's the original, super simple data type that we created. We just need to be able to map from those to this monadic type. Okay, and a function like that, it's just a universally quantified function, and we call it a natural transformation that's written with a squiggly thing in ScalaZ. So let's see what that looks like. It's really quite simple. So here's a very simple interpreter. It just says, okay, given a result set, I can produce this function. And all it does is, it says, for any of these result set ops that I get passed, just construct a ScalaZ IO value that performs the effect that we want. So they just line up one to one. Okay, so we have the result set ops here. And then the target monad values on the other side. Okay, so we never had to define what map and flat map mean. We never had to define any of these monad operations because now we've just shown how to turn this thing into IO, and we get all that stuff for free from the definition on IO. So if we wanna actually run this thing, there's this method called runFC that takes our program, and this is the thing written in result set IO, and then the natural transformation. And then we get an IO out the other side, right? So if we had this program that we looked at before, this accumulates in a list instead of a vector, but it's the same thing. If we call to IO and give it the program and result set, unsafe perform IO, and we actually get a regular Scala list of person. Okay, right, so what's do be then? We've gone this far in running out of time and haven't even talked about do be. So do be is basically what you just saw, but it's for all of JDBC. So we have all of these different contexts we can use to build programs. There's pure functional support for all the primitive operations in JDBC. And you may have thought thus far, this is all very mechanical. It's like really boilerplating. It's just kind of a formula that doesn't exercise you go through, and that's true. It turns out that the low level API in do be is entirely machine generated. I just reflect on the JDBC APIs and spit out the source code. So what I've shown you so far is very primitive. Do be would not be very useful if all you had were just this low level toolkit. So do be provides a lot of other stuff on top. So let's talk about that. So first thing is exception handling. Anything you do with the database is gonna throw exceptions. It's just a fact of life. It's non-deterministic IO. So we have this primitive operation called attempt that takes a connection IO of A or results that are in any of these things. Some, in this case, connection IO of A and lifts it into connection IO of throwable or A. And what that allows you to do is take a program that might throw an exception when you run it and make a new program that will return a disjunction with the error on one side. And using this primitive, we can build up all these other handlers. So we have these general handlers for any kind of exception. And then some that are specific to SQL exception because you might say, I only wanna trap like primary key violations and handle those and let everything else propagate. And then actually for Postgres who are nice enough to actually publish a spec for their error codes, there are hundreds of these special purpose combinators. So you can say, okay, well, on privilege, not granted, you can have a handler for that specific error. And that's all built up on top of that attempt primitive. Another thing that's fairly obvious is we have this kind of irritating program that we keep looking at where you read a string from one index and then the next index you read and you construct something from it. Well, using type classes, of course, we can abstract over the return type and just have a get operation. We can also generalize that to work on tuples and using shapeless, we can generalize it to work on product types. So we can just get a person at index one. And of course this for comprehension doesn't do anything. So it's the same as this. Normally when you're reading from a result set, you're gonna start at index one anyway so we can default that away. So it turns out that program that we've been talking about this whole time is actually that's how you would write it in Doobie. The type class machinery just kind of makes it all disappear. Okay, what else? Streaming. And this dovetails pretty nicely with the earlier talk. So this is one way that we've seen that we can read a result set into a list of people. Well, there's another way we can do it. We can just ask for a process of person. And now we can write a program like this. So we can say, okay, we'll take people and we'll filter and we'll take and we'll do all kinds of fancy stuff. And then we can terminate it and just say, okay, well I wanna collect all this stuff into a list. And what do we get back? We just get a program back. It's the same type we've been dealing with. So we can just take this hunk of streaming logic and just insert it into our program along with any other, just like any other action. Just like any other program. So what else? You saw that huge list of these algebras that I've defined. But we need a way to combine them, right? So the high level API in Doobie provides the ability to nest programs together. So, and this sort of mirrors the way a database program will tend to work. You'll have a long lasting connection pool. Then you might construct a connection and you have a connection program and then you might prepare a statement and you have a prepared statement program. You have a result set and you have a result set program and those results bubble all the way back up to the top. Some of those patterns of nesting are so common you just kind of get them for free with Doobie out of the box. So, things like prepare a statement and read the results into it, set some parameters and read the results into a list, right? You don't need to write a complicated program to do that. It's very common, so we can do it for you. Okay, so, here's an example. This is a value, it's called a query zero. I don't know why I didn't put an annotation on it. And so there's a SQL literal and it says, okay, I want to read some stuff and select it into, I want to produce values of type country. I should note that this is typed. That interpolated value gets turned into a question mark and compiled down to a proper set int JDBC operation. So that's safe, that's not an injection threat right there. And we yield country values. So how can we actually do something with this? Well, we can just say, we can just construct one and say we want the country's bigger than 100 million people or so. We want to view the results as a process. So we can take five, collapse it into a list and then transact is something I don't have time to talk about. Basically, this XA thing is a transactor that abstracts over connection pools. It gives us a way to abstract away the differences between methods of managing connections. So when we do that, we get a task. It's just a ScalaZ task and we can run it and print the results out. There's also YOLO mode for, this is just for experimenting around in the REPL and what it does is it takes a transaction, sort of makes it an ambient thing in the environment. So you can play around with queries and if you just say .quick.run, it will run the query and spit the results out to the REPL, it's just a good way to play around. A more impressive thing I think is that you can use YOLO mode to type check your queries. So I can say, okay, well I want to construct this query object and say .check.run and it will go out to the database and prepare the statement, fetch the metadata and compare it with all the type mappings that you have defined. So right here it's saying the SQL's okay, the parameter type is okay, first two columns are okay. The last column, actually we've just been lucky that our query didn't blow up because that's nullable and we were reading it into a non-option type. So it gives us a warning about that and it actually says you can either change the schema type to not null or change the scala type to option a big decimal. Pardon? Cool, so it gives you advice, it doesn't just say something's wrong, it gives you some advice for how to fix it which I think is pretty cool. And you can also use this in your unit test. So there's a specs plug-in or just a trait you can extend in your specs test and you just mention all of your queries and then it'll go out as part of your unit test and it'll fill your tests if something doesn't line up right. You had a question I think. I'm sorry? Oh yeah, yeah, yeah. So there's this Transactor object and the Transactor has a YOLO module on it and you just import that module and then it makes that Transactor sort of the default one for your rebel session. It's a lot simpler than John's thing. Yeah, yeah, yeah. It's just a module that you import. Yeah. If you have nested case class, like can it join you? If you have nested, yeah. So if you wanna, so right, yeah. So here I'm just selecting, let's go back and look at the definition. So I'm selecting countries, but I could be selecting a tuple of countries, of options, of whatever. They can be nested to any degree as long as eventually they all flatten out to a linear series. So yeah, it handles nesting of row composite types. If that was your question, maybe it wasn't. Can any of you remove like actually the SQL statement? Because can you infer this SQL statement? Infer the SQL? No, so Doobie doesn't have any knowledge of your schema. There's no knowledge at all. So Slick, you provide a sort of a scholar side representation of your schema and you can use that to generate SQL. Doobie doesn't do that. It could, you could build something up, but it's just me and I haven't. I haven't tried to do that. It's a very, very hard problem and you can look at the complexity of Slick and see that it's really a very hard problem. Okay, what else do I have? I'll say there are more arms littering too. Yeah, that's right, exactly. Okay, I'm about done. So I just want to mention a few other things. Custom type mappings are very, very, very, very simple. It's just by invariant functors. You can, as long as you can turn it into something that you know how to read and turn it out of something you know how to read, then you can add your own type very easily. Connection pooling, I provide a connection pooling, a connection pool that uses Hikari. It's very easy to adapt your own if that's not what you want to use. There's some syntax to make dealing with these types a little bit easier. I added some stuff for process like that dot list is something that isn't part of process. And I have to say the Postgres support is really good. It's better than Postgres support in any other database library I know of. So I support geometric types, arrays, post-GIS, listen and notify which is cool because you can have the database tell you when stuff changes, copying out and some other good stuff. So I'm out of time. So for more information on this, that's the repo. There's a getter channels and I'm usually there. And there's also a sort of an ebook that I'm working on that's, I don't know, 14 or 15 chapters now. And it explains sort of in a task oriented way. Here's how you select data. Here's how you update. Here's how you map new types in and that sort of thing. So I think the doc is pretty good. So, and that's why I didn't go into a ton of detail on what Undooby features because you can just read this and learn it. I'm out of time. Thanks a lot.