 I'm going to talk about rapture, which is a collection of libraries for doing kind of everyday tasks including I.O, JSON processing, cryptography, a few other things that I won't talk about, but the focus of the talk today will be those things. I've been travelling around Europe the last couple of months giving this talk to various people and various Scala user groups in different cities in Poland and Germany and so on. I had quite a hectic schedule, and they all kind of ran into each other and ended up being quite tired. But it's really good to be here in Portland, and good to see so many of you here. This is actually going to be probably the final outing for this talk. No, no. Down. That's fine, yeah. Let me leave them, okay? This will probably be the final outing. Hopefully this means that the talk will be well-practised, quite slick. No, probably not. Anyway, people who don't know me. People have made the complaint before that I don't explain who I am at the beginning of my talks, and I just dive straight into them. So I don't really like giving a sort of long history of what I've done and so on. So I'll hand over to Paul Phillips, who keynoted here last year, who very kindly... I think I must have said something to offend him, but he put this up as one of his slides in a talk he gave in Boulder. So yeah, it's taken me ten years to gain ten years' worth of programme experience. I think it took Paul one year to gain that much. And then he goes off and slags off Western Europe. Yeah, I think back then I was... I think the noise we've most about this is how blurry this photo is. There's loads of very sharp photos of me on the internet. This is not one of them. I think that's the noise we've most. Anyway, I quite like this John Pretty Slow. I was very pleased with that. So for Scala Days in Berlin this year, I had that put onto a t-shirt. I am John Pretty Slow and quite proud of it. More recently I've been working on Rapture. I'm now kind of full-time on Rapture. I quit my job in August and moved out of my flat as well so I could go travelling and telling people about Rapture as I am today. So effectively I am unemployed and homeless. So thank you everyone who's been very welcoming to me here and very charitable. The format of the talk today will be a series of one-liners. So I've called it the art of the one-liner. What I will do is I'll show you a line of code. It's the kind of thing you can type into the REPL and then the response you get from the REPL. So to get started... Oh by the way, if anyone has any questions at all as I'm speaking, just shout out or if you're reluctant to do that, put your hand up and I might notice you but I prefer to shout out. I like the feedback. I think an interactive talk is a better one. So this is a line of code from Rapture. This is how you define an HTTP resource. So nothing particularly exciting. We have a URL which is http://rapture.io.sample.json. It's a mini DSL. Nothing particularly exciting. And I'm going to call that a resource. I'll explain what that is in a second. Here's another one. This is a file URL. Looks very similar. So this represents a file on disk in the same way that the previous one represented an HTTP resource. That was actually the old way of writing them. So that was in Rapture up until the most recent release. We can describe an HTTP URL like this now. This is a string context. So you've probably seen a similar... Who thinks this looks a little bit weird? So if you... Eric, surely not. Okay, I'll have to ask a different question. Who thinks this looks completely normal? Really? Okay. So you've probably seen similar things with an S, locates S there which is an interpolated string. And what that means is that the content you put in here is interpreted in some special way. It's not a string. It actually invokes a method on what's called a string context and interpreted in some interesting way, which means as a result we get an HTTP URL. So that's all cool. We get to use the pretty standard syntax without having to convert to those slashes and strings as I showed you before. So this is better. This is easier. Now notice that we get an HTTP URL as a return type here. If we have a file here, notice that this is exactly the same. Pretty much, apart from the content here. But the type is different. So at compile time, Rapture knows by virtue of some magic, by which I mean a macro, it works out that this is representing a file URL and therefore it's able to give us a tight result from this. And we can put any kind of scheme in there that Rapture knows about things like class paths and HTTP as well. So that's all fine. Now these are all resources. And I'm using this resource phrase in a very vague sense. There's no trait or interface that represents a resource. A resource is a resource by virtue of having capabilities. And a capability is something like being able to read from that thing or being able to write to it or delete it or rename it. All kind of standard IO tasks that you're probably very familiar with from other libraries. So a resource is a resource not because it inherits from a certain interface or trait. It's a resource because there are capabilities available for it or defined on it. The way they're defined is using the what used to be called the Pimp My Library pattern, but that name has been deprecated in the favour of Rob Norris' name, which is Bedazzle My Library. So over the course of the talk today I will be bedazzling these resources with various capabilities. And it's a much better name, isn't it? I think everyone should go out and start using Bedazzle My Library instead of Pimp My Library. So there we are. This is fine in the case where we know at compile time what kind of resource it is. So we know this is the file it's there literally in the code so we can work out this is the file URL. If, for example, it comes from user input, there's no way at compile time we can get the type from that. It's just not known. So we have to specify. We have to say HTTP.pars and this is an ordinary string here. And then we'll get an HTTP URL. There's also a corresponding file.pars and the same for all the other resource types. So our first capability that we'll see is slurping. So slurping is to pull all the data from that resource into memory. It takes a type parameter which is byte. And as a consequence of that we get this thing back which is a bytes type. Bites is part of Rapture. It's a very, very simple wrapper around an array of bytes. Why have I done that? I don't like working with an array of bytes and we generally as functional programmers we prefer immutable data types. So although this is just backed by an array of bytes it's a simple wrapper around it and also we get this nice two string method so you can see exactly what the content is. Assuming you can read hex. You all can, right? It's slightly more complicated than that but that's the basic idea. So the question was do I have a type class for each possible parameter that goes here? The answer is effectively yes and it's actually each combination of parameter here and resource type. So if you introduce your own resource type which you're perfectly capable of doing or you're able to redefine an existing thing as a resource you specify a type class that tells Rapture how to read it, how to write it how to set the size of it and so on and in the case of slurping it needs to read it and we provide two one for bytes and one for characters but you could provide others as well. So this is bytes Bites aren't that useful in many situations we'd much rather work with strings because we can actually read them then. So what we can do is we can say slurp characters now you've probably noticed this big red error message here I'll read that to you. It says error cannot find implicit reader for file URL resources. File URL resources can only be read if a reader implicit exists within scope. Now that's not particularly meaningful except in the context of the question that was just asked over there the reader is the type class but it goes on to say note that if you're working with character data you will require an implicit character encoding for example import encodings.system or import encodings.utf8 well this is actually telling us exactly what we need to do to fix this error. Well I'm trying to address the fix the problem that Java introduced which is to make the really really really stupid assumption that if you don't specify the encoding that you must be using the system encoding. Now what that means is why do I think that's such a stupid decision? Well it means that the code you write if you don't specify which character encoding you're using you write some code it'll work fine on your machine it'll probably work fine on a friend's machine or a colleague's machine it might work on all the development machines in your organisation because maybe you're all using OSX and then you go and deploy it onto a server which has a different system encoding and suddenly it stops working just because it's assuming the system encoding some environmental information rather than having it specified in the code it starts failing in probably quite a subtle way to probably start seeing unusual characters or question marks in the strings in the system it probably runs fine without exceptions but you start seeing subtle degradation in the software so this completely undermines Java's portability so I've done away with that if you want to do anything in Rapture with characters with strings writing between strings and bytes you have to specify an implicit encoding and we do that like this import encodings.inr case we're going to use utf8 but you can specify anything here note that I've got the factics there which allow us to write this in the canonical form with the dash there and then we try exactly the same thing this is the same line we had before and we get this string result here so the return type and the characters it's not bytes it is now a string so is that all okay? I've only had one question so far I think does that all make sense? cool right what's next? so we're going to move on to some JSON processing now so rather conveniently this this string looks a lot like it's probably some JSON so given some JSON in a string what we probably want to do is parse it so I can call jason.parse this is in the raptor jason library now and what we get is well this looks like it's exactly the same thing except the type is now this JSON type which means that it's successfully parsed and it is it is in some internal representation of that structured JSON type okay now that was actually a two-step process because first we had to slurp it and then we had to parse it when I get round to the oh release of raptor.io you'll be able to do this you'll be able to specify simply any resource sorry this one liner splits onto multiple lines this may be a theme as we approach the end we may get up to three lines at some point anyway we can specify this as one line of code even if it's not one projector slide we can specify any readable resource here and it will be happy to parse that it'll read into memory and that will just work this has exactly the same result as the previous one okay so it depends on some stuff I haven't told you yet but in this particular case it will load it all into memory now if you had a jason parser capable of doing the streaming there are ways that that could be that could be made to work in a more efficient way without loading everything into memory I'll tell a bit more about that later ask me about that in about 10 slides time that is comprehensively covered but at the moment so for now if this fails it will for an exception and if that makes you feel uncomfortable that's good because I'm going to fix it later okay I'll answer that in probably 15 slides time any other preemptive questions I do love this it happens every time I give the presentation people anticipate the slides I'm about to show I sometimes wonder whether I just shouldn't have them and I should have a completely random set to see if people anticipate them as well anyway we've got this jason type and I'm calling dot candidates on it now does my jason type have a candidates method well it looks like it does because I'm calling dot candidates I promise this compiles and it gives a result type jason but did I anticipate that candidates which is from this this key in the jason method no this is scarla code I mean it looks like javascript it's been completely happy to compile this what's actually happening is this is using scarla's dynamic trait who's familiar with dynamic so who's not familiar and who didn't put their hand up who still didn't put yeah okay so probably about half and half I think so the dynamic trait is to ask you to tell scarla how to interpret a method call to a method that doesn't exist so what scarla does is it converts this call here into an application to the select dynamic method passing candidates as a string so all this is it's syntactic sugar in the pure sense so this is you can think of this as a dynamic access so we are very much in the dynamic world of jason here we can call dot anything here if you like this happens to to be something that exists and it works fine we can also index into that array so this you'll see returned an array we've got the square bracket there if we access the first element that will give us this result here note that every time I do this it returns something of type jason so jason is like the the dynamic type everything you do if you call dot name on that we've still got type jason here even though this is clearly a string the compiler doesn't know that it only happens to be known at run time that this is a string so how do we get from this dynamic world of jason because we can't call dot substring or dot index of on this because that will be treated as if it's an index into a into a jason object we want a real string to come out of this so that we can actually do statically tight code what we do is we call dot as we call dot as string in this case and it's perfectly happy to interpret that as a string note that the quotes have gone missing because it's no longer a jason string it is just a plain string and that's how the REPL will display that so we're back in the static world and we can do all the normal static things that we love about Scala and we can leave behind that nasty dynamic mess sorry say that again so it does throw an exception and the exception will be a type mismatch exception but if everybody's getting really uncomfortable with exceptions I still think that's a good thing because I will still fix that it will throw one of two things generally it will either throw a type mismatch exception or a missing value exception those are the two possibilities we can get an int as well so there we are, there's the value now this is actually something I tweeted me last night and said well I've got some I've got some jason code where my values are always integers but they might be in quotes or not in quotes so they might be represented as jason strings or they might be as represented as ints how do I do that well what you can do is you can rewrite your the extractor this is the type class which tells raptor jason how to extract integers you can compose existing extractors this is the new slide today because someone tweeted me last night with this question and the solution is quite simple so we can we can access the extractor type class which tells raptor jason how to do the extraction like that and then we can combine it with another one using or else so what it's saying is try this one first try and extract an int using the standard canonical int extractor and if that fails try and extract the string and then map the result to an integer so the return type of this is an extractor extractor of an int for jason as it turns out and then it will just work if you try and extract something that might be an int in jason might be a jason string it will always return an int using this if it's a boolean then you still possibly get a type mismatch exception so it's all very well being able to access single primitive values like integer strings booleans and so on but we probably want to work with more structured data as well so a case class let's say we want to extract a candidate now you probably noticed that the jason I was using was some sample stuff representing the last presidential election so there are two candidates in that jason one of them is Barack Obama and he has a name and an age which we extracted last time we can define this case class candidate with the parameters name and age here and we can actually just extract straight into that case class with no other specification required and we get this instantiated case class here so in one line well two lines we've got the definition and then we've got the extraction here we're pulling out structured statically typed results from our jason so what it does is it looks at every parameter in the case class and if it's able to extract that parameter it's able to extract the entire case class so basically they compose by parameters based on the name of those parameters so it assumes that the key inside the jason map corresponds to the name of the parameter in the case class is that okay it doesn't, by default it doesn't care about any additional ones that it's not looking at so I think I think it's not possible at the moment to to tell it to fail on extraction if there are extraneous keys there but it's something I would like to provide it's something I provide in a similar way later on which I'll show you but it is configurable using implicit to, or will be configurable using implicit to to specify how that should behave your five slides ahead of me again yes, you can do that right so I promised an explanation of how I handle errors so how am I doing for time I've got 20 minutes left okay this might get faster and I might skip it so we want to have some people want very quick results they want to get a jason value back that that will give you a value for example in the REPL that you can access straight away or have an exception thrown if it fails because in the REPL that's a very convenient way of doing things so everything I've shown you so far works like that now I invented this idea called modes of getting results from methods which might fail so parsing is a fallible method it could fail extracting a particular value type might fail I've already said it might return a type mismatcher it might return a missing value exception but by default they'll be thrown rather than handled in any nice way if you import this if you import modes.return.try then exactly the same thing we did before the return type will change this probably seems very magic and it is a bit magic but it's not macros this time what this does is it uses a mode type class which allows it to wrap the result in a nicer way of handling it so in this case we've returned a try of the candidate and when we call as that is the point at which this will fail and therefore that is the point at which we are returning a try alternatively this was kind of a discovery rather than by design you can have a mode that will return a future so every operation you call for example this slurp method here will get dispatched to a thread pool an execution context based on whatever your implicit execution context is and this will be handled on a different thread and there are other modes as well so in 25 minutes or so Brendon will be talking about ScalaZ and he will talk about validations and other other special return types for handling errors in ScalaZ you can use they are all completely pluggable into Rapture's modes and modes used extensively in all the methods anything that might fail will be affected by the implicit mode that is in scope at any one time so I haven't given you a lot of detail on how they work it is quite complicated but not macros so I'm showing you rather than explaining it so you'll have to imagine how I made this work but it's pretty useful if you have different contexts and you want to use the same API from my point of view I didn't have to implement this API for a dozen different possible return types so you can compose modes the mode is an implicit value what you can do is say implicit my mode equals modes.return either compose modes.return future and that will put one within the other and you can try it and see what you get right so we actually had several back to our Jason we had several candidates so what we can do is we can try and extract them as a list now we can extract a list of candidates if we can extract a candidate I've demonstrated we can extract a candidate therefore this will return a list of candidates here so was Paul Phillips talk here last year the one where he completely slated the collections library or was it the other one it was the collections library yeah okay well this intracts really nicely with the Scala collections library because you can extract any type you like so stack is one of the more obscure types in the collection library but you can completely extract this that's fine you get a stack or even something mutable like an asset that all just works fine and we can even define another case class so this is the question about composition of case classes provided candidates is extractable well it's an index sequence of candidates and a year well that's trivially extractable we can therefore extract an election so given this one line here plus the definition of an election and the definition of a candidate we can actually extract this entire statically typed result from our dynamic JSON so there is a single point of failure which is where we extract this and we are extracting this in one go transforming the entirety of our dynamically typed JSON into statically typed Scala objects and obviously depending on whether you had a mode like return try present this would be wrapped in a try so it might be a success or a failure and then you could handle the failure types appropriately so I think this is pretty neat that we have such a concise way of extracting extracting data from JSON oh yeah question in front ah no no so the extractors for case classes are generated automatically is that what you meant oh there is a macro we will examine the case class definitions and check that it is possible to extract all of the parameters and if it is then it will construct at the point of the macro invocation a new extractor type class instance that will allow it to allow this to work and that happens recursively so you can have nested nested structure there yeah Eric so it will I think it will it won't cache them it probably ought to it would be nice if it did let's talk about that later it sounds like a property we've already solved for something else possibly we can also create JSON so this is another string context we put our JSON straight into the string there and this is the result we get that's pretty simple we can also substitute values in so this will convert this into the appropriate JSON type which gets substituted in place we can add single values to a map we just use this syntax here which probably looks a little bit weird this is kind of a bit like a lens into the the vote so we are specifying the path into the into the JSON type this is meant to be equivalent to how you would add a value to a map you would maybe have a string key here and then you'd have the value there with JSON you don't have a simple key it's actually a path into the structure so this is specifying the year value there that we're adding to the original now this is maybe weird this is pattern matching on JSON who thinks this looks weird yeah it should look weird I don't know anyone else who's using this particular feature of Scala apart from quasi but you can pattern match on some JSON so what this is saying is our vote this is JSON type we're going to try and match that against some structure so we're saying first of all is it an object is it an object that has a vote key does that vote key represent another object which itself has a name and if all those things are true then we will extract the value at that vote.name value into the identifier C which will come out as something typed as JSON so C is JSON therefore we can call dot as string in the same way as we did before to move again from the dynamic world of JSON to the static world of Scala we have to do this and we get a string out and we can have multiple things matched at the same time we only have to specify here the the structure that we're matching on any any extra keys and by default won't be matched yeah a compile time it will check that this is valid JSON so if you missed off a quote or a brace somewhere it will tell you a compile time yeah not really I've thought about ways of doing this I think Scala would need to change a little bit first you could have an additional extractor in there like you could define a nested extractor that translates JSON into some specific type but you would need an instance of every single possible JSON type in order to write it there you can't specify the type in square brackets as I think you would like to do inline in the code there unfortunately you could put an if there that checks whether it works okay so someone asked me about have I re-implemented my own JSON parser now I thought that probably Scala has enough JSON parsers already does anyone disagree and also I'm probably not the best person to be implementing a parser I'm not really a performance guy so as Eric identified so what I do is I wrap other people's parsers so one possible parser you can use is the standard library Scala JSON parser is this anyone's favourite Scala so a JSON parser good that's the right answer it is deprecated in Scala 210 or deprecated in 211 and I think it's maybe removed in 211 and deprecated in 210 it is horrendously slow and you don't want to use it the only reason you might want to do this is to avoid any third party dependencies but by importing this it's like a configuration parameter you can choose how your AST will be represented how your parser will work that's one possibility you can use Jackson can we have a little cheer if I mention your favourite JSON parser so Jackson okay how about press oh that's possibly a bit more spray okay we've got a diverse crowd here lift well done well done for persevering all of these are supported by by Raptor Jason Argonaut oh a clap and I never get any cheers for this one yes so is there anyone other than Eric using Jorn you should Jorn is sometimes faster than Jackson so I've been telling everybody I've done this presentation 20 times in Europe every time I say Jorn is faster than Jackson but that is true I haven't been lying actually Eric could you come up to the front a second so I suppose you'd say quite fast would you would you like to wear that oh awesome so this comes about because the way Eric says Jorn is I think exactly the same way as you say my name isn't it which is obviously the most obvious reason to use Jorn rather than anything else so Jorn is a fast so use Jorn rather than anything else if for example you had to use too you had some library code that was using Argonaut and some other code that was using Jorn say this vote has been created in the Argonaut code and we were now in a context where we had Jorn what we can do is we can kind of extract it's not really extracting it's more conversion but we can convert that vote and one type of Jason to another just by calling dot as Jason so that's a really neat way of moving between two different kinds of Jason if you happen to be in the awkward situation where you are maybe two different libraries both of which depend on different Jason libraries and you want them to somehow work together this seems to be something that if you encounter the problem it's incredibly frustrating because what people normally do is they serialize to a string you pass it in the other parser which is just a horrible solution this will actually walk the tree and translate as it goes we can also do mutable Jason at least if we have time I've got ten minutes so we can do mutable Jason who likes mutability yay so it's available if you want to use it in the same way that there's a list buffer that's like a mutable list we have a Jason buffer that's a mutable Jason so we can create an empty one of those there it is just empty parentheses and we can specify that the life value of that is going to be equal to 42 so there it is that has mutated it's a reference to the same object but it now has this life being in here we can even like go deep within deep deep within a nested tree by the way I'm I do keep meaning to change my sample Jason to something less politically or people accuse me of being politically biased it's not deliberate at all I'm not trying to certainly not coming from Europe I'm representing I had a tag my tag says I'm representing Europe today so for me to come over from Europe and start suggesting Republican or Democratic candidates in my sample Jason trying to sway anyone's politics at all I'm in no position to do that and you're quite capable of doing it yourselves so nevertheless Hillary Clinton is my example of another candidate for another election and we can we can add that case class value there and it will serialise the case class to Jason so that is an object in itself any type on the right hand side will be fine as long as it is itself serialisable it's sort of the inverse of extractable and generally speaking anything that's extractable can be serialised back again one thing I'll say about this is that it's although it's we probably don't want to use usability it's actually quite a convenient way of having an inefficient but really lightweight object database that will store case classes that will store primitives that will store collections you basically specify the value that you want to change what you want it to be and it will mutate it in place, in memory when we've done that our JB value we can write that to a file now this is back in the Raptor IO world again JB is a resource it's a Jason buffer how is it a resource by virtue of being readable so a Jason buffer can be read a file can be written to so if you've got something that's readable and something that's writable we can write from one to the other that's all we do so we've then committed that Jason buffer to disk in a single line and we get a summary here which tells us how many bytes were written because I could return a unit but that's not very meaningful so we get a write summary back so that we can at least get some understanding of what happened when we did the write we can also do a copy to copy to is subtly different in the case of two files there's a faster way of doing that than streaming all the data into memory and streaming it out again we can use the the fact that file systems have a pretty efficient copy method that we can use if we know the source file name and the destination file name this will send a message to the file name it's effectively the Java file copy call and this will work a lot more efficiently than the write to method but in most cases they're the same and if the copy isn't available it will fall back to the slurping and writing in method Tom will it respect the loads no it's not that smart I would like to make it that smart maybe we should talk about that later yeah it's it's a reason things like this are a reasonably lightweight oh so I think it said loads I thought this was some like reactive thing oh thanks so will this work with modes yes so this is a fallible method if you import a future then this would be executed executed on a different thread now it's probably more relevant with the write to because that is a streaming write and it doesn't say streaming there it probably should do this is a streaming write a key point to be aware of is that although it's executed on a different thread it's not magically asynchronous it's still a synchronous code it's just not on the main thread so you get a future back instantaneously and one point in that future will be fulfilled when the synchronous code executes on the other thread so we can't magically change how the code runs but we can at least have it work on a different thread to get some some asynchronous was there a question there don't know I haven't got a good answer to that I'm sure there's something I can do oh as in if you're do you mean oh so if you're it's a matter of having a type class that knows how to write to for example whatever kind of resource you want to so I will show you an example I've shown you that one haven't I that's a stream copy Thomas you're about to tell me that I've next couple of minutes I've got some stuff on crypto I'll skip that and I'll tell you about this this is pretty cool in one line we can stream straight from our class path I'm looking at the hash set class straight on to S3 this might be similar to your use case where S3 has some additional connection parameters you define them in an implicit so you have to define your authentication which is the use name and as long as that implicit is available then it's possible to call this so in one line with copying I don't know why the hell you'd want to do this copying straight from your class path to S3 but it's a demonstration of how uniform how uniformly it works and how how consistent it is I hope crypto I will very quickly go over you can look at these slides you get two seconds on each one and just pretend you can see what's happening there it's probably quite I hope these these kind of things are quite intuitive as to what they're doing we've got method called decrypt and so on key.decrypt we're using types extensively we're using implicit for configuration we've got plugable plugable algorithms again it's it's not not that complicated not that exciting really so XML on the other hand is one of the most exciting things on the internet so I heard in 1995 we can so I thought seeing as I'm plugging in different back ends for Jason what would happen if instead of plugging in a Jason parser I plugged in an XML parser and then I just modified the type classes a bit so that it kind of accessed tag names rather than keys and objects so I tried all that and made a few tweets and it just didn't work but it worked well enough that I was able to persevere, modify some things that it's now in a state where you can do most I'll probably say 75% of the things you can do with Rapture Jason so Rapture XML is a very experimental project but it will hopefully give you the same kind of functionality as I've shown you already pattern matching case class extraction collection extraction all of these things but for XML instead of instead of Jason so we can do things like this, we can say XML.Candidates zero one thing we have to do is have empty parentheses there for the age which is a consequence of XML's representation differing slightly from Jason so it's not a perfect match but we get pretty neat syntax probably better than anything that any of the other XML libraries out there do we can also do Beason I've not spent time on this yet but it's a possibility it's very similar to Jason so at some point I will sit down and spend a couple of days hacking out Rapture Beason last slide I've tried to combine everything I can from the presentation so far in this one liner which spreads onto three lines who can tell me what this does by the way you have to guess at what key.encrypt does Eric? so it passes it then part of Jason did everyone hear that? hopefully if you didn't hear it it's reasonably intuitive so I'm quite pleased that in the space of half an hour we've learned enough Rapture Jason and Rapture IO to be able to to read it even if not to write it you would never write this I don't know why you'd want to have this as a single line but it's possible that's the thing that is I think the last slide so thank you for listening