 Okay. Is everybody here programming Scala? Who doesn't do Scala? How many people here have used Scala test or specs 2? Almost everybody has used those. Yeah, so you should really write tests. So, you know, I've been writing Scala code for a long time and used both those libraries. I think I started on Scala test, got frustrated and switched over to specs 2 and got frustrated and so I went off in a corner and said, you know, hey, we really ought to be able to do better and so that was sort of the origin of what I'm going to talk about tonight. I'm not going to cover all that. There's probably not enough time tonight. The good news is all the material I'm not covering. There's good online documentation and you can go follow up and look at examples and documentation. So, testing philosophy. I mean, this is sort of key to what was in mind. One of the things that bothered me about the existing test libraries, they were really big. Bloatware, huge monstrous things. They keep getting bigger all the time. I mean, how big does a testing library have to be? And I think this applies a lot to large systems that we build instead of, you know, building this system that's all things to all people. Build yourself a small kernel, clean, nice concepts and provide nice clean extension mechanisms where you can customize it in any direction you want because you're never going to be able to, I suppose you could, you could provide every option people want. Scala test is trying to do that and it keeps getting bigger and bigger. A lot of testing I've seen out there, probably in large companies, there's new testers for them or for the project manager, program manager people. So, they should be in English and, you know, they should be readable and they ought to be tied into these people that aren't programmers. I say nonsense. I'm a software developer. I should get the same kind of leverage out of testing. I get out of writing code and I want a tool for programmers, not for managers. And finally, it seems sort of odd where in Scala, really, the great stuff going on is all about functional programming and immutable data, yet the current testing systems are nothing like that. They're really your classic, you know, you could have written them in Java. There's nothing very functional or immutable about existing systems. So, can we do something in terms of addressing those issues? Here's some of the existing Scala ones. I've still seen people use JUnit. I think some of the light-band people told me they actually continue to use JUnit. That was fairly recently. And again, the two big ones, Scala Test and Specs2. I mean, I'm absolutely appalled. 400,000 lines of code for a testing library in Scala Test. Specs2 is a little better. There's something called MicroTest, which is sort of cute. It was designed by somebody who was interested in the Scala JS version and discovered they were having trouble porting Scala Test. How many people here view Scala Check? Scala Check is awesome. I mean, it's the one up on those. I'll come back and talk a little bit about it, but I think that's really one of the things that we should be looking at more carefully. So, you know, I think this just repeats, I think, the things I said earlier. I believe we're a small, clean, pure Scala, purely functional system. And the system's around a thousand lines as opposed to, you know, the previous numbers here, 400,000 lines. And it does, I think, everything those systems can do in a lot less space. And it supports Scala Check as a component within it. So, you know, usual thing, it's out on Maven Central. You can pull it down. And all you need is an import statement. Then you can immediately start writing your tests. So, let me pull up a, get some code up on the screen here. And this one. Is that big enough for everybody to see? Okay. So, I'm just going to walk you through informally, and I'm going to come back in a minute and describe in more detail what's going on. But you shouldn't be terribly surprised. I've got a little, the tests take the form of assertions is this assertion true or are these two things equal? Assertions can be grouped into tests. So, a test, the number of tests that work gets reported at the end. And there's also labels that can be used for higher level grouping. And, you know, you shouldn't find any of that terribly low. What have I done? Well, there it is. Screen. I'm in, I'm in SBT here. I'm in an SBT prompt. I'm just going to run that example I just showed you. You get a, sort of a feel for, for what it looks like. So, you can see the first, the labels print out and the tests. And so, I had one test with one assertion that worked. And one test with two assertions, one of which failed. And at report there were two tests, one of which failed. Fairly, fairly vanilla so far. I want to come back and talk about what's really under the hood a little bit more. And then we'll come back to that example, and explain it. In terms of some of the data structures involved. So, where are we? Back here. Hello. Don't make a computer. Oh. Oh, what's happened to me? Well, there's the slides. Here we go. So, concepts. I assume everybody here understands the difference between imperative and functional programming. Focus here. You know, the existing libraries are all imperative. What happens is there's some kind of big state data structure inside of things like a scholar test. And every time you execute one of those lines in the program, it makes some change in that mutable state associated with it. Functionally, there is no mutable state. What happens is there is a mutable data structure. And every time something occurs, rather than changing that, I create a new mutable thing which is derived by some functional transformation of the thing. So, before an assertion, I have a value. After the assertion, I've got another mutable value, and that value has the original one plus it captures whether that test succeeded. So, data structures. The first data structure is a thing called Lambda State. This is that state I was talking about. It's a mutable thing that captures all the information about a particular point during the computation. Transformations are the next one. They take one state to another state. So, an assertion is a transformation that takes a previous state to a new one. Actions are nothing but sequences of transformations, and they compose in the obvious sequential way. States have two subcategories within them. One is reporters. Reporters are ways of keeping track of the results in the output of the tests. And there's also options that control various things that can be done slightly differently at various points in the computation. So, you know, classic kind of diagram here of a sequence of functional transformations. The bottom one, you think about some initial state. Each test thing does some transformation. If I was being a purely functional guy, I would hold off until the end, and I'd take that state, S5, and I'd produce some output based on that. I'm an impatient person. I don't want to wait until the end of the test to see the output. So, I cheat a little bit here, and most of the time I actually incrementally output the results as the test runs. So, that's what's one slight modification on a purely functional thing. Although I could run it. I could run it in the, in that bottom model. I usually run it in the top model. So, there's a number of actions. Remember, actions are state transformations. We saw a couple of them. There's a assert, assert EQ equality, which I showed you on the previous one. Assert EX is for testing whether an exception occurred. Assert SC is for scholar check assertions. You can check things using the full set of scholar check capabilities. Those are primitive actions. There's also compound actions. A compound action is something that contains other actions. Remember, tests were a compound of assertions inside of them, and labels included other things. So, these are the compound ones. We'll see some others as we go forward. Again, that's just a picture showing a primitive action versus a compound action. Let's go back and look at that example again, if I can get it back. Okay. So, act here is, so I need to extend lambda test, which came in from the import up here. Act itself, it, I don't know if you can see this, it says that's a type lambda act. So, that's going to be a transformation, and that transformation is going to be what gets run. In this case, I'll come back and explain this a little bit more in a minute, but I'm actually running these as a top level app. If you notice an SBTO is actually doing a run main against the test library. And so, oh, and this run is a command that's coming from the lambda test coming in the run. So, what happening is each of these things is a transformation. The assert is a transformation. The label here is a transformation. The test is a transformation. So, each of those things are either one of those simple or compound things. So, they're basically is a very, very simple data structure. And I lost my mind trying to work my way through Scala test and decrypt the number of different libraries and classes and traits there. It was just horrendous. Here, I've showed you basically all the main mechanisms in the library. The only thing that I haven't explained is the plus operator here, which is a composition of two actions to produce a new action. So, in the fix plus, I think, any questions on this one here? It's pretty straightforward. It's almost embarrassingly simple. You know, let's see. Get out of here. Back to the slides. Is this object in the delivery for every test? Is what? So, the object extends out to the delivery for every test? Let me come back to that question. I've got a section talking about running things, which has got more details than I want to need to expand some things a little bit to talk about that. I guess I can say I can run it either with that object or I can use the SBT test commands as well. And I don't need the object for those. That's sort of the short form of it. Okay. I'm going to talk my way through some of these different features in the language. Each of them has an associated piece of code with it, which you can go look at. Stop me if one of them particularly interests you. Otherwise, I'm going to just, on most of them, just going to talk my way through what the features are and might stop and look at code in some of them. So, there is this assert EX, which is used to assert that a particular piece of code is expected to have an exception. And there's some fancy things like you can look inside the exception to assert what kind of exception it was. The system also has a lot of internal logic for unexpected exceptions that occur. And so it'll capture those and do the appropriate thing. It tends to firewall them off at the test level. Mutable and immutable. The system itself is purely functional and immutable data structures. Of course, that doesn't mean I can't test conventional imperative code. And all this is saying is I can do code that also has variables and changes. And they work just fine in the system as well, even though the system itself is immutable and functional. I'm going to skip that slide. Of those of you who've used Scholar Test and Specs 2, have you seen before all and after all? I hate before all and after all. As a functional programming, let me tell you why I hate those, is because they are separate. If I need to communicate between the two of them, I have to put a global variable in. My God, for functional programming, you don't want to have a global variable. And so that was one of the things that drove me to doing it. And so this introduces a purely functional, immutable kind of wrapper mechanism. And this one's probably worth bringing up. I'll show you the code on that one. A little more advanced example. There it is. So, here's that main object again. And here's the actual test. And I've got an auxiliary definition I'll show you in a minute. But the idea here is this thing, fileWrap is a wrapper. This is another compound action. It's a little bit more sophisticated action. What is happening is it's going to open a particular file. And if we look at F here, which is injecting, F is going to be type Java buffer greeter. So, what's happening in the wrapper behind the thing is the file open is happening at the beginning and the file close is happening at the end. It's not only happening at the end, it's happening even if there's an exception. The after all thing gets really screwed up if there's an exception in the middle. So, I'm going to call F read line, F read line. I've got a little file here that's got one line that says foo and another that says bar. And then I run another test, same file. Again, this is going to open it again. So, the first line zap is not going to be there. It's going to be the file, it's going to be actually foo. So, that one's going to fail. So, let's run that. Oops. And then I'm going to come back and show you the auxiliary code on that. And you notice that it does what I said. The first line was foo and the second line is bar. And then when I opened it for the second test, it said, ah, I was looking for zap, but the actual line it found was foo. So, let me go back and show you the auxiliary code. This is one example of some extensibility in the system. We'll talk a little bit more about, more generally, downstream. But here is the, here is the file wrap thing. There's that file name parameter. It's compound. And the compound that F coming in is going to be a buffered reader. And it's expected if it's fed a buffered reader to return one of these transformations, one of these actions. And the thing itself is an action. So, this is exactly one of those compound things I'm talking about in the form of a wrapper. And there's a little boilerplate here. T is the, that's the state. Remember, a transform as a state goes to state. So, that's the initial state. And what happens here is basically there's this magic thing called a val that takes the lambda action thing and runs it in the context of a wrapper. So, that's a little more complicated, but not a lot. And I'll come back and explain how this fits into the overall architecture a little later. This is one specific example. This is actually part of a more general pattern. And of course, open to questions as I go forward. Back to the talk. Don't talk the real. Yeah. So, I'm basically thinking in terms of the, what is it called, like when you need to open databases and stuff, I'm blanking on what they're called. And then you need to do the test on them so you need to reset databases all the time. Yeah, so that's kind of your method of... Yeah, this database is another example. Another one, if you're an ACA user, you've got to set up the ACA context at the beginning of the world. If you're Spark, there's Spark context. So, there's a whole bunch of things that basically have this open something at the beginning, close it, and this wrapper pattern works just dandy for all of those. Scholar check. Scholar check is really awesome. If you haven't seen it, it's basically property testing. And so, what you say is I'd like to describe a collection of, non-procedurally describe a collection of different values that are going to go in my test system and use it to automatically generate all the tests. And it really serves a somewhat different thing. There's, oh, there's a, my colleague from 47 Degrees, Noel Markham in London. If you haven't seen it, it's got this really nice set of slides on Scholar Check. If you haven't seen it before, this is I think one of the better introductions. There's a bunch of stuff out there on it. But you basically, you've got this set of values and I can say for all, for all these things, this property is expected to occur or exist, there's going to be some value for which this property holds. So, the thing to do is let's go look at some actual code. I think it makes it a little more concrete. If you haven't seen it before also, we'll see how it fits into this system. And it's a little bigger. So, here's your, here's your basic assertion. This is an assert SC. This is a Lambda test thing. Inside its body is a Scholar Check code. So, it's actually a Scholar Check code. And this says, this is sort of a classic example. If I, if I concatenate A with B, it's going to start with A. And I didn't specify the data set. Scholar Check has a set of defaults. I think it's a couple hundred or something. It runs by default. We'll see when we run it. This is one that's going to fail. If, if, if I concatenate two things, it's going to be greater than the A length and it's going to be greater than the B length. That obviously fails if A or B is the empty string. So, we'll see that. There's going to be some test data for which that fails. And down here, we see another thing where I'm specifying actually the data set. I want to run it from the choose. Again, this, all the stuff inside the assert SC is a Scholar Check code. Oh, and Scholar, Scholar Check has some parameters. So, I can slide that over just a little bit. So, you can set the various parameters in here as well if you've seen Scholar Check before. So, integrates very nicely. I think it, you know, I mean, they really serve complementary roles. So, let's run that just to see what it looks like. And that was Scholar Check. So, you'll see the first one. The follow-up pairs of 100 tests. It ran all 100 tests and succeeded. The first one failed immediately when it tried the empty string. So, I mean, oftentimes it's not the first one. Some thing way down in the world. And it, what? In Scholar Check, they're shrinking. Yeah, yeah, that's all there. All the stuff about, all the clever logistical things inside a, this is Scholar Check, okay? Just a little side note. One of the things I don't like about libraries is when you pull a library and it pulls 10,000 other JAR files in. Scholar Check does that. This library has basically one dependency. Scholar Check is the only real dependency in terms of external things in this library. I felt that one was really, really one everybody out of you at Scholar Check. So, over here. Options. There's a bunch of different options that you can set. And again, because options are part of the state, I can change the options as I go forward. More interestingly, if I have a compound, remember the compound. So I've got a set of options coming in. I can inject a different set of options down into its children and then go back to the originals of very common things. So if I have a particular set of tests, I want to turn on slightly different options for, I can do that. And tags. Most of these libraries have tags. You know, you talk about unit tests and integration tests or tests to run on Tuesdays or tests I want to run before release. And so the system has a set of tagging mechanisms that control which ones get run. And it's very much like everything else. It turns out the tags are actually managed by include, exclude sets which are part of the options data structure in terms of setting it. So you have the usual ability to set tags up. And again, there's some sample code running. Remember I said there's two ways of running it. One is directly. And the other is via SBT. It turns out SBT has this rather arcane API for attaching a test code. I believe it was done by our friends in Scala Testland. And so this system also implements that. But if I'm reporting via SBT, I've got to use a different output mechanism than if I just run it directly when it just writes the standard out. So there actually are two different built-in reporters. And if I'm running it with SBT, it uses a special SBT, the one that talks to the SBT API for integration. Reporters. I talked about deferring or not deferring things. There is one that holds it. And the reason for holding it, there's two different modes. And I don't know if I talked to them about it. I'll mention the two just because they're interesting features as well. One is the system supports, I can explicitly ask the children of a particular compound action to be run in parallel. So what happens is it actually uses Scala Futures. And so it will run each of those children. Now, if I just run it the standard out, the output from all those children is going to get intermixed. So what happens is the behavior, it outputs the first one immediately and then waits for all the others to complete and then outputs them all in the order that they were written originally. So it uses that whole reporter to temporarily. And so that information gets stored in the state, the reporter state, which again is part of the overall testing state. The other one is an interesting one. I don't know if you guys have ever run really huge sets of tests and there are regression tests for a really big system and you get page after page after page of output and I have to look for was there a failing test which was on page 23 or something. There's this cool option called only a fail and it only outputs the failing things but it's part of the test. So it actually runs the entire test to see if there are any failures and if there were failures it outputs that whole test block and so it uses the whole reporter until that test is done and then it decides whether to output it at that point. So that's how that's being used. I mentioned parallel execution. I think I said everything there. Extending. This is one of the key things is can you take a core library and really extend it into radically different domains? You know, I got this very simple thousand lines of functional immutable Scala code. Can I do some really fancy things with extending on that API? And I've written... Oops. Let me come back to where I'm going. There's two different ways I can do it. One is a programmable directly in place. The wrapper, that file, open and close file wrapper was an example of programmability in place. The other is I can actually build a whole new library if it's something that gets used. There's a couple libraries. So let's look at generate. Generate, I'm trying to get the idea that tests are not a cert one, a cert two, a cert 500, a cert 700. You know, this big, long, flat thing. It's a programmable thing. We're programmers and we really ought to be able to do that. So let's look at this generate one. It's a little bit trivial, but it sort of hopefully gives you the flavor of what I have in mind in terms of generation. So... Go here. Go there. Go to generate. Again, it's a little bit of a hokey example, but you got the idea. So I just defined an immutable list up here for the sake of this example. So what I'm going to do... Ah, real programming. Zip with index. So I'm going to take that zip with index and then a test and an assertion. There's a label and then there's a zip with index in the middle here. It all works out just fine as long as the types match up to the appropriate state and action types. And what I'm going to assert here is the value of each element is equal to its index in the thing. So if you look up here, the first and the last element are going to satisfy the test. The others are going to fail. And here's a... This is sort of poor man's scholar check down here on the bottom. And what I'm going to do is I'm going to generate a random number 10 times and assert it's even. That's obviously not repeatable observed as well. So let's run that and you just see what the output looks like. And this one's called generate. Okay, so you see zero worked, one failed, two failed, three failed. If I turned on the only a fail mode what would have happened is the first and the last one would have vanished in the output. So that's what that option would have done. I didn't turn it on here. And you'll notice here it found some things. If I run it again I'm going to probably see a different set of numbers there on the output. You see I got a different set of outputs from the random number generator. The idea basically is testing is not this domain in and of itself where you can't juggle anything. It's just another kind of programming and another set of data types that you can integrate. I think particularly in my mind if I'm building a large system with unique properties oftentimes I'm going to want to build a sort of test jig just for that system that recognizes some of the interesting properties of what I'm testing in the system. So I'll define my own custom testing library in the system and then have those custom tests tailored to the particular project I'm on. If I'm just writing a little piece of code probably I just use the stuff that's built in but if I have a big project and I have a lot of customization I want to do being able to build those is probably a good thing. So you're putting out this even there. Is that a custom assert? Let's go back a little bit. Let's go get the coding in for a second. One of the things I found frustrating about printing out they have built-in asserts that have useful but you can write your own thing. Well you'll see the first comment is you'll notice it's an interpolated string here. It was right here is where that line came from. On the other hand the more general question is sometimes I want an assertion that isn't like either of these either the assert or the assert EQ it's a trivial program those as well that would be a... I think the assert and the assert EQ are ones that you see all over the place and so those are sort of the standard ones that come out of the box but you can build custom ones to your heart's content as well. Answer your question? Here's sort of a little bit more of a picture of the architecture. There really are three layers. There is the core part of the system here that's got the state and the actions and there's all the really heavy-duty difficult to write stuff here so it handles parallelism and the reporters and all kinds of other weird stuff. There is, down here is the user code I write and so this is the typically user... there's this intermediate layer which are things like the assertions and the wrappers and things some of which are built in but I was just mentioning the fact that I wanted a custom assertion what I would do is I'd build it here there is a lower level APIs here we saw Val, remember Val was in the wrapper code if you remember back there there's this magic of Val that knows how to run it knows how to execute a lambda act and the transform and there's a whole bunch of primitives for the state so this is the stuff that lets me put more surface stuff on it and tailor it, there's actually a lot of freedom there and so there's a set of ones that are built in just because in my mind at least they were the ones that were sort of the more obvious ones I want to use day-to-day and common test but it's easy to build new ones so with that said I think that just said all that I'm talking about more information and we can I think we're running good on time I can come back and we can go back to earlier examples if you want so think about what you might like to talk about let me just do this here real fast there are extensions these are three extensions I've written these are all open source projects out there the first one is a timing library it you can time how long it takes to run a piece of code or you can run that piece of code multiple times and get the the mean and standard deviation on timings and things like that so it's a if you need to do timings it just you pull that library in and it just fits in as a new set of assertion time things that there's also the ability to write a test that says this piece of code has to satisfy this particular timing constraint so if you've got a time-critical code you can write an assertion saying this code has to run that fast, yes just a question on the timing we've used a scholar meter in the past that does a bit of warm-up before it does the timing yes there's an option for warm-up on this as well, yes one of the libraries I've written earlier in this was a JSON library and we did incredible amounts of extensive timing testing on this JSON library and a whole bunch of other scholar JSON libraries and it's a bear trying to get repeatable test results on a JVM the warm-up helps but it's definitely not sufficient I think there's a real black art in terms of yeah, again, warm-ups are good but if you're really serious about timing it's a hard problem this gives you at least the simple form the second one is could I do something with actors? there is an actor test kit that comes as part of ACA that I've never been terribly fond of and so I built my own custom extension to lambda test here that can be used for testing actors so in effect you know you can send messages and have expected messages it's obviously asynchronous so you know you send a message and then there's going to be some period of delay and you can set timeouts and things on those as well so you can basically do unit tests on actors the other one is all the output I showed you so far goes sort of the standard out I thought it would be interesting if I wanted to build this as part of a more automated system maybe I want to capture the output of the test and machine readable form and so what I did is I built another writer instead of a standard out one it captures the data and all the output that was standard out goes into a JSON file that you can then parse very nicely and take a look at the output with post processing code so those are just examples those three that I thought were cool to write but it's easy to write these I guess the the ACCA one was sort of non-trivial but it's because ACCA is non-trivial the other two were relatively simple to write and the ACCA one was non-trivial not because of lambda test but because of the complexity of using using ACCA how many people here have used ACCA most of you guys okay I think it's really important I think a really sad trend in the industry today is the lack of documentation or the crappiness of what gets produced and I really as a developer think it's really bad one of the advantages of this kind of a system being small is the documentation has to be small think about documenting uh... scolatest I mean they've got a huge documentation keep it up to date so there's a there's a bunch of documentation I'll go over and show you some of that in a minute but there's a set of there's an online users manual there's Scoladoc for all the APIs I showed you some of the sample code I tried to write short samples of most of the major features I've written a number of blog posts on the 47 degrees blog and plus there's the extensions I talked about see where am I on the slides yeah let's take a break and I'm just going to I'm going to go show you the the github so here's the github site and uh... most of the documentation is off in other places so um... the Scoladoc and sample code are obvious um... are these sort of nice little click there we go um... so this is a a little blog post that talks this is a sort of a repeat of some of the philosophy discussion I gave you earlier in a blog post that talks about the philosophy of building systems this way so that's an example um... also it uses this 47 degrees got this really cool documentation tool um... and this is what it looks like you got a nav bar here and uh... you know we can go through so um... comparison there's more detailed information on the different on the different libraries compare for example and uh... there's a quick start um... concepts we talked about concepts a little bit actions so these are the built-in actions that we saw there's a number of those you didn't see all of them there's like one to change options um... information on how to run it um... of course you know it's an open source project uh... anybody's willing is uh... I'm certainly willing to take um... contributions with one rule the one rule is I intentionally made this library small if you can't argue that you can't do it with an extension you have to put it in the core I think it's a very different philosophy than a lot of systems where you know we're getting this bloatware dumping of everybody's crazy ideas in an open source project I think it's a really bad trend there's a little piece of sanity in a world that doesn't necessarily support that point of view so anyway you get the idea and of course there's the usual drill myself down the library so okay here's lambda state that's that internal API um... I mentioned that you build this is the lower layer that things like assertions and stuff are built on top of so these are the uh... some of the internal APIs if you want to add extensions you'd work in terms of these APIs so they all tend to be fairly thoroughly docking and of course the code's all there as well it is an open source project with everything you need let me go back to slides um... a couple other things I wanted to mention I've done a number of different open source projects I just thought I'd mention and pass it here if you want to go look at them um... the world this is what I did a few years ago was a really fast JSON parser it is not functional and uh... there's been a number of great JSON libraries appear since then I happen to like this library because it's mine but you know it doesn't stand out as much as it did um... I hate the logging support in Scala everybody uses the Java loggers which are absolutely hideous um... this is a new logging library which I think is actually pretty good it's pure Scala no Java code uh... and it supports um... integration with uh... it's got its own logging API uh... it's it integrates with the Aka logger it integrates with all the old Java code so it uh... it's clean it itself is built using an Aka actor in the middle so if you're building systems with logging that's definitely one worth doing um... you may know about physical units just go over there for a second if I can it introduces type checking a little bigger where you actually statically type check your physical units meters centimeters um... number of people built libraries actually I think I actually did many years ago I actually did one of the first specs on this first implementations this kind of library way pre Scala so I felt I had to do it the Scala but again there's a number of other people who've done libraries out there it's sort of cool you know if you're working in those kind of physical units I think you know if I was in that I don't do stuff in that space but the example sort of cool from a development point of view and I did this other thing which uh... was sort of interesting how many people we have spark people here you know data sets they had data frames which were sort of sequelish and they introduced data sets in an attempt to get more static type checking and it turns out if you look at them they actually don't fully statically type check and there's some interesting deep reasons for this now there are significant weaknesses in the Scala type system Odersky is in the middle of hopefully doing a dotty this new version of Scala which is going to hopefully fix some of it I'm not entirely convinced but so I wanted to say could I actually build a library that was fully statically type checked you want to check two things you want to check the types are correct the other thing is if I've got fields uh... spark you know I've got you know subfields of a case class it doesn't check the existence of the fields it checks them at execution time so I built this library to do it and it was an interesting experiment because I used Scala macros I learned far more about Scala macros than I ever when I'm pleased to hear the Scala macros are now deprecated and going away forever what a horrible terrible design so the biggest thing I learned out of that is if you can all avoid it don't use them the best way to use them is there are a number of libraries that are built on macros that present a friendlier view of the world shapeless is one that probably comes most immediately to mind unless you're a total masochist don't ever try writing anything with macros so anyway just on that so what do you think about the new Scala macros I have not looked extensively what I've seen of it is it's looking much cleaner than the macros stuff it's always tough when you just read the documentation which is all I've done on it to really get a feel one of the reasons I did this project is I really wanted to kick the tires on macros you learn a lot by actually writing code and I haven't written any code with Scala meta so I think that's the sort of test in my mind to really understand deeply you've got to write code and I haven't done it yet the work on Scala test was actually done at 47 degrees because little stickers over there I'm no longer with them although they're my friends and since they supported this project I wanted to give them a little plug they're an international company headquartered in Seattle where I'm at they have an office in London they've got a large development team in Spain which is sort of cool in Cadiz they've got a ton of open source projects you saw the Scala the Scala examples they've got a huge number of projects Scala exercises Scala exercises as well as a bunch of other open source it's really a cool company because the people there have really been major contributors to a lot of open source projects they've really been active in the community so they've got some cool stuff out there they partner with all the usual culprits more interestingly they're a member of the advisory board of the Scala center so they actually are very influential they actually built not so with a lot of recognition they built a lot of the software for the Scala center actually so if you ever go to the Scala center website and stuff the 47 degrees guys actually built it anyway I wanted to give them a little plug since they're my friends and since they helped support the project anyway that is we're and we're good we're just about right on target on time so I'll take questions is there a plugin for I don't I don't know you know the Scala center's got this great let me back up here pull the website up if you haven't seen it the right button just a second the badge the right badge the Scala dex have you guys seen Scala dex? this is something out of the Scala center and it's a index of lots of open source projects this is the page for mine but if I was looking for mocking libraries just try to experiment real quick there's an aca mock it's a GTP mock anyway there's a bunch of libraries I don't know very much about them but yeah there are some mocking things out there as well as I think you can probably use some of the Java mocking I wouldn't be surprised if people using like mochito and stuff like that in Scala so the question was is there a plugin for Lambda tests that supports any of those would you like to write an extension for that that would probably make my adoption yes again it's open to the world the other comment is I have strong opinions and one of them is I personally think mocking is a terrible idea so that's probably why I'm not going to build a library myself but I have no objections to other people doing stuff with mocking that's probably another half hour discussion about why I don't like mocking but you shouldn't need it as much in purely functional code although there are things like databases we need building real systems there's potentially a need for mocking data structures and those as well dependency injection I don't like using external frameworks I like just using traits and things inside a Scala and actually build if I need to do that and again I try to minimize it I try to do those entirely inside the Scala language because in the outside it's just you can do it in Scala you should stay there what's the talk about free monads about free monads oh god everybody there is so much incredible stuff going on there's an interesting project 47 degrees of doing called freestyle which if you haven't seen it it's worth checking out there's a big community of people participating around basically free monads and some of the stuff around that my reaction is twofold one is the use of terminology and I know it and you don't know it attitude has really been a discouragement to adoption in the community and I think that's been unfortunate in scaring Java programmers and other people away from picking Scala up on the other hand forget the terminology and look at the idea of monads and the composition and the abstraction stuff there there's phenomenally great stuff going on in functional programming and so I'm a big fan I'm just concerned as a practicing engineer and somebody who teaches I'm always concerned about how do you bring more people into the community so I mean I went off on it I've been doing a lot of teaching and one of the goals of teaching is I wanted to enlarge I mean it wasn't because I wanted to teach it was because I wanted to enlarge the Scala community and so how do you bring more people in so there was a like I've been teaching at the University of Washington we do like a whole three hour lecture on monads for people who've never seen monads before and how do you start doing it in a Scala context but yeah it's good stuff what about using your program like what I had that's an abstraction so a lot of runtime times goes runtime times it takes for whatever things you want to do the distinction between isn't this beautiful mathematical and does it run poorly yeah it's an issue I love the purity on the other hand I tend to be a practical engineer if I'm working on systems real systems I tend to shy away from it some of it is the performance concern and some of it is in most companies I'm a little bit concerned I can't necessarily get everybody on the team fully up to speed and so if there's really a compelling reason for a particular set of functional abstractions or if you have a company there are some companies that hire nothing but really top in functional and that's cool if you're doing that but if you've got your typical company you may have some people that know some Scala some of which may even be good functional programmers but across the board the team can write Scala code they aren't necessarily have their heads fully around all the functional stuff I thought it would change over time but at least right now at least what I see in Seattle where I'm at the people still aren't ready to go full bore and again the performance scares managers for sure and on the cases where I've done it sometimes it's very hard a priori to predict performance effect given a lot of the optimizations that go on so if I'm going to do it and it's a really serious project I tend to do small test examples both ways and then time them out and see if the overhead is within if the benefit of going to the better abstraction is justified in maybe some slight penalty in overhead so yeah so basically there's a library for property based Spark testing which I once again forgotten that's basically Lambda test also have integrations that way for Spark? So is there any property based testing in the Spark context? I hadn't thought about it, well a scholar check would work on Spark, right? There's no reason why a scholar check wouldn't work on Spark Yes, but when I did it a while back like six months back I had to hack something it was a different library that I had to actually get working with a scalar check in order to get it to work on Spark and generate like the test data frames I haven't looked recently there were some version incompatibilities there were some problems in some of the Spark libraries compatibility with certain scholar libraries has been a problem and again I haven't been actively doing any Spark work for the last six months so I'm a little out of but yeah I don't see any reason why this wouldn't work and again I mentioned the wrapper technique and setting up things like SQL context and stuff like that would be a natural thing and I would think I hadn't thought about it until you asked the question now my first reaction without thinking very hard is the scholar check stuff should work for Spark testing as well if I thought about it and tried and I might find something I hadn't thought about but initial reactions it should work Yeah You mentioned how you were not a big fan of ADCAT test kits What's your thoughts on it? It feels a little clunky to me a little overly complicated and a little bit clunky I guess and I actually had a long online discussion with Jonas when I was working on the ACA extensions here about how to do things and I think he and I I have tremendous respect for Jonas he's a phenomenon but he and I had some slightly different philosophy and I'm not sure at the top of my head the main points on it but if you ACA test is not that big in this library from a user point of view it's not that big go look at both of them and judge for yourself I don't think it will take a huge amount of time to look at them both and compare them Everybody tired? Okay, thanks everybody for coming tonight