 My name is Keith and this is Tanya. We'd like to talk to you about a journey we've been on over the last year. Building a card generator. We wanted to focus on anybody who's learning Ellyxiaire. Hopefully we wanted to pull out some of the interesting things that we found as we were building this project that might be useful for you if you're building your project. Before we start going into building a car generator let's warm up a little bit. To get an idea what it is we are building, here is a cart with the question. Please shout out the answer if you know what it is. Anyone? Thank you sometime. You think that this the easiest cart I've picked up. This is going to return you? Next one? Anyone? Nothing. It's very interesting if you read it. Drop everyone. So it's dropping everything it has. Okay, empty list. Next one. Is it going to be true or false? True. Next one. Or a string. A couple more. Out of bounds error. This is something you just have to know. It's no way you could guess. Any idea what this might do? True or false? False. Okay, so this is the idea. This is what we wanted to build. Those are the actual physical cards. You could see them here. But we needed a way to produce them in a way that we could have a nice printed version very easily. Okay, so any journey starts with the first step. And ours started a long time ago. So this is Tanya back in 2015, our second meet-up. We started our own Elixir meet-up in Southampton in the UK. And as you can see, she wrote lots of little post-it notes. So as she was learning Elixir, reading the books, she was just writing these little post-it notes down and then collecting them together. And on one side would be the question. And on the other side was the answer. And I didn't have time for that. But then she'd test me. She'd show me these little questions like this. And then I started to realise that, oh, actually, those are pretty good. Those are really quite helpful. Cos I have a terrible memory. I don't know if anyone else has a terrible memory. But if you do, then these things actually really help. There is a little problem with those post-it notes, my handwriting. Whenever I show it to someone, they spend ages reading the card, and usually the answer comes faster than actually parsing whatever this is. So then Keith suggested that instead of writing all of this, it would be good to have a printed card, a printed version which is very easy to read. Okay, this is good. And this was the first point when we had to make a choice. And every journey, like a developer journey, and we see this as a journey from idea to a project, it starts somewhere like if you think if we are walking somewhere from the right to the left to like this house, you could see that to get from one point to another you could take any path to get to that point. And by the way, this is the map of St Augustine. So this is quite a useful slide for us if you want to go somewhere later to discover new things. Okay. Okay, so the first steps towards actually making this, taking it from a load of post-it notes to actual printed cards, Tanya didn't know anything about printing the print process or whatever, but she was very determined. So her first attempt at this was to produce using a package called Inkscape, which is a vector drawing application to literally create 18 cards on an A3 piece of paper and make each card manually. And I tried many times to convince her that this might not be the most efficient way to do it, but she was very determined. Yeah, until the point when I create this layout and I give it to printers and they're like, okay, this is looking fine, but instead of 5 millimeters gap, it should be 2 millimeters. Okay, fine, so I have to go back and change all of this and sooner it become a very, very painful process. So eventually I managed to convince her to stop crying and let's automate all the things. And really that's where the first idea of actually having a card generator was born. So our first steps towards doing that, we had Inkscape, it had the cards. So with a bit of Googling, I kind of thought, well, there must be some kind of mail merge thing that we could do. So Googling around, the first thing I found was, of course, there's a gem for that. It was pretty horrific though because you had to put some percent marks in your Inkscape file and then the Ruby gem, I wrote a bash script which would use the Ruby gem to then drive Inkscape. So it worked, but it was very slow and very painful. So I ended up deciding to rewrite it in Python. I just happened to be learning Python at the time. So rewrote it in Python and we realised that rather than generating a PDF, we could generate just pure HTML and then eventually we could, when we sent it to the printer, we could convert it to PDF from HTML. And that worked. That sped the whole process up. And from that process, we ended up deciding that if we put the CSV file that we had, we'd put it onto Google spreadsheets and then we could update that. We could end up with multiple packs of cards. So we needed to download the spreadsheet. We then needed to convert that to HTML and we could then send that off to the printers. So we kind of developed our process. So here the original setup was taking a spreadsheet using Ruby and Inkscape to convert a PDF, but that was hugely slow. So the next stage was improving it and removing some of the technologies, taking the CSV, using Python and creating HTML. But you may have noticed that this is an Elixir conference and here we are. Here we are trying to learn Elixir and using everything but Elixir to build this silly thing. But what we did find, this process helped us to identify what our application might need to achieve. So we worked out that generating the CSV, that was useful, that was working. And putting it into Google spreadsheets meant that we didn't need to worry about building some UI for editing the cards and stuff. So that helped. Generating the HTML was working because generating PDFs is actually quite slow. So we could generate HTML once we were happy with everything, we could then convert it to a PDF before we sent it off to the printers. And then obviously, as I've mentioned, using everything but Elixir wasn't quite what we wanted to do. So we set about building it in Elixir. And it was important to have a way to test the code snippets. There is a potential that when you type code, like we keep it in the Google spreadsheet at the time, that you made a mistake and then you send it to printers and then you find out that whatever you code is going to say is going to return. It's not actually returning that. So we needed a way to test this. So things we wanted to have and we wanted to make sure that Elixir is going to help us with is we wanted to test cards, we wanted to be fast, we wanted to be stable, reliable and we wanted to be scalable. So Elixir, this is the beginning of my favorite chapter. With Elixir we get a test with a little bit of metaprogramming, a dynamically generated test. It's fast, we use concurrency, it's reliable if we use supervision and it's scalable. We don't do anything with writing a scalable application just yet, but we want it for the future. So it's good to have it there. Okay, let's get started. So knowing that those are the steps, we need to download. At the moment all of the data lives in the Google spreadsheet. Okay, so the first step is to download it. Other one is to test and another one is convert or compile. You see that it's in pencil because we keep changing the name. We still don't know what is the best name for it. So we call it compile or convert. Okay, so let's take a look at the downloader module. So really all we needed to do was download a file from a URL from Google spreadsheet and then save that to a location on the file system. So what could go wrong with this? Well, it doesn't seem that complicated, does it? So what we're doing, we're using HTT Potion to get the URL and then we're going to get the body out of that. But if we're given an invalid URL, then that get, that map.get is just going to return nil and we're going to end up saving nil to a file, which is no use to anyone. So what we can do, we can pattern match on the result of the get. So we'll add that to a case statement and then we'll pattern match on the response. So if we get a response, we'll then save that contents to the file. If we get an error, then we'll just pluck the error out and we'll return that to our caller. That seems to make sense. So okay, we'll look at the save. So if we're going to save a file, we're going to be given some content. We're going to open the file path. We'll open a file handler. We'll write the contents of the, the contents that we're given. We'll write that to the file and then we'll close the file handler. Seems pretty simple. So what could go wrong here? Okay, well if we look in IEX at file.open, we can see that the file could not be opened. So then in that instance, we'll get a tuple back with error and a reason. So okay, here you'll see when we open the file, we're not, we're not going to, we're not pattern matching for that case. So we're actually going to end up crashing our caller because we're, we're the match error. So we'll get this nice match error, which you know, we want to try and avoid that. The Erlang philosophy of let it crash doesn't mean just don't handle errors. So it would be nice if we didn't crash on there. So if we handle that, so now we're going to, in our save file, we're going to open it and we'll check and we'll pattern match on the result. If we get a file, then we're going to go ahead and write to, write the contents of the file and then close it. If we get an error, then we'll pattern match the error and then we'll respond with that. What about the, oops sorry, what about that being right? So is that going to error? Are we going to get a problem there? Well, if we have a look in IEX this time, it doesn't actually tell us what the response is. So if anyone's looking for contributions to IEX here, that might be a good spot. If you look at the source code for bin right though, we'll see that, if we look at the spec here, I'm sorry if you can't see that in the back, it will return okay or an error and a term. So we know that it's possible it could error. So let's pattern match on that as well. Okay, so now we're opening the file, if we're okay, we're then pattern matching on whether we can write to the file. If that's okay, then we close the file. Otherwise, we return the error. That's great, but there's a problem here. Can anyone spot it? So in our error case here, we're not actually closing the file. So let's fix that. So now if we've now got file closed on both sides. So that's great. So oops sorry, we'll just check. So what about this file closed? Do we think that might cause an error as well? Surely not, but we'll have a look. If you look in IEX, it mostly returns okay except for some severe errors, such as out of memory. Okay, we'll check the spec for that as well. Okay, so yes, it definitely could return with an error. To be honest, if we get this kind of error, we've probably got bigger problems to deal with. So today, right now, we're going to stop there. So we'll go back to our beautiful little monster here, this save function. So if you look around you, you can probably see the seasoned Elixir developers. They'll be fidgeting in their seats looking at this code, feeling quite uncomfortable. But before we dive in and refactor it though, let's consider what the spec of this function might look like. Let's understand what the return types might be. So if we look here, we're returning from the file closed, we're returning an okay or a tuple error. If we look here, just below the reason, we're only returning the result of the reason. We're not returning a tuple. And then at the bottom there, again, we're plucking out the reason. So if we look here, our spec right now is going to look like okay or term. It's more idiomatic for Elixir to return either okay or an error, a tuple with the error atom in there so that you can pattern match. So if we update our function to respond in that way, then here we're now matching the whole error and just returning that in both cases. And then from our callers perspective, they can now pattern match on the error tuple and if they want to do something with it. But maybe we could even take this principle and pass our error even further back up the chain. So then our download now looks like this, where we're basically returning the result of the save, which could be an error tuple or the error from the HTTP potion. Okay, so we'll go back to our save function. I haven't forgotten that we've got that little mess there. So the issue with this is where is the happy case? So if you look here, we're opening the file successfully. Here we're successfully writing to the file and then finally we're closing the file. So this code doesn't make me happy. It's very hard to discern from all of the noise the complexity of handling errors is difficult to discern the intent of the code. So perhaps there's a better way. Use the force Luke. So the kernel.with special form is designed for this kind of situation. So we can rewrite our code using the with. So if we rewrite it using with, we'll start our function with with okay and we're back to having the explicit calls that we started with. File.open, file. bin.io.binwrite and file.close. If each one of those returns successfully, we'll then, we'll drop into the do and that's where our happy case will return. If for any reason one of those calls responds with an error, then the error will pop out the bottom. So it's now quite clear when you look at that function what the intent of the code is. Now the thing with this is there's more than one way to skin a cat. So for completeness, I'd like to demonstrate some of the other ways that you could achieve this. So the first way is using within your file open, you could use the try after statements. So around the bin right, we put the try after block. So the result of the try will be, or the result of the bin right will be what is returned after we'll always be run. So we know that we'll always close our file. Now you could use the file open, Arity 3 function, which takes a function. So the only issue with this approach is that when you call your bin right and then do file close, the only thing that's returned is the result of the file close, not the result of the bin right. If, for any reason, our attempt to write the file fails, we don't get the result. We don't get that back to our caller. So we can of course add a try after block to that so that we ensure that we're actually returning the result of the success of the right back to the caller. So any of you that maybe have some knowledge of Haskell or other strongly typed languages might be sitting there saying, use a monad. Of course, very possible. So if we create ourselves a macro here, this is the equivalent of the bind operator in Haskell, which will allow us to combine operations together. So if you look inside the unquote here, we're going to pattern match on the result of left function and if that returns us with an okay and a value, we'll then pass the value, the X through to the function as the first argument to the function of the right. If for any reason that call fails, it will then drop into the error case and just pass back the expression. So now our save function using that macro would look like this. We open a file, we write a file, we close a file. So the thing here that we need to remember is that we need to lift these functions into an either touch so that they do respond with a tuple for okay and a value or an error and and a value. So for example, the open file we need to, the open file, we know file.open returns with an okay and a file, so that one's good but the right file we need to take the first argument which is file and then return with okay and file so that the following statements can work and then close file or do the same thing there. So they all need to adhere to responding with either okay and term or error and term. So for completeness, this is the function that you might want to add to your module and there are libraries that can help you with this. Libraries that are attempting to address this kind of scenario so a good one is written by a guy called crowdhaler which is called okay, another one by Rob Brown, Monad X and there are many more. If you're interested in this kind of approach of dealing with errors I can recommend watching Scott's railway oriented programming, in fact any of the talks by Scott are really good he makes very entertaining talks. Anything by Brian Lonstof I can highly recommend he definitely knows what he's talking about and generally learning about Monad's funct as applicatives is actually quite interesting and fun and dealing with handling errors in this way. Listening to Lambda Cast and Magic Read Along and Functional Geekery will help you to start understanding these things and of course listen to Elixir Fountain but I assume you already are. So now with all that in place we should just be able to run from mix we should be able to run download pack one and we should be able to get our pack one downloaded. Okay, so in order to run this function you have to launch the IAX and then you need to run your downloaded or download and when you quit the IAX and then you launch it again then you wouldn't have your history saved and this could be a little bit annoying to retype it so this is the way you could remember the history and this is just a side tape. Okay, so running it from IAX is good but it would be nicer to just run it from the command line and mix task is what doing that so here we are going to create a simple mix task which is going to help us with this. How many of you have written mix task? Okay. Okay, so once people who haven't written once you see how easy it is I think you write mix task all the time because this is what I get to do. Okay, so first you need to think how you want to run your mix task so this is going to download card so I want to run it as mix cards download and then I want to pass parameters which is going to be a language and the pack I want to download. Okay, great. Let's create this. What we need to do is to use mix task. If you go to documentation it's have a good walkthrough how to do this. It's just a few lines so basically this is all we need. This is the module which is using mix task and it needs to have only one function run and in example you could see that it's going to take any parameters which is good for a CACHO case or whoever is passing wrong parameters or no parameters or whatever it's good to have a helpful message to tell them what they need to be doing. Okay, so this is good and then you want to try to run your task once you compile it and if you run it you're going to have your helpful message saying how you should be running this task. Okay, this is good. You want to see your task which you created in a list of tasks. You do mix dash H and your task is not there. You're like, why is that? Why is it not there? Okay, it's not difficult to fix so all we need to do is to add a short doc with a nice message saying what the mix task is going to do and then you have to recompile it again and if you do mix dash H again you're going to see your task in a list of tasks. I'm not really good with following methods, if you see all of the other mix tasks they are starting with the uppercase and they don't have a dot at the end but after I've added this right there was two way to change it so okay this is a unique task here in the list of the tasks. Okay, so that is good but actually we want our mix task doing something useful so let's our task call our downloaded or downloaded our function. All it needs to do is just to say okay as I'm going to accept multiple arguments I need to have it in a list and then I'm going to call my function with those arguments. Okay this is all good but could you see that there is a potential problem here? What happens if whoever is running the task is going to pass some gibberish and then we are just going to call our downloaded or downloaded the arguments which are going to possibly cause some problems inside the downloader. So here is our responsibility to make sure that we are not allowing that case and that's very easy to do with the guard closes so here we are saying that okay if we are having a combination of this language in the pack then okay we are going to call the function and you see there is allowed and packed combination so it's not just separate arguments they are in a combination. Okay so this is good so if we are going to if we are going to run this task with the correct arguments we are going to call our function otherwise we are just going to straight into a catch all case and we are not breaking anything. And as the result we are going to find out that the actual card is downloaded and it says where you could find it. Okay this is good. The next big subject is okay how do we test the card? So I had this functionality of ability to test the cards for a long time and it has happened that on the hard day we had James there in the corner soldier in the way LED's and we asked him if he could help me to write this and he kindly agreed to do that. So here I'm going to show you what we did and what I carry on doing. So let's walk through the tester. What we have is the CSV file which looks like that it has a question and an answer and there are different types of questions and an answers. You may have a snippet which you could run and evaluate and test and there are like theory questions which you can't run so you need to somehow filter that. Okay let's think about how you may want to call this tester. So this is the way I thought it would be nice is that you say okay so if I provide this one glitch in the pack then I want to have all of the tests running. Great. I'm going to try to create this so this is just the beginning of this. We are going to just fetch a file knowing the language in the pack. Okay great and here we also could use our guard close for full control to make sure that we only accept arguments which are out. Okay the next thing is to filter out of all of the questions and answers. Only questions which we could create tests out of cold questions. So we are going to do that here if we look a little bit closer. So we are going to take all of the questions which are cold questions and all of the answers and we are going to have them in one chunk. Okay how do we do that? Let's have a get cold cards function which is going to do all this for us. Okay so we have a file. We stream it and then we read from that file there is a nice library Nimble.csv If you are doing anything with CSVs this is a good one to use. Okay and then we parse that stream and the next thing this may look a bit like strange why we do that. So if you think about it we have a file which contains 54 questions and answers and each role is basically our card number. So we have like a file which contains 54 cards and each card is going to have like number like ID, like 1, 2 or whatever and it will be useful if when we have the test failing not go through the whole file try to find if we do with index what we're going to have as the result we're going to have a tuple for each one which is going to contain a list with the question and answer and a line which is useful for us when we generate tests. Okay and out of this stream we need to get this question and an answer. Okay so let's do that. This is just an internal implementation detail because I know that answer starts with a hash rocket. I could use a power of pattern matching in a function head to get those answers and otherwise I'm just going to return an empty list. So here as the result I'm going to have questions and answers which are cold questions which I could run. Okay and this is just a function which is going to use a power of regex to take this content of those question and answers. Okay so this is our cold card so we filtered them. The next thing is once we filtered those cold cards we need to somehow create or dynamically generate a module which is going to contain all of the tests. Let's have a closer look at this. So here we have a question and an answer which are strings in this case and we need to take them out of this where it is the filtered block. We need to take a question and we need to somehow turn it into a code. We need to take an answer and also from a string turn it into a code and here we could use our line number as we have it as a table so we know which card is going to fail when we do have that. Okay so let's do that. How do we do that? First define a test module. Okay if you look at how you create a module you need to pass a module which is an atom then you need to pass a body which is whatever goes inside the module an environment. So you may see here so we have a module and we have a content where is body okay let's create a body if you think of how when you write in tests and you open a file and you define the module and then at the top you write in use xunit case and then you write in your tests so this is what we are doing here we are saying okay so we use xunit case and here we need to start writing our tests but we need the way to generate those tests. We are now writing it manually okay so we need to generate those cases. We have a content which is each element of this content is a tuple which is containing a line question and an answer and as I mentioned we need it to be a code and we have strings so turn strings to a code there is a method for that okay brilliant so we have a question and an answer which are quoted great so they are quoted then we need to have a test also in how the quote here when I think about quoting and unquoting it's a little bit like string interpolation where quote is like your string and whatever you need to you need to get the value out of variable you put it in this string interpolation syntax so here is a little bit similar you have quote and then you have unquote so you need to take value out of whatever is in your variable you can't just call it so here you need to unquote line unquote question and unquote answer and run the assertion okay so those are generated cases good we define the module dynamically this is great the next thing is to run tests okay so we have a function run I'm not really good with creating names for modules so I just call test it and this is quite simple we just call tests run and as the result I'll show you here so if we call this function test the test passing the language and the part if the test is failing we are going to have a card which is failing so if I then go to my data with csv I need to go and look at the row 6 I don't need to then try to find which one is failing which is quite nice the actual failure and the happy case when everything is good looks like this great okay if you managed to make it through that well done so we have now successfully downloaded the data for the cards we've tested the cards to make sure that the content is valid and that we're comfortable that the cards that we're about to create are correct so now we just need to take the contents that are in the csv and convert that to html so that we can then render a pdf ready to send to the printer so we started with this process thinking I can't be that hard we'll just use some redgex so we started writing regular expressions trying to do some pattern matching capturing and then replacing p tags and code tags and so on but it got very quickly got very difficult and it was very difficult to test especially once you start getting into recursive regular expressions I don't recommend that so we ended up changing it took a bit of pain before we realised that we were going in the wrong direction with this and we broke it apart into three pieces which is a tried and tested method breaking it into a tokeniser, a parser and a compiler and the advantage of doing that was that we broke this problem into small chunks so we could then test and I can't recommend that in whatever language you're ever using just write small things, it works so the tokeniser is basically going to take a string and chop that into a list of tokens the parser is going to convert that list of tokens into a tree and then the compiler is going to take that tree and turn it into html we could have different compilers for different output formats but at the end here is just the template that we're going to use so here we're taking in a string the tokeniser is going to take a string and then just return us a list of tuples the parser will take a list of tuples and turn that into an AST and it's a very simple AST we deliberately chose it as simple as possible and we can increase this complexity from there but we were able to test it in isolation which made it very easy to write and then very easy to test and then finally the compiler would take that tree and spit out some html and then of course Tanya loves mixed tasks I'm not going to explain you how to write second mixed tasks I'll just say that this is great to create mixed tasks for everything so we've now kind of recreated what we had in Python and then previously in Ruby they're just the pieces and what we actually did was do them as individual applications so they could then be they could have their own little test suite they could be run individually they could be if we chose they could be replaced if we chose to do it differently and the first one we decided well this is Alexia so we should be able to use concurrency to improve the performance so so how difficult it is so we used before we go and do something with concurrency we thought we want to see how fast we are going to do concurrently so we used this benchmark benchmark library and the setup is quite easy we have not concurrent and concurrent downloader and those functions are pretty much simple and quite similar to each other one so they both grabbing packs from 1 to 20 and downloading not concurrent do it sequentially and concurrent the only difference is that it's spawning this is the simplest way to do concurrency just spawn a process and call this function within the process and then we run this and we get the result we're like wow look at this faster we should do everything concurrently forever and then we thought okay let's see how it looks on the graph so if you see on the left is this concurrent you can't even see it and there is a sequence so we thought wow this is really good okay and with this we've done our three pieces we have something working we still have a lot to go and we have a lot of plans and ideas and we want to of course do everything concurrently from now on it will be good to have like at the moment we have like separate pieces like downloader test and we could run them with mixed tasks but it will be good to have some kind of pipeline when you go from one end to another it will also be good to have some kind of like state machine some process which is going to keep a state of of each park and it will be good to have some UI to manage things and keeping data in spreadsheet doesn't feel quite right and we want them to have to move it into a database and at the moment I mean we kind of made the decision that using the spreadsheet was working for us and it was good enough for us to grow but obviously what we would like to do is have our own UI by choosing to use Google spreadsheets meant we didn't have to go ahead and build some UI that we didn't have time to so we could focus on the important parts of actually creating a pack of cards that we could print now that we have got these packs of cards we can actually think we don't want to use Google spreadsheets anymore we want to create our own because we've built this as using OTP apps using an umbrella apps we can just add in another web app whether it's phoenix or another web layer and then again for different public UIs we're thinking we can introduce multiplayer games where people can play and that can all plug into the same system as individual little apps and that's really what the apps gives us our journey was fun Johnny caught us somewhere like in the beginning of our journey and said okay come on come and tell us so this is where we're at at the moment and to summarize what we talked about it's a lot of ways to handle errors and a lot of questions whether you should or shouldn't handle errors mixed tasks are easy and fun dynamically generating tests are interesting learning and it's very useful concurrency is when you use when you do things concurrently you gain a lot and starting small writing small functions and then keep growing and keep building up this is what worked for us and this is our journey in the picture and I don't think we have any time for questions so whoever is going to this theory you could ask all the questions there or at the lunch tomorrow today any time after now thank you