 Okay, so today, we're gonna talk about something fun. But first, I wanna say that I'm very honored to be here. I remember a couple of years ago when Impact's New York City happened and I wanted to be there at the time and seeing videos and feedback online about how awesome the conference was and how amazing the surroundings were. I was very jealous that I wasn't gonna be able to be there, so I am very honored to be here today at the first West Coast Impact's and they delivered, didn't they? I mean, this place is amazing. Everything around, like even the pianist in the background coming in is like, what the heck, we got live piano as we're walking into a conference, a tech conference. How cool is that? That's pretty cool. All right, so I'm Jeffrey Lussell. Today, we're gonna talk about Life is but a Stream. We're gonna dive a little bit into the Stream module and so what I'm hoping to do today for you guys and gals is if, well, basically to compare Enum to Stream and give you a couple of examples of when you might wanna switch out Enum for Stream and sometimes when you may not. So the title of the talk is Life is but a Stream and I apologize, I may murder this analogy over the course of the talk, so I apologize up front, but bear with me, okay? So in the background here, we got this nice-looking, nice-looking Stream, right? It looks really nice, you know? You kinda wanna spend some time on that bench over there. There's a little bit of water trickling down over a little, I guess you can call it a waterfall and then we've got another picture here, another Stream, kind of fall time. The water's a little bit rushier, you know? The water's getting a little crazier but still, you wouldn't mind spending some time either in the Stream or by the side walking by it. Then we've got this one, which I don't know where that is but I wanna be there, that's pretty amazing. It's a little rockier, so you may, you know, if you try to float down that thing, you may hurt yourself, but it's beautiful surroundings and then you can imagine that those little Streams may go from that first one to the second one to this one to this, right? This could be where your Stream ends up and you're towering over this, or you're falling over this towering waterfall into that huge, maybe like a two-foot thing down there, you're gonna die if you go over that thing, right? So this is where Stream and Enum can lead you. So if we take a look back at this Stream, I'm gonna come back to these pictures in a moment but keep those in mind as we talk about Stream and Enum and just the fact of how the water moves through those pictures, okay? All right, so the first thing is Enum is awesome. So how many of you in here have written at least one line of elixir code in your life? Okay, cool, so how many of you have written one thing, one line of elixir that contained a function from Enum, the Enum module? Okay, cool, so the Venn diagram of those two groups are basically the same, right? Everything you do in elixir, not everything, but almost everything you do in elixir at some point is gonna touch the Enum module. That's because there's so many helpful things in there and it's typical in functional programming that you're gonna be dealing with collections and those collections are gonna be passed from one thing to the next and you have to do something with those collections and transform the data somehow. And Enum is basically the module where you do that. So I love Enum. And Stream is kind of like a souped up Enum that can be switched out a lot of times directly with Enum. But we'll get into that a little bit more in detail in a second. So for right now I'm gonna dive into an IEX session. Hopefully you guys can see this. Do you think it'd be better with the black on white or the white on black? Okay, cool, I'm gonna take it just a little bit bigger. All right, so let's say we have, let me clear, let's say we've got a list and our list is just gonna be three items. Okay, we're gonna say elixir, let's say impacts and LA. Okay, so that's our collection, right? Just three items, not a big deal, three strings. Now what if we wanted to do something with each one of those items in the list? Now a lot of this is gonna be kind of, I know how to use an Enum but we'll kind of make this a little more complicated as we go on. So let's say we wanna capitalize or upcase each one of those items in the list. So we can sit in that list and I'm gonna do it this way. We sit in that list to enum.map and for each one of those I'm gonna use the shorthand notation here. I'm gonna upcase every one of those. That happens pretty quickly, right? It takes elixir, in your view it's elixir. So we've got elixir, impacts and LA all upcased. So basically what enum.map did is it took that whole list we gave it, all three items at once and enum kind of put it in its head. It put it in its mind, in its memory and said okay I have this list now on every one of these items I need to do this one particular function. In this case we told it to do string.upcase. Now of course we can take it a little further. We can say okay we wanna do that then we wanna send it to the upcase. So I put a fret to do the post fix thing here. And then we wanna send it to another map. We wanna transform it again and this time we're going to just take the graphemes and get each individual letter. So now we have a collection of collections, right? Each one of those is broken up into their individual letters and so that's fine, that works. So that's what enum does. It takes the whole list that goes to the first enum, gets it all in its memory and says okay I'm done now I'm gonna send the new list that I have into the next enum. And then our final enum says okay now I'm gonna take that string and break it up into its individual characters. So this is fine, right? This is a fast performant, whatever. This is milliseconds of time to spend in code. Now there's a problem though and that's when you get into large collections. So let's go back to this stream, right? So we have this stream, let's say you wanna operate on every single piece of water molecule in this picture as it's going over the waterfall. You know we can probably do it, it's not that big a deal it's just these little trickle but then let's say you got this and you wanna operate on every single molecule of water that falls over that stream as it comes by. That is a large data collection. So for you guys and gals if you're in Elixir maybe on the web you've got a log file for example that's very large or maybe some user input maybe a text of some book that you wanna analyze. Your data collection can get pretty large and in fact even in the stream module definition or sorry documentation it says that the data size could be infinite. So we have no idea how much water is gonna come over this waterfall in the life of this waterfall, right? We don't know how long you start your little web project and you're like this is gonna live forever and so the log file could grow infinitely big hopefully you do things to keep it from doing that but it could grow infinitely big you don't know when the end is gonna be so we could be operating on that log file to the end of eternity, right? Could be always always always operating so stream is something where you can operate on that. An enum it would have to get all of that into its head at once. So if we look at that waterfall it's not very easy to think of every single piece of water molecule in that waterfall, right? That's just a lot of stuff to put in your head. Same thing with enum that's a lot of stuff to put in enum's head. So here's your solution streams, okay? Let's go back to IEX. In this case I'm just gonna do a range let's say one to 10 million and for each one of those things we're gonna pipe that into another enum and we're gonna map it let's say we're gonna change it change them to strings. To string, okay so I hit enter and it's taking a little while that's because what it has to do is has to take that entire list of 10 million put it into its brain and on each one of those say okay the first one's gonna be a string of one and the second one's gonna be a string of two. All that whole list is going into its memory and then it's gonna spit it back out. So let's say again we want to take it one step further and we wanna pipe that into another map that changes into the graphemes. Now this is gonna take a little while. So let's pretend I am that final enum.map, okay? I'm sitting here looking up at the pipeline and at the very top I see the list of 10 million items it's huge, right? And then you can kinda hear the sound as it moves to the pipeline. So it goes to enum that map and we got the two string and two strings like I've got this this is my time to shine. So enum two string goes here's the strings I'm piping them out, I'm piping them out and so it's got that whole thing it's minuses I'm done it's your turn and I'm like oh crap here comes this huge list, right? So I gotta get my mind ready clear it and say I've gotta allocate all this memory now to figure out how to operate on this huge list, right? 10 million maybe not huge but it took a little bit of time to operate on if your user was waiting for that they would probably go to some other site and buy some other thing, right? So that's what enum does it takes all that into memory operates on each one of those huge collections at once and then spits it down to the next one spits it down the next one. So stream what stream does is something a little different in this case we've been using enum.map there is also a stream.map so let's just go in and replace stream.map or enum.map with stream.map yes that's a good idea did that wrong but that's okay okay so one to 10 million pipe that in to stream.map and for those we're gonna do the integer.toString and we're gonna pipe that directly into stream.map again and this time the string.grapheams okay so now remember how long the enum one took, right? So what happens if I press enter now are you guys expecting something real quick? Hopefully so because boom that's not what you wanted is it? You wanted the list you wanted the list of the graph themes, right? You wanted those characters but now we got this weird thing it's hard to see because it's cut off on the side a little bit. In fact let me do this. Maybe move this a little bit forward slightly. I'll just do that. Okay and then now the bottom is gone a little bit. All right there we go that works. Okay so now we have that stream and we have a list inside stream of enum it's keyword list and so we have our enum in there and then we have funds and that sounds like a fun thing, right? We got funds but inside funds is another list and they are two different things and they're functions. So what this returns to us is not what we were expecting at least it wasn't what I was expecting when I first thought well I'll just drop in stream and then nothing came out and I was like why not? Stream in order to operate on the pipeline has to have something at the end that actually requests something. So in typical cases you would use a function from the enum module. In this case you could use like to list or take or something like that but the very end of your stream has to actually request something. So streams are what's known as lazy. They don't actually execute and so there's a need for it to execute. So that's kind of where the life is about a stream is. So you're like streaming you're like in a, I don't know, what are those like doughnuts inflatable things called I can't, I'm blinking right now but anyway you're in though in those inner tube. You're like floating down the stream in the inner tube, right? That's lazy, you're just kind of whatever. So that's kind of what stream does. It's like I'm not gonna execute anything yet. I'll set all my stuff up but you don't need anything. Why would I, why would I work, right? Forget you guys, I'm just gonna sit here. So what if we need to take something? So let's go through that again. Two string and then the graphemes and then let's do enum.toolist. Now so what are you expecting here? I already pressed enter. Sorry, I should have gone up again. I already pressed enter and you might be expecting that this is gonna take hardly any time at all but in fact you see well this isn't hardly any quicker than the enum, it's still taking a long time. So this is where you have to really discover what you're trying to do with your collections in order to learn how to do stream correctly. Because at first I thought it was just gonna straight drop in and I would get these huge speed bumps but in this case the enum.toolist is still requesting the entire list of items, the entire collection of 10 million things. It's saying I just want everything you got and I'm just gonna put it into a list. Now what if we wanted to take only four of those things? We've got a collection of 10 million items but really we only want four of them. Maybe it's a list of questions or it could be a log file, we just want the last four things. In that case, if we did the enum thing, the whole list, again we'd go gjong, gjong and then it would go gjong to me and I'm like thanks for these 10 million items, I only wanted four, you jerk. So I got these four, you forget your other ones. What stream does is let's say we have a stream at the bottom, or a stream pipeline so let's pipe these things through again. One through two million, two string, we got the graphemes and from there we're going to take four. All right so again you might think this is gonna take forever because an enum it would but now, wow, right? Real quick, that didn't take hardly any time at all because here's what's happening now. Instead of enum pushing all those lists down the pipeline, stream goes at the very bottom, this enum take says, I need one and so at the very top that range goes, okay I'm spitting down one and then it goes down into the first enum or the first stream function, stream map function, the energy to string and goes okay one to string is the string one. Passes that down the next function and it poops that out, right? All the way down to the enum take one or take four, enum take four is like, all right I got one of my four, I need two and so the second one comes down three, four and once it gets to four it's like I'm done and streams like, well I set up all these 999,996 others for you, are you not gonna use them and I'm like no, I don't need them. It's like okay, thanks, you saved us a bunch of work. So that's where stream actually makes a huge difference in collections like this, is when you only need a subset of the entire collection. So let's take a look again at my slides here. Streams are lazy again as I said. So I thought well let me take some benchmarks because then I was like it all depends on your code, so let's use some benchmarks. So in this case I set up something where I had four functions in a pipeline and I thought okay well I'm gonna pipe those through in the functions through each other and at the very end I'm gonna call enum.toolist and see how long it takes. So I piped in 1000 items and the enum iterations per second was almost 1.6,000, almost 1600 a second. Stream was actually just under 1200 a second and so I thought well that's weird. So in this case stream is actually 1.34 times slower than enum, interesting. So why is that? Okay well maybe I just need a bigger data set. So I got passed in 100,000 items. Enum is 11.53 a second, stream is 7,000 a second. I'm sorry that should be 11.53 is the wrong number but stream actually happens to be 1.57 times slower than enum. Again the enum, I copied that wrong, sorry. But it's slower, enum, the stream is still slower. So let's go to 10 million items. Enum, 0.04 iterations per second, stream is 0.38 iterations per second and stream is 1.29 times slower. So even with a large data set if I'm just at the very end calling toolist there's really a detriment for using stream and that's where you think well why is that? Didn't you just say it was like a drop in? Kind of, so enum and stream are two different things. You notice that when we created that pipeline but we didn't actually take anything yet it returned us this weird data structurey thing, right? It said here's what I've set up for you but we haven't operated on it yet. So all that set up actually takes some computational power and so that's the case where it may not make much sense to do the computation and using the power to set up the stream pipeline if all you're doing is taking the entire list at once because the stream's like well I'm doing all this work for you but you could have used enum and actually been a bit quicker. So I thought okay well I've got four functions, I've got a whole lot of functions in the pipeline. What if I send it through eight functions, I pipe them through eight different functions, one with enum.map and the other with stream.map. So in this case I had 1,000. Enum did 581, stream did 527. Again stream is 1.1 times slower. 400,000, stream got a little quicker so it's 1.07 and then in 10 million stream went backwards a little bit but that could just be because of some deviation. So in this case stream again is slower. They're getting closer to where you wouldn't notice much difference between enum and stream. But here's where the big stuff comes in. So we have eight functions that we're gonna pass through so we have a long function list but at the very end instead of just calling two list what if we just needed to take 10 of those things? So for 1,000 of them enum can do almost 560 a second whereas stream can do almost 74,000 a second. So enum is 132 times slower than stream. This is where stream starts to show its usage. For 100,000 enum is 13,000 times slower, okay? And then for 10 million items enum is almost 2.4 million times slower than stream. That's because as we'd mentioned enum has to contain that whole entire 10 million collection of 10 million items in its brain and each step of the way. Another way we can illustrate this, let's go back to our list here. Say list equals elixir, impacts and LA. Okay, so what I'm gonna do now is in each step of the pipeline I'm gonna have our pipeline basically spit out what its current state is, like what the current collection is. So we're gonna take our list, I'm just gonna do it this way. So we're gonna enum map our list and for each one we're going to, let's just inspect it. And from there we're going to pipe it into the first thing we did which was our string.upcase and then let's pipe that in again to map and we're gonna inspect it again. And then finally we're going to do our graphemes. Okay, so in this case you can see here our first inspect goes through and has the entire collection, right? Our three items are impacts, or sorry elixir, impacts and LA and we see all three of those come out first. That's coming out from this right here, our first thing. Then we upcase them and you can see again that we have our upcase collection all at once. Now what happens if we change that up to stream? So let's say stream.map and we're gonna inspect those. Then we're gonna pipe that into stream.map. We're gonna do our string upcase. I'm sorry, oh yeah, thanks. Stream.map and then we're gonna inspect it again just like we did with the map version and then we're gonna map those to our graphemes and then at the very end, since stream won't actually operate this until we request it, we're gonna call enum.toList. Okay, so in here I inspect impsect, awesome. So let's not inspect, let's inspect upcase and then we're going to inspect there and then we're going to the very end. Okay, so in this case, remember the first time we saw all three lowercase items first and then all three uppercase items. In this case stream is saying, I'm ready for the first item and it sends elixir down the pipeline, right? It drips it down. So the first one comes elixir lowercase, which we expected, but the second item is actually our uppercase elixir. So we can see that that actually did just drip that first one all the way down to us. And then the second item was error impacts and that was drip down from lowercase to uppercase into our final graphemes and the same thing with LA. So this is another way to look at the fact that each one of those items are kind of drip down as the final thing says I'm ready for it. So this is where stream really comes into effect and really comes into its own as something that's very useful for your applications is when you take a subset of items or you just need to make sure you're doing one at a time through your huge pipeline or in your huge dataset. Yeah, yeah, absolutely. So let's go through that again. So we're doing our list inspect and then we go to uppercase and then we do another inspect and then we go to graphemes and instead of that, we're just gonna end it. Is that what you're talking about? All right, then we have that stream thing. Yes, yeah. So the question was, can you take something that may be that infinite stream that you don't know when it ends, it may never end. Can stream trickle those down as you need them even if it's not a complete set at the very top and that answer is yes. Another thing to keep in mind is there's, I mentioned that enum kind of finalizes that stream and takes them. There's another function in the stream module called run and that actually is something where you can use it to execute the stream but you may not need the final result of the stream. So in this case, let's say it's a log file and you need to write different things to do something to that log file line and then write that to another file, maybe two files, maybe three files, maybe send it to an FTP server, whatever you need to do to it but at the very end, you don't need to really know the output of that pipeline. In that case, run can be used to do something that you don't really need to know the end result. All right, so that's a run through stream. Again, I am Jeffrey Lessle. I'm writing a book called Phoenix in Action. I should have some discount codes for you guys later today, I'm hoping but you can buy it there, jeffreylessle.com slash phoenix in action. It's in Meep right now, Manning's early access program. I think there's two chapters available but the third should be available sometime, hopefully next week actually. This is where you can find me online, jeffreylessle.com. I do some blogging about Elixir and Ecto and stuff like that. I'm Twitter, Geo-Lessle and then GitHub at Geo-Lessle as well. So thank you guys so much. Thanks a lot, chef.