 Yeah, I am super pumped to give this talk. I've given it a couple conferences thus far in the room has been equally as packed So thank you guys for continuing that trend for me It makes me feel great and that I hope you guys have had a great conference and that you leave here feeling inspired Especially after this session may be inspired to never again write another loop. So you guys laugh you guys laugh I think that I can convince you that this is worth considering. So my name is John Kerry I'm a software developer in the Kansas City area. I'm actually from Lawrence, Kansas, which is a little bit west of Kansas City and I kind of have lived there for a while and helped do a couple user groups in the area. So I love talking at my user groups I love talking at the areas around Kansas City. So this is my first time here at Drupal con, but I'm not a newbie to PHP I've actually been coding in PHP for about 12 years or so and I think this talk Feels really good and true to me and that it's kind of a culmination of that kind of 12 years 12 years of experience and Also with working with my juniors mid-levels and seniors throughout my career to kind of see where they struggle and where their code kind Of balloons into this unmaintainable mess and I think that really loops are the crux of that Loops I think have become a magnet for complexity and if you think back to when you very first started programming You likely learned about loops really early on in fact maybe you were taught that on your second day of at a keyboard trying to learn to program and The problem is is that we have this early success with loops and we continue to go back to them we continue to reach for them when they're easy when they're at hand and I think there's some other Programming concepts that we can use if we really want to communicate better to the rest of our co-workers to our future selves and I don't know whoever might be reading our code in the future, so Let's get down to business So I'm from Kansas City, which is just a short flight in from New Orleans and like I said I Help organize one of the user groups there and that is the Kansas City PHP user group So we have a great mix of developers that come through our user group we have everybody from absolute beginners that need help installing a web server and PHP and Have really never programmed before all the way to people more like myself that have been programming for a really long time So it's always a challenge as a user group organizer to keep all these people equally engaged So we kind of alternate between coding sessions like hack nights presentations social meet-ups we kind of try and cater to a lot of different people so one of the nights we were having a hack night and I had to think of a way to keep my people entertained for the entire night So I reached for what's called a kata or this repeatable programming exercise that you can do to refine your skills to Try out new techniques and things like that So there's a really famous one called fizz buzz and fizz buzz is super simple So you print out the numbers from one to a hundred and for multiples of three you print fizz for multiples of five You print buzz and if it's divisible by both three and five for instance 15 You print fizz buzz so the output for such a program would look something like this and a Canonical example for that would look something like this Nothing too difficult here nothing extraordinary There this certainly wouldn't win any awards But often this problem is a nice little signifier when doing a job interview just to kind of figure out One can you program under a little bit of pressure and two can you actually program at all? So this is super simple, but when somebody sitting there over your shoulder, it is a little complex so This is a really simple problem But I find that I learned best as a more senior person when I limit myself when I voluntarily put Constraints on myself to say how might I solve this really simple problem Maybe in a different way and then what can that different way lead me to believe about other ways of solving bigger more complex problems So I thought to myself and I tweeted this out at the time I said here's a coding challenge. Can you implement fizz buzz without using control statements? without using ternary operators without these different ways for switching and doing like if and else and and while and switch and looping and doing all this stuff and This guy that follows me Jeff Madsen Tweeted me back and he said that he had solved this in a really creative way and I think that this is just hilarious So you basically take an entire array and you build keys with those numbers three six nine with fizz another array fives five ten fifteen twenty with buzz and then multiples of fifteen with fizz buzz you take The the range computes basically a sequence from one to a hundred and then we replace all equivalents of three with fizz Five with buzz fifteen's with fizz buzz. I just thought that this was so creative So next time you have a job interview if you were to write this for me in a job interview I would Absolutely hire you because it just shows that you're thinking outside the box and maybe showing off a little bit. So Sometimes that's good. Sometimes it's bad that would just be awesome So if you don't get hired for that and you use this just let me know and I'll talk some sense into him So it really got me thinking and it said Why don't we use arrays for almost everything and I realized that this is a Drupal conference And you guys have been using arrays for so long you'd be like man We've been programming with the race since Drupal for three and four, right? so This is nothing new to us, but it's more of not just thinking with arrays But this idea of collection oriented programming and this actually isn't a new concept There's entire languages that are focused on this simple construct of collections of data and transforming them from like piece to piece The unfortunate consequence is that a lot of those languages things like APL K and J have this syntax. That's just it looks like calculus And that's not easy to type in like a regular ASCII keyboard, and it's really kind of mind-numbing to read It's a good mind-bender to actually look at it, but it doesn't make a lot of sense in day-to-day. So we can get back down to functional programming and this idea of having collections of Data and transforming them over steps to create new collections at each step I think is a lot more readable and a lot more maintainable, but you guys are probably also thinking this is a Drupal conference We've been programming with hooks and functions for a long time. This isn't quite the same thing This is a little bit different. This isn't the same functional programming that you might be used to so Before I kind of got into looking at some functional languages. I kind of thought functional programming was this This is a lisp. This is a modern lisp called closure And I don't exactly know what this does, but I pulled it from their standard library Just has a nice example that fits on a slide. So this isn't the kind of programming I'm saying you need to write code in this style to succeed. That's not true at all But I think that there are some core concepts that are in functional programming in this idea of collection Oriented programming that can help us in writing our PHP whether you're writing procedural code Your new object-oriented code in Drupal 8, whatever that might be So there's three concepts that I want to talk about today The first one is called map and what map allows us to do is to take one collection and make a one-to-one Transformation into a new collection in between is a function that we basically run every single element through to create a new a new collection so if we think about this in terms of cooking and Cooking analogies always seem to work their way into programming But if we have onions and we chop all of those onions all of a sudden we have chopped onions if we have a collection of ingredients And they all need to be chopped in the same coarseness We kind of have a map operation across those ingredients Filter is the next one and filtering allows us to segment our collections into Well, it doesn't allow you to split them like other operations But filtering basically kicks out ones or keeps ones based on a certain criteria So you could think of like a number like if numbers are greater than X keep them in this case for limes If we're cooking with a lime, we really only want the freshest nicest limes You might buy an entire bag and say I need four of these but I need the best quality ones So you might quantify that and then look at all of your stuff and then keep the best ones, right? Reduce reduces kind of a wild card It's kind of hard for me to explain reduce because I'm not sure that I fully understand all of its capabilities because you can do so much with it so I kind of paired it down to a really simple example for this talk, but reduce basically let you put into Let you take a collection put it in and run a function over it But you kind of accumulate data through each iteration and that accumulated data is available to the next Successive call and this will make a little more sense with a little more code But for our cooking example think of it like you've processed all of your ingredients into wet ingredients and dry ingredients And then all of a sudden you need to combine them together If we combine them together, we could bake a nice delicious cake So Something like that. So let's see what some code for this would look like So PHP in its core library has a couple different functions each a function called array map array filter and array reduce Array map is super simple. We can start with our collection here at the top Just one through five and we need to multiply all of those by two So the map operation first takes a callback and that's any callable whether it's an anonymous function closure I'll show you some other ways different ways of defining those callbacks But the result of that creates a one-to-one mapping We have five elements here and five elements here in the very bottom and we get our new results now This is a brand new array the result of that is a brand new array. So we end up with two arrays still in memory so that One of the common questions with this is John. How does this affect performance? So sometimes depending on what data types are inside of your arrays you can get increased memory performance But under certain circumstances you can also improve CPU optimization. So Long story short, you should definitely profile your code before and after Implementing something like this to see like where are my slow points or where am I consuming a lot of memory? There's no hard and fast rule. I can't say Only do it under these circumstances. You have to take each situation uniquely Sorry for the tangent there, but So a ray filter So we have in this case, we're taking the modulo operator, which is essentially it finds the Remainder in a division operation. So in this case, it's basically just defining all numbers divisible by two It works the same way that map does except for our resulting collection only keeps where our callback returns true So two four and six are all divisible by two and they can be kept into that new collection And here's an example of array reduce So in this case, we are only accumulating and summing all of the data that's in this array Now there are core PHP functions to do that operation But it's easy to understand reduce just from a basic basic example with some code like this So the result of that callback is passed to the total variable as the first argument to that callback there So we get one plus two is equal to three plus another three is six plus four is then ten plus five is then fifteen So if you're a fan of Voltron or if you don't remember Voltron power Rangers This is kind of a visual way to resemble this right so you have your guys They transform into these other guys and then they become the big guy So that's a thing So I think that these ideas this map filter and reduce are actually more expressive to their reader What happens is and I'll talk about this a little bit later is when you get to a loop You don't really know what's going to happen inside of that loop When you have an operation like map or filter or reduce You communicate to the reader that what they're about to read is a one-to-one change a one-to-one map An operation that goes over the entire collection and you get a new collection out of it This is all about communication But also we get benefits of code quality. So I'll talk about that a little bit later, too so This idea of expressive language I want you to think about the word cool for a second The word cool is used all the time in our just daily conversations We reach for it because it's easy, but it's not very descriptive, right? Like think of think of chess like you might say yeah, what chess is cool. I like chess Roller coasters are also cool if you like that sort of thing But have you ever played chess on a roller coaster because that's really cool And there's actually people out there that try and do this Which is hysterical, I love it. It's just like so rebellious. It's so against the thing, right? So um So if we're talking about chess chess isn't just cool. It's contemplative and Roller coasters aren't just cool. They're thrilling and Playing chess on a roller coaster isn't just cool. It's connivingly rebellious You have to sneak the chess onto the roller coaster and then get a picture with it And then you know maybe get kicked out of the park. It's I love it So anyway, yeah, this idea of expressive code map filter and reducer are more expressive Yeah I just thought of this so it's also like if you were reading a novel and let's say you're reading something like Stephen King And he's like the wolves emerged from the hedges and they looked cool Nobody really wants to read that that doesn't say anything It's like you want to hear about you know the snarling breath and everything so Don't use cool. So here's an example of a really simple for loop This is something you would probably see where it needs to grab the current element It needs to do some kind of operation on it and accumulate it into a new variable. Well, why do we need to use? Four in this case. Why can't we just use map? It's the same exact thing and it looks like same the same exact way that our example did from before if We were getting the even numbers from a list Looping over them would look something like this and we could say why are we doing this in a for each loop? When we could do this with array filter That's the same code that we showed from the example before Same thing with summing again. That's a really simple example and it's harder to communicate that in this talk, but Anyway, just for one more example before we kind of dive off into another tangent Let's imagine that we have like a list of users Oftentimes one of your first loops as a PHP programmer might be looping over collections of data from a MySQL database I don't know Lots of people probably copied that from the PHP docs and then just used it right away and not thought much about it But again what happens is we start sticking more and more code inside of this for loop over our records And that could grow to be 20 or 30 lines long. That's not maintainable. So in this case We're just transforming this array to we're casting it to an object which then becomes an instance of standard class Let's we'll use this in the next slide to where then we say users of age less than 18 are kicked out We need to find it where we need to keep them like Jamie and Megan are our minor kids And maybe we need to do something with them and maybe we need to count how many Miners we have so of course we have to or we could have counted the filter from before So that's another instance of reduce. We're taking that collection and we're reducing it down to a new value in this case Sometimes it's a string sometimes. It's an int sometimes. It's a brand new array. It's really kind of whatever you want So this idea of functions really really like functions And I'm going to continue to show you why but I think that this image shows a really good example of why and that is Functions that their most simple core are are very simple. They have an input And they have an output that might seem like well, what do you mean John? Of course they do but The real benefits of functions happen when we start to drive all of the all of the functionality inside of this function to depend Strictly on the inputs that the output should be directly Related to the input that we shouldn't be able to execute this function multiple times with the same input and get a Different output this allows us to reason about the code and not need to reason about the other state That's in our system which might be a database. It might be a cache. It might be some global variable that you're trying to depend on Strive to write functions that depends solely on their input and return the same output So just a little bit of quick review on Callables if you're an experienced programmer this will end up being some review if you're still kind of getting into Writing PHP this should be a good little introduction So we can define functions as global functions where they're not within a namespace. They just kind of Exist and can be called from anywhere. So these are our global functions They usually have input like say hello if we need to say hello to somebody's name We can pass that name in as an argument Static functions can be defined on a class and then those static functions can be invoked from anywhere They're similar to global functions, but you need to qualify that call with the class name before you invoke it and often Time static functions are great because you need to kind of group some functions together Maybe on a user maybe in some type of utility class It's just a good way of grouping functions together and they can be invoked from anywhere Object methods are a little bit different if you've been playing with Drupal 8 You should be familiar with what an object method looks like but what's cool about an object method is it has access That object has access to its internal state. So you can see here in this line We're defining a new user and we're passing in the name John later when we actually call get greeting We have access to that name John And then it just says hello John So we don't need to litter the outside of our program by always continually passing in John We get this joined thing together where we have the object all of its state and its behavior kind of passed as one Anywhere in our program Anonymous functions are are kind of close to what I was showing so far, but they allow us to just in sorry, they allow us to Define a function anywhere, but instead assign it to a variable And what we can then do is pass that variable to different places in our program to either invoke it later Maybe even to serialize it and store it somewhere if you have some special code to do that There's a little bit of code in here to do that Later so anonymous functions are are just that And then we have this concept of scoping Scoping has to do with what data your function or your code has access to when it executes So this is the most basic example of scoping so we have this function called hello and it takes a name as its argument and We have this extra variable method here and methods actually not used in our input, but it's used Outside of this function call at the very top. You'll see that when we actually invoke that method There's no way that method can be bound inside of that method Sorry, that was confusing Yeah, so you can't use it you'll just get a PHP notice as a result saying hello with a global Method or I'm not going to pick bad terms for this. Sorry guys If you use this keyword called global Which you may or may not have seen within Drupal before when you kind of want to reach out into the global scope and say I want whatever value you have inside of this method value here, and then we could actually use it This is definitely a discouraged practice because that global value could change From other code that's executed before you again execute this function If we really wanted to have access to that method we would go ahead and pass it in as a method argument There to say hello, so then we can actually use it as expected this other This has to do with Trying to use a variable that's defined inside of say hello So this dollar sign name is not accessible outside of the function Rather, it's just again like an undefined variable at that point There is this concept called a closure and the closure is a type of anonymous function that has context After it's defined so the context that we have here is this method variable at the very top when we go ahead and define that function We kind of pass in method And it's kind of stored along with the with the anonymous function inside of the class So then later when we invoke it like with John down here. It's still already knows about method We don't have to pass it in again and again So that's a little overview of functions and why we want to depend solely on what's inputted input into our functions and Make sure that that returns the same output no matter what so if you were asleep now is the time to wake back up I know it's kind of weird in the afternoon. Everybody's tired from a long conference So we're gonna go back to some more high-level code now So here's I want to talk a little bit about the different callables that you can pass into these array functions because people say I don't want to pass in anonymous functions because maybe I need to reuse them Maybe I need to put them as static methods on a class somewhere and then test them through my test suite I don't want to define them as anonymous functions there. That just looks weird to me So let's walk through an example of how to do that So what I'm going to end up doing is taking this collection of rows that are at the very bottom from our database and transform them Into these user objects. So this is a static function called from array. That's on a class called to user It's pretty simple it's going to take in each one of these rows and pass it as an argument up there as data Basically unpack that data store it a state on the user and then we'll have a new user object so Using it with an anonymous function would look something like this So we have our build function, which is just an anonymous function it delegates to from array And we pass it right in and you might be thinking those of you that are experienced with this Why not just use user from array as a callable because it's only taking that same argument and passing it directly to our callback And we can in fact do that There's several different ways that we can define these callbacks instead of an anonymous function We can use this syntax here Whereas a string we can give it the class name and the static method name and PHP will go ahead and invoke it Same thing with if we have an if we have a tuple just user and then from array Or if we have the instance of the object itself We can also make that call So we all sat through a nice keynote this morning basically about being nice to your brain And I think a lot of this code that I'm trying to show You might think at the outset this is really confusing. Why do I need to do this? This feels like more mental tricks to go through stuff But as we get into this pipelining technique I think that you'll see that it gives us ways of isolating these little blocks of code into functions We don't need to worry about other variables that are defined and the rest of our code so When we're being nice to our brain There's often code that looks like this. I'm gonna switch to this microphone real quick while I'm at the keyboard Can you guys still hear okay? Sweet, so This is an example of a long for loop that I pulled from the smarty library Which is a kind of like twig. It's a predecessor to twig. So it's responsibilities for rendering html And this is kind of hard code to read those of you that are in the back If you kind of squint your eyes at it You can kind of see the shape that this code starts to take and I don't know about you guys, but it's kind of Disjointed it uses a little bit of different naming syntaxes and ways of representing the code and here I'll just help you guys out So you kind of get a shape of this code You can step back from your code and apply what's called the squint test now Does this code look uniform does it look like it's easy to maintain just from squinting at it like this? And I would say the answer is no and that we can maybe Go towards structuring our code in a way that does look more uniform to make it so that we can kind of put our brains on Autopilot when looking at some of this extra syntax that's there and That this gets back to working memory and working memory. I think I can go back to this guy so working memory is Basically like the RAM of your brain. It's this really fastly processing part of your brain that can make decisions quickly, but it can also hold values or hold context from time to time so you might be Walking outside and there's a lot of things going on things like maybe there's a jackhammer from construction that's going on There might be some birds flying overhead. There might be people walking around you Maybe you're on a college campus and you're trying to figure out exactly what's going on And if there's anything coming at you that might be a threat So your brain is kind of on autopilot because these things you see are very very familiar And this kind of starts to happen with code when we get a lot of variables that are in our function and What this does is it makes it hard for our brains to remember The values of all these different functions the longer those variables have to live So if we can reduce the number of variables that are in our code We can make our code easier to reason about and we don't have to think and hold in our brain like seven different variables That take up space in our mind So it it decreases our ability to cognitively process and think through the code that's in front of us So I have an example of this of how it would apply if we were to just look at some pros If we were just look at like a list of facts here and then derive some conclusions from it And then maybe I'll show how we can clean that up So imagine if you will that we have three dogs and they run into your living room Gina who Maybe your sister She feeds the dogs each a strip of bacon Except for your dog's back because he pooped on the floor and bad dogs don't get bacon Spotwags her tail and then rolls over You're definitely hungry and you eat two strips of the bacon from an oven that's set at 400 degrees Gina hands you fresh bacon and you drop it because it's too hot Five pieces of fall to the ground and all of a sudden there's no more bacon So how many bacon strips were prepared today? How many bacon strips were actually eaten today? Now there's a lot of Information here and we could certainly spend a little bit of time and read back through it to figure that out But if this were maybe written or communicated in a way In a different way, we could discern that a little bit easier. So let's try this again We have three dogs and they run into your living room And we know that Gina gives each dog a strip of bacon except for spackle. Well, we can kind of reduce that down to two When she only gives two dogs a strip of bacon So we kind of know there's two bacon strips that have at least been prepared Spotwags her tail. Well, that's not really relevant to us. So let's just remove it You're hungry and eat two strips of bacon from an oven set at 400 degrees We don't care what the oven was set out. We just care that you have the two pieces of bacon Gina hands you more bacon and you drop it, but we don't care that it was too hot We only care that the five pieces fell to the ground So all of a sudden we've kind of removed some of this context that we don't care about For the sake of answering the questions that we do care about and I think we can do these same things with code So there's a cartoon that's kind of been making the rounds over the last couple months And I think it resembles this idea very well So imagine that you're sitting at your desk and you're working really hard on kind of wrapping your mind around this code problem And you know you're digging into it. You're figuring out what all the different classes do Let's pretend that maybe it's Drupal 8 and it's the first time you've ever looked at it And you've got this big object graph in your head. You're trying to figure out Okay, this piece goes to here the config system loads stuff from the files. Oh my gosh There's composer what it's that and then your boss walks up and says hey did you get that email that I sent you all that context just goes It's gone It's into a black hole and then when your boss walks away. You're right back stuck at step one So These concepts we want to try and keep as little as we can in our head at any given time This has to do with small functions and simply small programs Jessica care is a host on the ruby rogues podcast And if you don't listen to this yet because it has ruby in the title You should definitely ignore that because it's a fantastic podcast rather whether you do ruby or not so she says a program can only change if it's alive in a programmers head and the code that we write is Not constantly changing but bits of it are changing and to be useful to the people we work for it needs to be able to change easily So one of the coding terms that I've kind of come up with and I haven't heard this described elsewhere Actually, oh, that's my next slide. So this concept here of nested control structures This is what I was talking about with where loops grow in their complexity so much We often throw these different if statements in there We often have nested loops like more and more for each is what we can really see if we look closely Is that each one of these control statements is actually its own little program? It could actually be isolated into its own little function. So instead of by the time we get down to this echo statement There's a lot more context that we've already processed through and I don't know about you guys But when I get that deep into a function, I kind of get a little fuzzy-brained and don't know what's going on I kind of like to have more assertive boundaries in the code that I'm looking at So for me it actually gives me a little bit of anxiety when I get down and I see that kind of stuff And I call this control structure anxiety If we look at the shape of that code that we just had where it's for each if for each if else Blah blah blah. It's just like oh my gosh. What is my developer doing or what was I doing when I wrote this code? It's really hard for us to remember that context with so much code on that same block You know are these new rules that we need to consider at each control structure? Are they new variables that we're introducing and when can we forget about the variables that were introduced before that? Are these just minor details or are these brand new contexts that we need to consider with each if statement? The readability of the code is also another issue. So like in this case you might think well these look like pretty well-named variables Custom field values. We always hear about not using X not using I but when we use the same variable so many times Like custom appears like ten times in this code block Field the same thing Value the same thing. There's so much naming redundancy here. It's hard to really see what's going on So just because we have these longer names doesn't necessarily mean that it's more descriptive and One of the things is that local variables are just noisy In my mind they they kind of take up They take up mental cycles as I'm reading through code So this is maybe something that you would see in like a common controller in an MVC application So, you know, we have to unpack some data from the request. We need to throw some data at the database do something with our students extract some data from those students and then return a response by the time we get to the end of that function There's seven variables that we have to keep in our head of how they've been defined How they've changed and things like that now seven isn't that much But it's just a little bit of mental weight and juggling that we need to do to even make sense of this code I would say that we should ruthlessly eliminate local variables If there's a variable that you use only once Just in line it. There's no reason to have it standing around in the rest of your code taking up mental space for your reader when we can just in line it and We know that it's not used after it's actually used in that if statement if your Objects have what's called a fluent interface where you can chain method calls together. You should definitely do that because each of these Instance variables this DB this connection and then the roster Each of those again is just local variables. They take up space We don't know where when they're being used if they're being used again and we can when we can forget about them So this idea of chaining Let's get back to map filter and reduce because I think this is where the real power lies Let's imagine that we have an array of numbers and we need to multiply that by two We need to remove any results that are over the number nine and then we need to sum them together So if we were to use the concepts that I just said earlier Obviously, this is more readable right and you're thinking John that's not more readable You're crazy and you're right and really the the reason for this comes from the argument Mismatch here between map filter and reduce and you might be thinking. Oh, this is just another dumb PHP haystack needle thing It's actually not map is actually very attic meaning it can take multiple arrays So that when that callback is invoked you don't have just one value You have two arguments or three arguments if you're passing a multiple arrays So there's a good reason that these arguments are flipped But it doesn't make for very good and easy to read code now does it? so thankfully I Have a solution for that which we will get to here shortly So you might be thinking well, what if we just put our code into these little Well-named functions, maybe that will solve everything and then we get back to this problem where we're reading From the inside out at first. We're calling keep under length of six which its value is passed to add Greeting which is then all passed to join We could introduce instance variables to hold these values But then we're right back to that other part of using instance variables when we really don't want to do that so this is Douglas McElroy he worked at Bell Labs and Bell Labs is where they invented Unix in the 60s and early 70s And in fact Douglas McElroy was the inventor of Unix pipes We owe a lot to this Mr. McElroy for piecing things together For taking collections of things and passing them on to each other So if you're not familiar with Unix pipes, let's look at an example of that So Unix pipes work on the command line. They take it where we can take small little programs That are written and based in the Unix operating system and we can pass to the output from the previous program To the as the input of the next program So here we're using the program cat to output the list of friends that we have We can take the output and each one of those lines and we can pass it to AUK and AUK can allow us to filter out all names that are less than six We can take the output of that call and pass it to XRs where we can then prepend the name or the word hello So what's really great about Unix and I think a lot of people have heard of this is that has small little programs that you can piece together and rearrange for different Different functionality and that's the power of it is that you don't need this one big program that does exactly what you need You have small pieces that can be rearranged and tweaked with options to then make larger bigger programs So if we can think about that from our code perspective What if we could create little pipelines in our code? And we could pass data through them and then maybe pass it on to another pipeline and then another one We could have aren't like an entire web app here We could have this on the screen where our input is Maybe jason from like our front-end client or it could be a rest call an API call into our system We can process that data unpack it throw it through some pipelines and then we can we can output jason xml Whatever our response format is So if we can build these small programs like think you know We've heard microservices are a thing these small little appliances and processes that run on our network and That same idea of unix pipes and building small programs Exists at the network level This is a slide from a talk called building real-time metric pipelines And this was delivered by Samantha Kenyonis who now works at aol and she was describing their ad feedback system where they They run metrics on all their ads and impressions and click-throughs and conversions and all that stuff So they had like a nice pipeline of data of how data gets from Like feedback from users all the way through to being stored in elastic search. So this isn't just a Pattern that's in unix. It's it like all levels of our stack So let's look at how this kind of could be in code At the very top of the very first line. You'll see haystack H-array This is a library that a co-worker and I have been working on if you're in the php community His name is Eric Poe you so you may have heard or know Eric But Eric was a co-worker with me at a previous job And he's also a member and co-organizer of the Kansas City PHP group So I get to work really closely with Eric and I've seen him kind of grow as a programmer So it's been great to see him kind of come up with this idea of Haystack is basically using an array and a string and Not having to worry about the infamous haystack needle arguments for all the different functions So instead array and string can be thought of as objects So I was looking at this library for him and I thought hmm I see that this is actually pretty nice for doing pipelining. I wonder if I could make that happen So I worked with him to make it even better. So let's see kind of what this would look like this is hra or basically the code that powers hra It's essentially just an object wrapper around our different array operations Pretty simple, but when we get into Actually chaining these calls together like it shows on line, too. We get some nice fluent interface stuff So what's really great about pipelines and the way that they can be formatted here is that it kind of resembles the English language We read left to right top to bottom Code can read a little bit like English and not in the way that Like Ruby tries to say that their code is English. That's not kind of what I'm going for here It's just more of being able to reason about it in the same way that you do English And what's nice, too, is that we can plug in then our return statements We kind of can read across we can see what operation We're doing what our function argument is and know that the previous output from map here is Eventually passed as the input for reduce I think what's really great about this whole thing is what's called path to completion and this idea is something I learned from Luke Rob lusky and Luke is previously he previously worked at eBay and By proxy PayPal went on to Yahoo and is now like a product director at Google So Luke did some research on web forms and conversion rates that PayPal had back in the day So if you remember this age-old interface This is like a checkout screen where somebody needs to finally confirm that they're sending money to somebody This is a super important action in the PayPal UI But if you have to read through this page It's very disjointed in where the eye needs to track to observe and confirm exactly what you need to do so instead If you have labels aligned to the right and then their descriptions aligned to the left you get this nice pretty line That goes from the top all the way to the call to action And I think that pipelining in the way that I just showed It resembles the same thing it makes the code easy to read its uniform Things and keywords are where we expect them to be So there's the example that I showed before of where we needed to take that collection multiply by two Remove anything over nine and then add everything together A little more easy to read than that nice big nested thing we showed earlier, huh? So let's refactor to pipelines really quickly. I have another example here. So this is this is something That's maybe something you would find in like a normal PHP code base So this is actually a recorded video that I did within my editor So it's not like I'm coding this live or anything So let's start pulling some code out of this for each and see what we can do with it So we have this state that we need to keep only users that live in Kansas So we can actually instead of using it as a temp variable We can just inline it And actually we can do the same thing with the age as well So maybe for this case, we're just taking users that are 30 and living in Kansas Next we can also inline these getters for the student. So instead of breaking them out into Instance variables, we can just use the result of the function call that they have We can do that with state and then of course we can do that with age and And already that for each loop is starting to look a little bit cleaner Now we can kind of better see what the developers original intentions were They're actually just accumulating found users based on this predicate based on this if condition here So before we can actually start doing pipelining. We need to introduce our h array object up there So we're going to use Roster and we're going to now use it as this object oriented h array So now that we have roster we've recognized that this is definitely a filter pattern So we want to definitely just call filter and we're going to pass it a callback So for each item in this roster, we want to execute this callback We will receive a student object as you can see online seven or six seven eight So each one of those guys will be passed into that function and then the result Of this filter callback will then determine which Students make the cut into the into the next iteration of our callback So we can extract that if statement throw it in as a condition for our filter and then We can actually get rid of this noisy for each statement down here And we certainly don't need this found variable this accumulator anymore because we've just Pulled it out from the result of the filter Same thing with the name the name is actually just converting our student objects And extracting the name out of it. So this is a one to one mapping That resembles our map operation from before so we can map over each found student And simply return their name Of course, then we no longer need this for each statement And we can clean up the accumulator variable from names And then we need to since this is an h array Implode doesn't work with an h array by default So we can call this other method just to array where it returns the native data type array outside of h array So this is good when you need to pass it back to a function that absolutely needs one So this next operation is a little bit weird We just removed this temp variable for names and we're basically chaining this map call onto the result of the filter call I think that's the last Nope So we can instead of using roster We can actually immediately invoke our h array object immediately after it is constructed Where we no longer need this roster temp variable We can just say that the names are eventually what this pipeline is going to return So we have the collection we filter it We map it and the result are just a list of names of the students that fit that criteria And then we can pass them down to implode and use them just like we would a normal array So i'm running a little bit short on time. I think so i'm going to kind of skip through this example If you want to check slides later, they will be posted But basically there was an operation where I needed to Look through a list of directories and find all php classes that could be instantiated I'm just going to skip over it because time This is what it would have looked like if we had it broken down into The different array functions and then we can take this and put it into a pipeline So we have our list of files that are in Scander The list there we see dot in dot dot Which are actually just the current directory and then dot dot is the previous or parent directory We can map over that to return the full file name for all the files that are there Because we don't just need the file name. We need the actual full path And then we need to remove all instances that are not That are not directories. We only want the files out of this And we have an interface there at the very last part and an interface cannot be instantiated on its own So we don't want to try and Create that because it would just cause php to die out So we can actually check using reflection if that interface could be Instantiated or not and then we can return that array from there So the good news is that haystack as a standalone php library is available today You can go out to github and install it from there Or you can download the source code and include it in your project If you're using laravel You actually already have access to some really similar functions. In fact laravel has a collection object that's returned from its illuminate ORM or its eloquent or i'm sorry You can call filter map reduce and all kinds of different stuff on on their returned objects Same thing with doctrine if you're using that in a symphony application The even better news is that this week we went ahead and merged in php 559 support So instead of using php 56 which would have precluded drupal from using this I was I got here and I was like, oh wait. Yeah, drupal 8 559. Let's make that a thing So the next major version of this will have this fix in there And but you could install it today if you wanted to depend on our 2.0 branch There's so many other collection libraries out there, but they all kind of do things a little bit differently a couple of these are Ones that you might find interesting I think particularly interesting is this php ds extension if you're using php 7 This is basically a replacement for spl and the Some of the original authors of spl and some of the more performance minded people in the community have Rewritten those to be much more memory efficient or cpu efficient, but they won't ever get into php core So you can actually install them as a php extension for php 7 So what does the future of this look like? What's awesome is within the last two weeks Sarah goldman who previously worked at facebook and helped work on hhvm and their language hack It just proposed this rfc to the php community and rfcs are basically requests for comment on brand new features for the language She proposed this pipe operator, which essentially does kind of what we just showed today Where you can pipe the result from different functions to the next stage here so this is This is the actual pipe operator And then this is what's called a pseudo variable And this is the main stick point for this really right now people are very very confused and saying What other place do we actually use variable or dollar sign dollar sign? You know that you could actually use dollar sign Dollar sign name and then it's called variable variable and that would expand into dollar sign john for example This is like some crazy php stuff there, but um, so that's kind of one of the main stick points If we were to actually use this in an example that we had earlier it would kind of look like this I don't know that that reads as well as the h-array stuff that I was showing earlier because we still are stuck with variable and the pseudo variable here By basically the output From the previous stage as input to here. So we're still stuck with this Back and forth syntax argument order and I think it's not so cool I don't know if it'll get into core yet or not. People are still discussing There's also an rfc out there for arrow functions Basically what arrow functions allow us to do is to implicitly return a result from a function callback So you'll see the main difference is this guy here We no longer have a brace or a return statement. So it makes our function a little bit shorter If we were to use that it would look kind of like this Again, not a major change, but what it's trying to do is put those function callbacks onto one line Is really the goal of it so Again, I don't know if that's going to get in or not If you're thinking well john, this is all great. Um, I use this other big Legacy project or whatever. How do I actually just start playing with this? I created a composer based project where you can download it It'll have haystack already installed along with a couple other libraries And there's a test suite that's in there where I I wrote the canonical example using just array map filter and reduce And then I want you to use these different libraries to make the test suite pass So it's a good little place to play around with some of these concepts Uh, if you want some more feedback on like mental things to think about or like how I kind of started to think about these ideas There's a bunch of different stuff here as well specifically the this idea of pipelining has been talked about um by Martin Fowler the second article there. That's kind of what got me started on thinking that this was a real thing And showing it in a bunch of different languages. So that article was really influential and then um Michael Feathers is another Programmer I really respect and he's doing a talk recently called collection pipeline design techniques Kind of more like the refactoring angle that I kind of showed today So writing functional code doesn't have to be like this We don't have to write lisp enclosure to get all the benefits or some of the benefits Uh, if you wanted to go whole hog and write your entire application response In a big long function call you could do what adam wavin did and you could basically write it just like this This is the like the main code for this ci server that he has um If you had to dig into it, it would probably be making more sense, but I kind of find a little bit hard to read as is so in conclusion The next time you guys are seeing loops Maybe think of using map or filter or reduce think about your other developers And maybe that you can communicate better what you're doing Because at the end of the day Bugs can't exist in code that doesn't exist If we can push all of these variables out all these other function calls out and we can isolate it down Into small little functions that their variables can't escape the scope That we've defined It makes it easier to reason about it's easier on your brain. It's easier on you as a reader And simply just be expressive Thanks So if there are any questions Oh, man that jumped to the end of my extra slides. So if there's any questions Oh, I need to uh Join us for sprints if you're sticking around on friday They're I think most of the speakers have been talking about them so far Sprints are awesome ways to contribute back to the drupal community And there's certainly rooms for if you're a first time sprinter or a more seasoned sprinter I have a list of sources since this is a recorded talk where people can find the sources And I'll take questions. Yes, sir. Yeah, hi On the front end in drupal. We have the underscore library which does seemingly the same thing as the hra right I've seen there's a php version. Do you recommend for or against using that? I remember looking at underscore for php And I don't remember if it fit my paradigm or not I've had like 20 different array or filter libraries Kind of thrown at me over the course of giving this talk and underscore dot php is certainly one of them So I would encourage everyone to check out that list that I had or ones that you find and just find one that fits your style Find one that either fits your requirements for Your version for your php version for your personal affinity for a certain author You know just kind of go about it that route I don't know that I have one specific one Even though I've helped contribute to one that I would recommend or say use or don't use One more question How does my idea work with multi-dimensional arrays? You if you think about multi-dimensional arrays, sometimes you need to Loop over them multiple times or at different iterations. It's like a loop within a loop within a loop I would say that you would probably want to break those out into independent pipelines. So whatever you're trying to do Whatever level you're trying to do it extract an array that fits that level and then process it through And then if you have additional steps you need to do to it I guess write another pipeline for it after that Because it's about breaking those apart. We don't want those multi-dimensional arrays Like processed inside of each other for the reasons we showed before there's too many variables. It's kind of hard to read and I don't know things like that. It's always kind of hard to recommend advice for where to and how to use them So every case is different So like I said before um You should definitely benchmark your code before and after using these Operations because you will see a cpu or memory issue Blackfire io has been fantastic for me to figure out what the performance of my system has been They're not paying me or anything to say this just like I really really like it It gives you a really great gooey interface into Looking at the performance of your system. So you can kind of drill down and see what your expensive method calls are What your max memory usage was what your response time was and all that good stuff. So super useful. They have a free plan You'll probably find a really memory inefficient or cpu inefficient spot and increase the performance of your application Cool. One last question Using iterators. So If I know correctly Iterators don't work with the core array map functionality So our library within hra I don't think currently supports iterators But if you really need to get into like cpu efficient and memory efficient ways I wouldn't say the iterators are necessarily the example. They definitely can help But there's a new concept in php 5 6 called generators Which basically are just like lazy evaluation of a list So you could get to the middle of that list Let's say it's a range from 1 to 100 and you end up only needing 10 of those elements There's no reason to generate a 100 element array if you only need 10 of them So generators can help you lazily evaluate Those lists. So I haven't fully wrapped my head around how that's best useful And I know a lot of other people have including some of the libraries that I showed so Again, it's all like Performance benchmarky stuff and things like that. So yeah, unfortunately iterators don't work in core array map filter and reduce But some of the libraries do accommodate them Awesome. Thanks guys so much. See you later