 Okay, so welcome back and Now we are going to have the next speaker and so I'm going to say welcome to Scott Hello, how are you? I'm very good. Welcome to a robot in 2021 Where are you streaming from? I'm streaming from Brooklyn, New York in the United States Cool. So what's time? What time is there? It's quite early for you, right? It's about 815 in the morning Okay, thanks. It's not so bad. Not so bad Cool. Who is the one in New York? Okay, summer like here. Yeah, it's you know, the usual hot and humid Okay Yesterday was about 30 degrees Okay, I I don't miss that. I'm reading in amsterdam, but we don't have that in the summer here So I think that's that's super good. Uh, I really like your t-shirt by the way. Oh, me too Support by ladies. Thank you for that. So Scott is working for Bloomberg and He's a senior engineer and a manager in Bloomberg and he's helping also Of course to develop partner applications and he's also teaching internally in Bloomberg So as I say, it's super good and he's going to be talking about fun tools. That's a lot of fun. So Let's put your screen in the stage Cool So all yours. Good luck. All right. Thank you So as I was introduced, my name is Scott Irwin and I work for Bloomberg engineering And today we're going to talk about a hitchhiker's guide to funk tools and it's kind of interesting that I proposed this talk at the very beginning of June and thank you for the conference organizers for accepting it And about three weeks later martin hines wrote a blog post pretty much about Funk tools and and all of the wonderful features that are in there So there's something sort of in the python zeitgeist that is is generating interest In what has actually been around for a while So this is not going to be in depth in any one aspect of it Several of the aspects you could spend a whole Conference talk just on that one thing but this is more just a An overview to give folks an idea of what's there for further investigation within your own projects So what we're going to cover is we'll talk a little bit about definitions and histories But that's very quick And then we're going to get into The substance of funk tools. We'll talk about simplifying function signatures function wrappers caching We'll touch on order types. We'll touch on reduce and we'll Actually introduce this concept of function overloading in python for those of you who are coming from Maybe a different language and you heard that python doesn't support function overloading. That's Not completely true Python does have a way of supporting function overloading Via the funk tools module and then we'll wrap up and I'll Include some references including a link to martin's blog post So quick definition in history So the python standard library module funk tools contains several higher order functions So what does that mean? What is a higher order function? So they are functions which act on or return other functions So basically they take a function as an input or they return a function as an output um A higher order function that many of us are Familiar with our function decorators there. That is an example of a higher higher order function um Just a quick history on funk tools. It was actually added back in python 2.5 in 2006 and its initial Uh members of the module were wraps update wrapper, which those two are very closely related and partial um additional functionality was added Reduce was added in python 3.0 and back ported to python 2.6 a total ordering and Comp to key which we're not going to talk about at all was back ported to python 2.7 plus lru cash was added 24 brought in partial method and single dispatch and python 3.8 brought in cash property and single dispatch method and Most recently python 3.9 brought in cash. So you can see that this A module is evolving over time and it is continuing to evolve Even in the newer versions of python so Signatures and what that basically means is partial and partial method So partial is a a function that takes a function as an argument And it takes a set of arguments both positional and keyword and those Inputs are essentially locked in arguments to the function And then partial returns what's called a partial object Which behaves like the original function with those arguments already defined and so that's a lot of words Let's look at a quick example just to get a sense of what that is so here In the example below we're taking the function pow and we're essentially pre defining the exponent to be two and so We're taking the return of that and creating a new function called pow two And here in the example we can see if we pass pow two just the single value five that we end up getting five squared or five to the power of two the value 25 And so this is just a quick example But where you would really want to use Partial is to transfer transform multi argument functions into a single argument function in places where it's required So an example here is map map requires Only a one argument function, but what if you have a two argument function or three argument function that you want to use in map It's possible without partial, but it ends up being much more verbose. So here's some At the top here we have the example with partial and uh You can see that it is relatively compact and fairly straightforward to read So i'm doing map on pow with an exponent of three I can just kind of read that across the list here is just because map returns a generator And I just wanted to realize the the results of that generator um, I could do the same thing by It by defining an external function and then using that external function in map but this is like two extra lines of code and and And where it's actually used it's not exactly clear what this pow three is So imagine that the pow three is defined in some other place Distant from where it's actually used now. I don't easily know what this thing is about I could also use a lambda which kind of brings the definition close to the use But there's a lot more typing here and and it's not 100 clear what is going on So the partial just becomes a way of Creating something that is a little cleaner and a little easier to read I can also use partial to simplify code I can use it to define functions that are easier to type read and importantly friendly to code completion So here I'm I'm using partial to Define a function called print standard error Where I'm just using the print the built-in print function, but I'm setting the output to be standard error So now when I want to print something to standard error elsewhere in my code I can use this print under bar standard error And because this is defined as a function I When I'm in my editor, I can I can do Code completion on print under bar standard error It'll be available and it just makes it simpler and easier to use where I could use print open print File equals blah blah blah, but that's a lot more to type in every place. I want to use it And it's not very friendly to code completion related closely related to Partial is partial method and the easiest way to think about this is that it's partial for class methods From the python docs it returns a new partial method descriptor Which behaves like partial except that it's designed to be used as a method definition rather than being directly callable The the function that's in the argument here in partial method must be a descriptor or a callable But all that really means is that if you have a method in your class it it fits the bill So here's a quick. This is lifted directly from the python 3.9 docs. There's nothing very Original here But I have this class cell. I have a property called alive And I have a set state for that property that will take a boolean either true or false to set the State of the cell to be either alive or not alive Now I I can use partial method then to create these two convenience methods set alive and set dead That just basically pre-define the true or false for set state and now inside In this case a repl but also in terms of uh, code completion. I now have this set alive convenience function available to me that I can just you call and use without having to type out The more input more verbose implementation so that's Simplifying function signatures. Now, let's move on to function wrappers Which is wraps and it's closely related update wrapper So wraps takes a function that it's wrapping Or that is to be wrapped. It's a function decorator used when defining a wrapper function It updates the wrapper functions attributes To be the same as the wrapped function and and as I've alluded to it's a convenience decorator factory for defining Defined using update wrapper. So it's basically it's it's syntactic sugar for update wrapper So if we didn't have wraps what we would end up with something like this So I have my decorator. I have an wrapper that's defined within that decorator and I Use that to wrap a function but when I And when I call that function the wrappers called then the functions called then that's fine everything functionally works But when I look at the function name It's it's this function up here wrapper It's not the original fun. So Where this is important is like in stack traces or in error messages It's going to it's going to be confusing as to where the failure actually is also the doc string Is picking up the wrappers doc string and not the original functions doc string So if we use wraps we essentially it just becomes a decorator on the wrapper function itself And it takes as input the original function and so now Functionally everything still works the same the wrappers called the functions called but now the The function name is what I expect it's the original function And the doc string is coming from the original function And so this is just to make my code behave more like what I would expect it to be Update wrapper is rarely used directly It updates the wrapper function attributes to be the same as the wrap function It's useful in situations where the decorator wraps cannot be used Such as wrapping a function after it's defined or wrapping a function you do not own So here's a quick example I'm grabbing From the string library the standard library function cap words And I'm I'm wrapping it in this thing called my cap words And when I call all my cap words it it works like I expect But the the name of the function as well as the doc string the doc string is quite long So I didn't include it here It's not reference. It's referencing mine, but maybe I want it to behave in the context like it It is the standard library cap words So here I can use update wrapper to take my original function and say Those function attributes just make them the same as the original string doc cap words and so now if I look at the name for string doc cap words for The local cap words and my cap words. They all are the same And if I call the cap words I It ends up calling mine. Oh Something important to note here is I update wrapper in addition to doing this work will return Essentially a reference to the function that I can then use later, which is what I'm doing here caching so there's a couple of different features of caching in the func tools The first one is lru cache. Um, these are its default arguments It wraps a function with a with a memoizing callable it saves time When an expensive function is sometimes called with the same arguments It caches the results of the most recent max size calls. So in this case the default is 128 So that cache will hold up to 128 entries Lru by the way stands for least recently used so as something more recent comes in the oldest one gets kicked out If typed is set to true The function arguments for different types will be cached separately. So if you have a function that takes a number integer three would be cached differently than the floating point 3.0 Otherwise, they're treated the same because they're equivalent values The l u r cache when you create it has some attributes um It has cache info, which will tell you the current state of the cache like what's in it. How big is Um, it has a cache clear method which allows you to clear out or invalidate the cache And it also has a cache parameters method which will return A a new dict so it's it you you can't use it to modify the value of the cache But it'll basically tell you how it was instantiated. What's it? What was the max size and typed used to create this l r u crash? Here's a quick example. Um, this is a fairly standard example. So I have a Fibonacci and as Those of you that have seen this before know that with bigger and bigger numbers It gets more and more expensive because I'm making all of these recursive calls So what I do is I put an l r u cache on it And what that means is that I can calculate out to fairly large numbers um But Fibonacci to calculate say for the Fibonacci value for 10 I calculate the Fibonacci value for eight and nine Well because of the way we've done this we've calculated it for zero for one for two so on and so forth So by the time we get to these larger numbers, we've pre calculated all the smaller ones and they're now sitting in the cache um so this runs fairly quickly uh, and What we can see is that we have 28 cache hits 16 misses is basically means that The first time it sees each number it doesn't know what the value is so it has to calculate it um The max size we got the default of 128 and it has 16 entries, which is the values for zero through 15 And here we can see what the cache parameters even though we didn't define them up here in the decorator We just got the defaults And we can see what those those values are Some caveats with lru cache. Um, the function's positional and keyword arguments must be hashable Which means that it can't take things like lists or mutable classes The underlying storage for the lru cache is a dictionary. That's and so those Function arguments that are used as a key into that dictionary. So that's where this requirement is coming from Um, it should only be used with pure functions. What that means is that the same inputs always produce the same output Which implicitly means that the function that you're caching should not have side effects. It shouldn't write to a database or or do something That would mutate the system closely related to lru cache is the recently introduced cache and this is just a simple lightweight Unbounded function cache. So it's the same as lru cache with no maximum size Um, because there's no eviction because you never throw anything away from the cache It's it can be written in a way that makes it smaller and faster Compared to lru cache with a size limit because you never need to compare about Have we hit the cache limit? What do I need to do? What's the oldest one? So all of that logic goes away um, the caveat with that is that if you're using it on something that takes a very large variety of inputs You can end up with a very large cache. So Use that one with caution and forethought uh related to these two or is cache property which is um Similar to property with the addition of caching The value is computed once and then cached as a normal attribute for the life of the instance Um, unlike property cache property allows, uh, rights without a setter method being defined Cache property only runs on lookup and only if the attribute doesn't already exist on the object Once the attribute exists subsequent reads and writes work like a normal attribute So quick example here, um We're Taking in a sequence of numbers. We're storing it as a tuple of a sequence of numbers. So now we know that that sequence is immutable um, and then essentially we're going to add a cache property standard deviation And which will return the standard deviation of that sequence of numbers So here, I wanted to give you a sense of like what that Cache property buys bias in terms of time Um, so I'm taking a very large sequence 10 million And I'm calculating that standard deviation on that sequence um The initial one it's a little hard to read here, but you can see that it started at Call it 11 seconds and finished at 39 seconds So it took somewhere on the order of 28 seconds to calculate this standard deviation The very next time I call it I still get the same answer But now it took somewhere in the milliseconds to get that answer. So it's basically just a property lookup So for things that are expensive to Calculate but don't change once they are calculated. This is a very handy thing to have uh, we're going to just briefly touched on order types using total ordering Total ordering is a class decorator. Um, that makes it easy to create well behaved totally ordered types um If a class defines at least one rich comparison operator, it will so the the decorator will supply the rest so what that basically means is that You need to define One of these four less than less than or equal greater than or greater than equal and the class decorator will generate all the rest of them Uh, additionally the class should supply an equal operator though. That is not strictly necessary um One caveat with this is it does come at the cost of slower execution and a more complex stack trace for the derived comparison operators because essentially they get defined in Using the one that you defined as uh, it's implementation So here's a quick example. Um, I created this car class It takes year make a model Um, and I define equal and less than and that's it and these are just relatively simple comparisons uh on year make and model and so What this shows here is though even though I only defined the less than operator I I'm getting the greater than operator for free And I'm getting it because of this class decorator this total ordering One more that we'll touch on is reduce Um reduce takes a function and an iterable and it applies that function of two arguments cumulatively to the items of the iterable to reduce it to a single value That's a lot of words, but if you've ever used the sum function. This is an example of a reducer um Well python has a built-in sum function. It doesn't have a built-in product function. So here we can use to implement A product function. So we take reduce we give it The operator mul or multiplier and it takes an arbitrary iterable and it has an initial value of one um, and so now I can call product on a list A range or any other iterable and it will just multiply them all together in the same way that sum adds them all together And so this can be a handy thing to use Um for for these kinds of situations the last one and this one in particular could Occupy a 25 minute talk all on its own and this is single dispatch. This is where Uh, we get into function overloading single dispatch is a function decorator which Takes a function transforms it into a single dispatch generic function This means that the implementation is chosen based on the type of the single argument. So you have a different function for Say integer versus a string versus a float versus a list versus whatever The overloaded implementations are decorated with a register attribute. So it so that When you call the function it knows where the implement where to find the implementation um If the implementation is annotated with types the decorator will automatically infer the type of the argument Otherwise you have to declare what the type is in the decorator. So what does that look like? So here I have a function fun that takes an argument. This is my generic function. So So This is the in some sense the default implementation So now I register um a function And notice here. I'm not even bothering to name it its name is immaterial It it's going to be an implementation detail for fun When it's called with an integer argument Um I'm going to also register it with a list argument And then just to illustrate These are both using type hints where if I don't use the type in I have to declare the the type up here in the function decorator So now I I've Defined my generic function. I've defined it for an integer type for a list type and a complex type So if I call it with an integer I get the the results of the integer Implementation if I call it with a list I get the implementation of that if I call it with a float I get the default implementation because I haven't provided Uh, essentially a type specific version of the function and if I call it with a complex number I get the the complex implementation Single dispatch method is the same thing But for class methods. It's a function decorator which transforms a method into a single dispatch generic function The dispatch happens on the type of the first non self or non class argument. So that implies that you can use it both on Object methods and on class methods So with that we're going to wrap up So the func tools module contains many higher order functions, which are both useful and powerful And using these functions can result in more readable and more maintainable code some references Of course the python language reference is an excellent place to start These slides are available on github at the link that you see here. It's in my github repo sj urwin As I mentioned before martin hines wrote a blog post Um Pretty much on the same topic Uh called funk tools the power of higher order functions in python um, there's a somewhat more, um, I think This one is from 2020 florian Dalits wrote a blog very similar introduction to python's funk tools module so Between these four pieces this should get you more interested and more Informed on funk tools And with that we're done Okay, thank you scott. That was really really a lot of information. I was learning a lot I know but there was a really important, you know, like I was seeing in the channel like a lot of people saying yeah today I learned something Uh, I I like uh funk tools. It's like a eater tools, right those two Yeah They are really really good um Okay, so I have a few questions um, the first one is um Will using partial method for the examples set alive and set that so that was those two examples that you are using Actually be pythonic Would it be using protection? Um, so I would argue that they would be pythonic because they are lifted directly from the python documentation themselves Um, that's a good answer It it's It's a way if you didn't use that you would end up having to essentially write A little two-line function that implements that I mean it would be relatively straightforward to write But now you're being more of a vote for both than you need to be Yeah per point Okay, so I have one more. I'm checking the tunnel if someone is And this was me. So I'm cheating But yeah, you know, I'm a volunteer. So I have I can I can ask questions. Yeah, what's your favorite funk tools too? um, so Probably based on slide count you can guess that partial is one of my favorites Yeah, um, but in terms of like most useful. I think I use at wraps more than anything If you're writing decorators you pretty much need to use at wraps That's true. That's true um, and then Yeah, we are I think we're okay um Could we define product with reduce to be in the scope and attack or talk? Could we define product with reduce? um, I think I'm a little confused on the question Yeah, me too. I'm I was just Copy pasting that so maybe that person can uh Yeah, I'm happy to answer the question, but I don't think I I fully understand what um, let me see Yeah, maybe Oh, maybe the question is if you can define product using reduce I think Oh, could you use map to achieve the same end result? Yes, you could use map to achieve the same end result um The the main idea behind using reduce is it's just a little cleaner and easier to read what it's doing also, if you're coming from a Uh functional programming background or you've worked with functional programming languages Reduce is going to look very familiar Yeah, that's true Okay, and I think I have one more Oh, no, no, that was just people interacting so So that was the last one Excellent But if you want to keep a talk with scott You have two options. You can go to wonder me scott and see if someone reached you there Or you can just stay in the break up room, uh, brian I'm I'm sure people will will be I'll be more questions And again, thank you very much for being a for being a speaker in the conference Thank you for having me next year See you in Dublin