 Hello, thank you very much for coming My name is Antonio. I'm a software engineer at Yelp. I work on the metrics team So what I currently do is working on observability and real-time metrics and real-time processing Yelp mission Okay, the old mission is to connect people with great local businesses. We have more than 127 million reviews on our website and On average every month we do 26 million unique mobile devices accesses via the Yelp web app and eight four million unique visitors via the desktop With a desktop website We have offices all over the world and we are more than 400 engineers working both on the product and the infrastructure So a new introduction before to start This talk has been inspired by the talk stop writing classes by Jack the other each I highly recommend you to watch the talk is great and By the blog post the controller pattern is awful and other objective oriented heresy by EV again great blog post highly recommended All the code that you're gonna see in this presentation and trust me is gonna be quite a lot It's available on github at that URL so all the code runs and that's test filter crony to play with it It's for you Now I'd like to start from the very end of the talk with the main two takeaways so They might make little sense to you at the moment But keep them in mind and maybe at the end that they will be like, you know Cool ideas that you can use at your daily job So the first one is let users utilize all Python features instead of just in irritants And the second one is go for decorators when your classes have only one method and are instantiated only once alright So what's this all about why we're here? Well, this is the reason How many of you have ever used celery? Quite a few nice nice Well, celery is an asynchronous Job queue based on distributed message passing. It is focused on real-time operation But also scheduling and this kind of stuff. So the way you define a task in celery is Defining an app using the Celery API And then calling the decorator task on the app object It you just created in order to decorate your task function So this is the first example on their website And this is the pattern that I'm gonna show you today This pattern is very very common actually and is used by a lot of cool frameworks. For example, you know flask, right? and flask is a micro framework for web development and The way you define logic for your routine flask is exactly the same as you define task in celery you define a flask up and Then you call the root decorator over the app object you just created with some parameters in this case The URL you want the function to be executed for and then you just give it the function That is going to encapsulate the logic you want to execute This is a very common pattern for web frameworks In fact also the pyramid frameworks use it in order to define views Which are basically the same concept of roots in flask as you can see you just call the at view config decorator You pass you pass in some arguments and then the function you decorated It's gonna be the logic that is gonna be executed This is again the first example on their website Now what about classes as you notice the name of the talk is right more decorators and fewer classes So if you ever try to write a test in python It's highly likely that you have read about unit test unit test is now in the standard living python 3 so I guess is the recommended way of writing testing pythons and The way you do it is you define a class in everything from unit test of test case and You define some test methods that are gonna have your assertions in order to understand if your code is doing the right thing That's what at least what you expect and in case you want to execute some setup for teardown methods What you do is that you override a meter with a specific signature in this case set up and Here you may do all your setup logic in this case for example what the example in the python documentation does is instantiated a widget object and attaching it on the fly to test class now By test which is an alternative framework to write tests avoid the usage of classes Substituting them with the decorator So if you use the py test dot fixtures decorator in order to decorate your setup logic Then you don't need a class anymore and in order to access the object that is returned by Your setup set up function what you do is that you pass the name of the function to your test Functions and it works magically so as you can see the interface is much more linear You can use all the function all the concept from functional programming at everything and well at the end of the day is just shorter so That's cool. At least that's what I think So I want to understand how you implement this stuff It's like how do you write these decorators in order to avoid using classes or to build clean interfaces? Well, what you do when you want to understand how some code works you go and get up and read the source, right? so I went to the pie test project and This was the result. It is kind of complicated So I say to myself well play test does a lot of magic Maybe Pyramid is going to be easier nope It was even a little bit worse maybe but yeah again Pyramid is a mature framework And there's a lot of business logic and I'm those a lot of you know corner cases So Celery Celery was simple, right? Nope Celery was the worst of all like as you can see it's a wall of code I have no idea why they put all this code probably because they wanted to handle a lot of corner cases And so my reaction after seeing the Celery implementation was this one And I say to myself decorators are hard. I'm not gonna use them. So Who believed this sentence is true? No one that's cool. That's cool. Well, maybe someone there Well, it's actually not entirely correct a few months later I looked more into how the creators works in Python and Basically the way they work is that you define a function Which receives as an argument the function you want to decorate and the result of the decorator function is a sign to the function name That's pretty easy, right and In case you want to pass arguments to your decorator the way it works Is that the decorator function is gonna be called with the arguments before and then the result is gonna use It's gonna be used as a decorator as we have just seen And in case you have more than one decorator The way they get executed is that the closest one to the function that you want to decorate gets executed first And the result gets passed to the decorator. This is upper level So actually I changed my stance after that and so I decided decorators are Pretty easy to use but maybe I just are too right Well, this is not entirely true as well So this is the simplest decorator ever made I guess um As you can see what it does is just when decorating the function it prints decorating function Pretty useless, but it's a decorator and it works in fact if we execute it in the Python interpreter That's what we get the first time we import the function The decorator function is going to be executed and decorating function is going to be Is going to be on the output and then every time we execute the function nothing happens. So just execute the regular function So after these maybe you can think the decorators are easy to write Again, that is not entirely true so In case you want to implement the decorator with arguments as we saw before The way you do it is Defining an inner function in your decorator function And the inner function is going to be the real decorator the one that we saw before So then we're receiving the function as first argument While instead you can use the arguments that you pass to your ideal decorator In any other part of the function itself So the way it works For this decorator implemented over here is that when the function gets imported It prints decorating function with bar and buzz which are the two arguments that we pass to the decorator And then when the function runs the function runs without any modification at all Now if we want to do something more useful with our decorators like for example Changing the way the function is called or saving the result into a cache and the kind of stuff We need a wrapper function as well There's going to be in defined inside your decorator function And these wrapper functions receive an argument for arguments the arguments of the function you want to decorate So a piece of advice if you want to write code that is reusable Just use args and vargs. So you need to care about what the function you're going to decorate is taking as parameter and the way This decorator function works Is that again when executed in the interpreter? It decorates the function and at import time it prints decorating function. But now Every time we call the function The code that we've written in the wrapper function inside the decorator function is going to be executed before The function get executed for real Now with that in mind My stance on decorator is that they're neither easy or hard They're just a bit tedious to write you see quite a few corner cases to remember What once you know how to write them once you have this free example in front of you that it's kind of okay I mean doesn't take much time And in fact if we look at the flask framework, which is a great piece of software in my opinion This is the way they implement their decorator Which is not much different from what we saw before and it's much much simpler than all the others decorator we've seen and so far So basically in the flask class, which is the one we use to instantiate the app object We define a root method which takes some parameters And since it's a decorator that takes some parameters We need an inner function And this inner function is the actual decorator and in the actual decorator What flask does is defining an endpoint Using the options parameter that we provided and then adding a rule for the end point to the app Then it returns the function and it returns the inner function So Might all cause supposed to stop here to finish here But then I noticed I booked a 45 minute slot instead of a 15 minute slot So I decided to go for a real life example Um So basically we're not going to see decorators for a while What we're going to do is looking at the class-based interface that these example implements And uh, then we're going to look at how we can rewrite it with decorators. So the example I want to take is stutmoster Stutmoster is a real internal framework that we use at tl to extract real-time metrics out of logs where logs are just stream of events And uh, this is not really important for the presentation. They just give you much more context So, uh, the way stutmoster works is that it consumes log from Apache Kafka, which is a distributed streaming platform usually used to you know Put log like carry log around Your infrastructure and then it does some custom logic which is implemented by the users via an api And emits some metrics to signal effects Which is a third-party services that we use in order to visualize and analyze our metrics and to kairos db Which is a fast distributed scalable time series database on top of kassandra that we maintain internally So to rate rate The input for stutmoster are logs and the output are actually metrics This is an example of what stutmoster does So in this example, we see that the input is a log line, which is jason formatted with some information Then some user code gets executed inside stutmoster A stutmoster stutmoster emits a timing metrics So basically just you know timing the result of a function or something similar Which has a name a timestamp a value and a dictionary of attributes that we call dimensions in the metrics world Now let's look at the metrics interface the metrics interface. It's composed by three little pieces The first one is an anum defining the two kind of metrics that you can Emit which are basically just counters in order to count events and timers in order to time them Then we have the metric definition. We use the name topple here And a metric as we say as always a name a timestamp a value Dictionary of tags, which we call dimensions and the type and at the very end we define two commodity functions in order to Give the user the possibility to instantiate a counter and a timer without providing the type Using partial. Okay. So let's look at the input interface instead. So the logs So this is where this is where the class is already come in play basically the way a log class is defined is Defining at the coder function, which is going to use in order to decode your log line In this case, we provide the json decoding as default, but you can just override this default just setting your custom decoding function as a class attribute and then the base class which you are seeing at the moment Just define at the code class method, which is going to be used to decode log line And the second thing is the name of the log you want to tail. So stutmaster is a dynamic system So the way it works is just inspect at runtime all the classes defined in the framework and it checks for this name field And it just tails from Kafka all the logs named this way So if we want to implement a log to just say to stutmaster tail this log for me, that's the way we do it It's pretty easy. It's reasonable So we just subclass the log Base class and we just provide the name in this case events Now in case you want to provide your custom decoding function The way you do it is just that you define your function at the very beginning and then you Set up the decoding function as a class attribute of the log Now let's look at the stutmaster interface. So the real code is The stutmaster interface is based on the concept of trigger A trigger is a class which encapsulates the logic to extract metrics from logs And this is the base trigger class It does very few things So the first one is defining goners for every trigger So we always know which team is responsible which logic And what he does is asserting the owner Then this field is going to be a list of strings There is no way for us to encode this information in python And so we decided to start writing a little bit of a tutorial so people could read it and You know understand what they need to pass to the function to the to the class and then Every trigger needs to implement a digest method the digest method is what's is used To transform the decoded log line into metrics Actually in the it's supposed to be a generator And it's written in the tutorial as well So this is the core stutmaster. This is all The logic that really matters and that's the process function that receive the log we are consuming from The line we just consumed And then it iterates over all the triggers and it tries to yield from the digest method called on the entry so the decoded log line and in case of any exception it sends an email to the owners Now this is the way a user basically Instruments stutmaster to consume a log and then to emit some metrics out of it So as you can see you first define the log as we saw before Then you define the owners for your trigger in this case just me and Then you define your digest function Your digest method What it does at least for this trigger is just yielding a counter saying we saw an event with this timestamp And we just count one Here it is another example you define the resource usage log You define your trigger with your list of owners and then in the digest method here We loop over Just a custom table as time and you time and we emit your timing metrics based on the content of the of the log line So this all worked fine We published a tutorial and we started to have some users But then we realized that our users are not just regular users. They are engineers And you know what the problem with engineers is right you give them a lego set and then they build a flying death start out of it so The first question we got as soon as we released the tutorial and sat was How do I narrate from a base log class? Well, uh, I didn't think about it So, uh, I answered well Let's let's do so. So you define a module starting with underscore And here you define your base log class In this case what the base log class is doing is basically providing a decoder for Apache log lines and now With a bit of magic we tell stat master not to look For these underscore prefixed modules So we don't start consuming logs that don't exist Okay, that's cool So that's the way you use this kind of base classes in your regular module You just import them and you set the name of the log you want to consume pretty easy the updated tutorial and we thought we were okay well That was wrong because as soon as we updated the tutorial We got this other question How do I narrate from a base trigger class because now I saw you I can inherit log class So I want to inherit trigger classes as well All right Didn't find they didn't think about it either Um, so what we decided to do Was basically reusing the same pattern. So in this Same underscore module you define your base trigger with some logic in this case, for example, you Ask the user to set the metric name and the endpoints That are written in the log line as class attributes and then You also ask him to implement your get additional dimensions metric method, for example Where you can actually customize the metrics we are going to meet and then you provide a digest function Who does some logic and calls the get additional dimension method And uses the endpoints and the metric name that you define as class attributes And at the end yield the counter So with a bit of magic we say to start monster again Just don't look for these underscore modules for triggers as well And that's the way you use them. So you just import them And then you just set those owners endpoints and metric name attributes for the subclass And then you implement to get additional dimensions in order to customize your metrics and that's it. It works So again, we updated the tutorial. It started to be a bit long but users were happy for like 24 hours Because then I got this question How do I know it's from two trigger classes at the same time? Well I didn't think about it either, but you know python supports multiple inheritance. So we started to occur around it and our answer was You just don't So let's keep things simple. You implement your base class with some logic And then you inherit from it in your other base class and you define some other logic and then What you do is that define another base class. Well, you still inherit from it and you define some other logic And then in your real module you just subclass from both of them and provide to them the same parameters as class attributes So I get it. This is not ideal, but as long as it's documented, it's going to be okay, right? Then after more or less one week, we got the first code review in for adding a new trigger And this is the first question we got. But how do I test my trigger class? Because you want to write tests, right? Well, guess what? This time I was prepared You remember PyTest, right? So the way you can test your trigger is just defining a fixture in PyTest When you return the instantiation of the trigger you want to test and then what you do You call the digest method on the trigger And you test a certain metrics It's in the result of the digest method. Easy One minute after he asked me But how do I test my base trigger class? All right, what's the difference? I don't understand. Well, if you look at this class It inherits from service base trigger And service base trigger in its init Assers that the derived class define a metric name and the same time It inherits from trigger which in this init Assers That it defines owners as well So You cannot really instantiate it because the assertion is going to fail I was very puzzled by this. I stayed one day thinking about this even under the shower I didn't find any good solution, but then a colleague of mine Came up with a very good idea Let's use in the PyTest fixture The type function that pythons provide so with the type function You can instantiate a subclass of a certain class at runtime And also you can provide class attribute as a dictionary That worked And we updated the tutorial it started to be long enough that people didn't want to read it and Yes, I guess it's okay But Let's try to make things better, right? So I'm going to demonstrate to you That my face before using decorator was this one and after switching to a decorator based interface It changed it to this one So Looking back at the satmoster module We got logs as input and metrics as output. So what's missing here? What is missing is the trigger What is a trigger? So we say the trigger it's a class that encapsulates the logic to transform logs to metrics Well, just get rid of it. It doesn't have any actual Reference to the real model. It's just an abstraction that we constructed ourselves Why do we did in the first place? No idea. So let's remove it and Let's not force people to utilize just inheritance For the interface Let's give them the power to use all the features that python provides So in the new interface the metrics interface stays exactly the same no modifications The logs interface started already to change a bit So instead of using classes only via subclassings We decided to give the user the ability to just use classes either as they were meant to be used by python So you can instantiate your class this time and you pass the name of the log and the decoder function as parameters of the init method And no more session over here And the decode class method now became Simple method the only real difference from before Is that now every log class defines an empty set of functions Those functions are what are going to are basically the equivalents of triggers in the new interface Now this is the way you define a log in the new statmoster. You just use the class That's it. You don't even need the file for each log I just you instantiate it with the name and it works And in case you want to pass in your own decode function You just call the class and you pass the parameter. That's it. It's already much cleaner But now let's look at the statmoster interface So first of all the process function stayed exactly the same but became a method of the log class And again the only thing we did was substituting The trigger with the functions set that is attached to the log class Now this is the way we wanted the user to utilize the new interface The way you define some logic in order to transform your logs to metrics is now importing your log Object that you created somewhere else and then Calling a decorator method on top of the object Also using another decorator to pass in the owner's metadata to the function And just just define the function the function is exactly the digest meter of the first trigger we saw without any modification You just remove all the class thingy and basically The way the decorator is implemented. It's the simplest decorator ever What he does He just received the function as an argument And he just adds it to the set That is already attached to the log class That's it Now this is even simpler than flask And in case we want to provide some more helpful Decorators like for example this one which makes you able to register the same function for multiple logs at the same time What we can do is implementing a decorator with arguments and the arguments is the basically the list of logs And in this case what we need is just an inner function where we iterate over the log And we call the register method once again because remember it's a decorator But at the same time it's emitted And in case we want to implement this owner's decorator that gives some metadata to the function What we do is exactly the same as before We pass it to the endlers as parameters of the function And we just attach on the fly the endlers to the function because every function in python is also an object So I know that some people are religiously opposed to these So in case you don't want to monkey patch your function You can always define a dictionary as a global and then use it In the closure of the upper function. It works exactly the same Now this is the way these Register decorator is used as you can see we import two logs and we pass two logs to the decorator And actually I also want to point out that the owner's decorator removes these ambiguity about Oh, is the owner so just a string or is the owner's a list of strings like if you want to pass in more owners You just pass in more arguments and the python interpreter is going to check that for us As we don't need more assertion because the python interpreter at import time is going to check that we created the two logs We want to register the function for The code of the function once again is just a digest speed of one decorator we saw before Now let's go back to the questions we got So how do I narrate from a base log class? smiley face If you want to subclass your log class you can But if your idea is just passing at the coding functions and then avoiding to pass it over and over and over What you do is just use partial and then you instantiate the new logs using the result of partial Next question, how do I narrate from a base trigger class? Well, no more base classes no more assertions What we do is just defining a simple function We define a function yielding some metrics and having some attributes As the attributes of the class before as parameters in this case the endpoints and even the additional dimensions method Is now just a parameter for the function And then we call it That's it So we just yield from the function within our You know logic and that's what we do. We don't even need methods. We don't need some classes We don't need to remember to Assert for class attributes. It's all done by the interpreter for us for free But I'll do an error from true trigger classes double epi about this one What you do is just you define true functions. This is The digest method of the first trigger that we saw when we saw the example of the multiple inheritance This is the exactly the same code of the second trigger that we saw before You just call them both You don't even need to ask me just call functions Now let's go back to tests How do I test my trigger class now? Easiest before even easier You don't even need the pytos fixtures anymore. It's just a function. You call it that's it and How do I test my base trigger class? That's the thing. I'm the most steppy about It's Basically another function. You just call it again And the way you mock Your stuff is just passing some random strings like for example test as parameters. You don't need to use the type Function that pyson provides you don't need to remember about the inheritance and everything We're just passing a string So but now that we removed all the assertions, how do I make sure that all functions have owners? Well, we write the test What the test does is calling the collect function Which is the one used by stat monster to collect runtime all the logs We iterate over the logs and we just check that they have the owners attribute sets So well That's even better than before because we transformed a runtime error To a test that could fail before we even push some function without owners in production So it's everything we indeed so implementing these decorators and changing a bit interface only for user's sake now actually for The most part it was a same fishy interest Because you remember that he talks a lot about the import system of stat monster the thing that makes stat monster understand Which logs he needs to consume with classes he needs to execute and the kind of stuff Well, you don't need to understand the code, but this is the import system in the new stat monster Just look at it and compare it with the old one It doesn't even fit in the slides and actually this is a version that I wrote in python 3.6 So it's much more compacted the python 2.7 version that we used And that makes me very very happy now closing up The main takeaways of the talk were let user utilize old python feature instead of just inheritance as we saw and go for decorators When your classes have only one method For example, the trigger class has only the digest method and they are instantiated only once Actually, we never even instantiate the trigger class ourselves the framework was doing that for us And now instead what you do is you instantiate your log class whenever you want you even want to subclass it You are free to do it. You don't want to don't do it And all the triggers disappeared. They're just functions so Just to clarify. I'm not saying that we should move entirely out of classes from it's just Use classes when they're useful In this case the log class is still a class Because it has some states and it has some behavior and it's used as classes supposed to be used in python not only for inheritance So when it makes sense go for decorator when it makes sense go for classes And that's the end of the talk Remember that we are hiring especially for our offices in london and umburg So if you want to work with python and on big real-time Processing streams and kind of stuff just come to talk to us at our booth just in the big goal And don't forget to follow us on every social network possible facebook twitter and github. Yes. Nowadays github is a social network and Please read our engineering blog where a lot of smart engineers why better than me Speak about all the cool stuff that we build every day Questions Thanks, antonio for your presentation. We have time for a few questions no questions Did you run into any specific situations where Switching to using decorators made it hard and practice to debug certain problems So where the decorator obscured the actual problem that you were trying to get to That that's a very good question. Actually, so You may have used decorator they're very complicated like to do a lot of stuff and They are then very difficult to debug because they change the way the function is executed And then you got these weird stuck traces and you really understand what's going on Well, my true sense of this is that as in any While always writing code You want to keep it simple. So my true sense is decorators should always return the function untouched It's like the function should be able to be cold Without using the decorator and the decorator should stay really small So the frameworks that I showed you before are very complex decorators But they are very much your framework With a you know good open source community and a lot of features just came into it So my true sense, especially for your own Production code is keep your decorator simple and avoid to do a lot of stuff into them I hope you answer the question More questions to Antonio No We thank you, Antonio for your presentation. Thank you for coming