 All right, let's let's get started right on time for the last morning session today before you'll guys are free to leave for lunch the speaker is Well, the speaker is a Robert who's done pretty much everything That one could potentially do so I don't even know how to start introducing him to be honest Many of you will have seen him before I have talked to him before so I'll just It's just right dive right in and please get started. Cool. Thank you So I've got a I've time tested this talk and it's too long. I Had to choose between skipping out enough bits. I thought it wouldn't make sense or trying to fit it in So I'm gonna try and fit it in we might not get time for questions at the end tough But if we can they'll be great So why why talk about this is probably something that's worth taking a little bit of time to To mull on and my speaker notes are not Well, that's going to be debugging the slides is step one Right, okay. Yeah, so I've worked with Chris Armstrong before and he's the guy that wrote this package I'm gonna talk about today and Chris is a really good program He's been around for ages and ages and ages and who here by the who went to Tommy's talk about kinesis this morning Right, so half the room so statistically we've just completely around them. That's great and One of the things that the kinesis talk didn't really touch on is side effects Like I talked about how things can be linked But when you think about reasoning about a program Side effects is kind of the thing that makes it impossible for you to tell what your program is going to do and Python it's an imperative language. It is entirely about side effects There is nothing you can do in Python that doesn't sort of have that feel to it and That makes it really hard to reason about Python programs at scale Small ones are really really easy You look at the program you figure out what it does as soon as it gets slightly bigger that gets kind of hard And so he wrote this library to start bringing some of the things that been done in the functional programming world To Python programmers because like me he loves writing in Python He does it as his day job, but there are better ways to do some of the things we do Unfortunately, they are available in Python So I'm gonna give you guys a test. What's wrong with this code? What's the bug in this code? Yeah, yeah, just No, it is entirely legitimate. That's just reveal.js doing something a little bit wacky So yes, the highlighting is wrong. That is not the bug with the code Tom The code assumes English. Okay. It's not It's not accessible. It's not diverse. I don't care about that for this talk Right It has a side effect, right and this code will actually what would you expect this code to do if Standard out was not able to be written to what would be the correct behavior for such a trivial program The correct behavior would be to trace back and exit non-zero because that's what processes that fail to do the job I meant to do they are meant to do something like that Fantastic it works. It's exactly what you want No, it's not That simple program does not actually meet the contract of a posix process. It doesn't exit non-zero Now there's a Python bug for that But this is the simplest example I could come up with undefined behavior in Python writing really really rigorous code in Python Can be quite hard and this talk is all about trying to figure out a way of doing that that is Pythonic and tasteful performance reasonably well and doesn't make us all actually want to switch languages because this is just so much worse than Yeah, right, so let's let's see if we can get somewhere interesting about this Things that can cause these side effects for us global state yay My slides just did something unexpected must be written in Python. No, I'm using reveal.js bad bad bad joke. So What went on here? No, I don't know so if you look through the Program I did before and you just try and break it into little little bits If you've ever looked at Python bike code, this is kind of a no-brainer for you But what it's really doing is it's looking up in the global The name print it's marshalling some arguments to it. It's calling it. It's looking up sys.standard out inside that function Another global that could be defined to do anything It's part of the power of Python It's how things like mock work that you can switch out names at runtime in your tests and have it have completely different meanings and You actually then call the right method and then the output gets buffered if you haven't changed anything else You just run that code your output hasn't been written to the socket File descriptor whatever it's still sitting in your process Then Python exits And it goes oh, I've got a file descriptor here. I've got something in the buffer I'll write that out it fails to write that and then it starts to clean itself up and it's got nowhere to go Can't run any more Python code your code's already finished There's a bug in the interpreter, but If you want to test this code and be confident how do you test that you have to run a separate process Because you're testing something right in the boundary conditions, and that's just really really ugly So if we ignore that Mostly we could monkey patch stuff in we could provide a new definition of print and then we're only having to reason about the code That's in front of us. It's very local. It's very easy to think about I can say yes That code does exactly what I expected to do because no one is going to change global state on me No one is going to run me in a context. I didn't expect to be running, but that is exactly where the bug came in You could run in a sub process as I just said you can redirect the IO You can run it in the same process, but move the file descriptors around and that will give you kind of a bit of both worlds It's not as bad as monkey patching, but it's not as comprehensive as running everything in a sub process and really been absolutely sure So if we wanted to fix that code that kind of looks like fixed code We get rid of our assumptions we get rid of our globals we make everything explicit and visible We can reason about it. I know it's going to write exactly that file Yes, there's still a global there, but I know what global it is anyone reading it even if they don't know The exact definition of print in the standard library can see what that's going to do with a bit more fidelity And we're going to flush it. We're going to say we care that this is actually being written somewhere before our code finishes and if we do that We get to trace backs because of It's a long story It's not that long we still have the data in the socket when it exits and Because our flush fails. So it got into the file descriptor buffer. We tried to flush it. It failed It's still in the buffer then the Prython itself exits and it's still in the buffer So we get two trace backs one of them a full trace back the other one just the exception Which is all the interpreter can do but we have fixed the bug Okay, I've now fixed the simplest shallower's bug ever still wasn't super nice to test when I look at that and I Want to talk about Haskell now because I like talking about Haskell. I don't know anything about Haskell. I'm an enthusiastic Experimenter in Haskell, but I'm not actually an expert But one of the things I find most striking about Haskell particularly when I try and talk about those benefits to people who Have heard of it and heard of you know this wonderful thing called monads It has all these terms to solve problems that Python doesn't have a Monad is a way of getting Data dependencies Across pure functions and that's not a problem Python has because we're imperative So trying to pick up Monads and using them in Python is a bit a bit crazy And the reason of this is that Haskell's pure math it gives us huge amounts of rigor and being able to think about a Program and the sorts of side effects we saw here are much harder to create in Haskell, but You're not actually telling the program to do anything you're telling it how to calculate something if it were to decide to do it And it might decide to do it at some point before the heat death of the universe So This here is actually Equivalent to this Haskell code and it's kind of Ridiculous right what I'm trying to show here is that in an imperative language I can set up something that Doesn't matter like that This is one mean by throwing this off a problem. We don't have But I want to talk really really quick through Who had does no Haskell? Right, okay, so I'll I won't skip it So in Haskell this Greater greater equals an in fix function just like pluses in Python and it's called bind and It binds two things together and it creates that data dependency and it does that by creating a lambda So on that slide the backslash is equivalent to the Python lambda colon thing We create a function right here and now so it creates Action one bind to a function which takes a parameter called x1 and Is going to have a body definition so the arrow says the body is off on the right Action two is going to be bound to a function that takes a parameter x2 And that's going to pass x1 and x2 to action three and all of that's just going to be evaluated as a function It's a whole lot of boilerplate to be able to just write three lines of code So if you try and translate that straight to Python, it's just it's mental so You've got two lambda functions and you start out with the other side and you have to evaluate the function all the way down but actually this is a little bit wrong because the Dispatch and Haskell is polymorphic on type and The bind functions polymorphic on the on the left hand argument, so we don't have this monad context And you could go and create one you create a monad Class and you can create a binding unit and there's a thousand of these on pypy and I'm not going to add to that mess But you end up with something that looks maybe a bit like this We pass the monad into all of your functions because you don't have that capability that Haskell has of sort of having it as context and You pass one and two and we return the two together. Okay, so that's all cool But this is not feeling Pythonic if you guys are looking at this and going well no wonder I don't like learning Haskell. This is because this isn't a Pythonic approach to the problem But it does have a couple of redeeming features if you were to think about that initial program I started off which had a side effect and it did some lookups All of these things could be broken out of separate steps and they could be tested you could control the interactions with the global state Very very precisely if we were doing this So maybe there is something we can get at that's valuable from this Thing which is you know seven times eight times the initial code we started with and And that's testability now I Don't know about your experience with Python, but my experience with Python is that if it's not tested It's not a question of if it is or is not broken. It's a question of when it will break And that is in large part due to the things that make Python powerful the mutability the ease of expression and The fact that you don't have to go through a long TDS compile force the type checker to be happy to every step along the way You the flip side is you find out that the type checker would have told you about the problem now the problem is that That's still if you go back to that code there That's really not code you want to look at and test because everything's nested you can't get at those inner functions You can't really pull it apart and and and control it easily so The other related thing I'm jumping slightly around my slides is actions can do anything so you can still have all the global side effects You can't really control it. I mentioned that and Here's a better title, so There's a thing called the free monad and the free monad lets you put a language a domain specific language and write your own Interpreters really really really easily, but again, it's a monad which is solving a problem. We don't have but the technique That technique could be very valuable and Chris did that with effect So the free monad says you create a language to express what's going on So I'm not going to talk about the Haskell side of anymore that bit's gone. You can start breathing again You create a language to express actions now actions can be actual physical IO or they could just be your change in the global variable or if you wanted to go down to the Finds grain possibility you could even do just things like looking up names in your global program state as an action Like you can completely disconnect your Python code from global state using this approach if you want to so This here is an intent. It's the intent of printing a line. So it's just a simple three line class It doesn't do anything itself It's what goes into your language you build your structure This is what is going to be going on and you write some pure code and I'm by that I mean code that itself doesn't interact with the rest of the world you give it a given input You get the same output every single time So when you test it you don't get a surprise and production because your tests don't have anything that can trap you up and trap You and make it break You can return a generator which will let you treat this as a co-routine or you can return an effect object and And You might for something very simple just return an effect But when you start one who flow control and loops and so on I find it's easiest to express that using generators and co-routines rather than returning callbacks thunks that are going to chain on to other things and that's I think because the boilerplate gets in the way Generators are great for avoiding that sort of boilerplate the function except a single parameter Which is the intent? That it's going to action So you write an interpreter Sorry, they yeah, so the output value everything gets threaded through the interpreter And so that single value is you doesn't do star eggs or keyword eggs So you pass a dict of your parameters or whatever you need to be communicating amongst your program parts You write an interpreter and the interpreter is the bit where you encode the actual real There is something happening here. This is where your IO happens This is where you can talk to a random number generator and this is where you can consult global variables or whatever other things are appropriate for your production environment, so this here print and flush is my actual implementation of the print intent that's the thing that evaluates it and the interpreter object there is a composed dispatcher so the dispatcher is something that takes Intense or effects and hands them off to executors and glues it all together Type dispatcher is something that does that based on the type of the object Because this is kind of part framework and part solution. You can actually write your own dispatchers that would dispatch on Rather different things and one reason you might want to do that is because you wanted to glue it onto async IO or tornado it comes with a synchronous thing So you can just write normal code and a twisted thing so you can write twisted code using this But it's written so that it's very very easy to port to many different frameworks Event list would be another one that you might want to write your own dispatcher and the Base dispatcher here is the one that knows how to deal with generators regular functions and all the sort of boilerplate That you'd kill yourself if you had to recreate it every time and there isn't really doing global status Just understands how to map from one type to another Testing this is actually quite quite interesting. So We just write a different interpreter. We wrote an interpreter to use the thing in production We write an interpreter for testing and the interpreter for testing. That's where we avoid all the IO We avoid all the expensive stuff like talking to disk and we avoid all the things that can cause confusion like global state by putting That into our test interpreter So test print says I'm going to get some outputs Sync performer that decorator there says that this is going to execute this code synchronously I'm not going to be going and returning a twisted deferred or anything like that. So the Particular interpreter is tied to the execution model. So if I want to write stuff and twist it I would write twisted executors and All I'm going to do when I get that print effect is I'm going to pull out the Like sorry the print intent is I'm going to put up the line Parameter from it and put it on to my outputs and then in my test I can inspect that I say I can't assert here in the general case because while I could write an assertion in this deaf perform test function here if I was Throwing five or ten or twenty different lines at the user during my program I would have one perform test function here and I can't write an assert that's going to handle all those different inputs Without it becoming really hard and unreadable. So it's kind of easier to just accumulate everything that goes on and then write I've moved my mouth. Have a night. There we go and then write a so I have to create a dispatcher that will dispatch out to that and So that gives me my test interpreter. I Perform it synchronously note that Sync performer and sync perform terrible names and we'll get some much nicer stuff in a second And then I look at the outputs and I write my assertions here, but obviously we want to be able to do a bit better than that However, we have fixed all of these things. I did not need to monkey patch my program. I did not need to use subprocesses I did not use to use IO direction. I need to test my production interpreter once and only once I Might test that using subprocesses and pay the cost there But all the rest of my code however big it gets I'll never need to go and touch that thing again because I've exhaustively tested this bit that has all of those nasty complexities and That code was the production API. So the if I go back Over here, this is production code The real print function and the program method there It's not beautiful big code because I'm trying to keep really tiny examples for us, but it is production Little bit awkward with some of the closures. I needed to do when I was testing it But not terrible. I could write quite a few tests that way and be quite happy In fact, I have ad hoc almost exactly this in the past trying to solve these problems with Programs again a bit big and a bit slow to evolve and you want to sort of get your velocity up So you just go, okay, I'll fix the thing right in front of me Now If we get time I'll come back and talk about some more Haskell stuff here, but not right now There is however a dedicated testing API. Yay so it's got this thing called sequence dispatcher and Oh, one of the fun things about writing this talk is that as I went through the talk and I went to Chris I said, hey, this is a bit hard to use. He went and fixed things So this is the third maybe fourth iteration of the talk when I filed a bug and he Either accepted the bug or the patches, you know, I've been here how far along I've gotten that the evolution of the thing and It got better, but I thought we'll show you the path we go through So sequence dispatcher makes this much much more straightforward each intent is going to come through and It's going to look it up and match it. So did I put I? Think I managed to skip a line there. So Print is going to be compared by a quality here So the print object I wrote will fail because two different print objects with the same line parameter are not going to be equal Unless I implement Dunder EQ But what I can do is turn into a named tuple at which point it has all of those methods for me And I don't need to ever think again So the first thing is make your intense named tuples makes your code smaller easier to read and it works with a testing API in a really nice way And also makes it clear that you're not going to put Complex functions on those intents they are buckets for passing data around between bits of your program not actual classes You'll have classes you'll have real objects, but they'll consume those things rather than being combined together The lambda none there says I don't need anything back from this performer And like it's not like a read function where I'd like to return some strings this one. It's going to be doing a print It's not going to return anything so my Main code that the main function I had right back at the beginning is going to print this thing It's not going to try and evaluate the result sequence.consume There is going to oh I know what happened. Yes. Okay. You're getting the final API No This is the intermediate one. Sorry. I'm sorry about this It is why they speak so okay This is going to work through that list of things and dispatch them out using sync perform So with sequence.consume what that does is it creates a dispatcher out of that sequence that's ready for use and So you can then see sync perform sequence So it's kind of the same object. It might be nicer to have it be a context manager and Anyway, so this is the name tuple bit that I said just before so I can skip past that and this is what the Thing starts to look like if a bit bigger if I wanted to be able to read something in I'll create a read line name tuple and I have to define a production implementation of it So sys.standardin.readline and I have to update my dispatcher. So this is one of the costs your Interpreter has to be expanded every time you add a type that you're passing around You have to tell it how to work with that type that intent But I can now write a more powerful program I can write a program that prints something out And if it succeeds, I want to read it back in and then if that succeeds I want to print it back out again to you does that seem like Pythonic code though Yeah, so some people are asleep and some people shook their heads. So that's good So this is how you might write a test for this and this is using hypothesis. So it's just throwing random data at it It will throw a whole bunch of stuff and see what happens. It's a wonderful thing look into it But it's the same testing style I create a sequence I create some objects for it to compare against and I checked that they executed Sequence.consume also checks that they all executed if you just ran through half of your your program would be buggy And so flag that is an error in failure test Right loops so This is the generator API I want to talk about before you this I think is Pythonic code I yield the effect of printing. What is your quest my I get a line back from the yield because you can use it to get Values out and that is the effect of read line and then I'm going to yield that line back out to the user It's a little bit more boilerplate than just doing it as three direct functions But it doesn't have all of the huge mess of boilerplate that we have before and it's in exactly equivalent So I can run the same test and it will pass But I can also Make it a bit simpler by using a single composed dispatcher. Well, I have to use a composed dispatcher because I had a generator a new type in there, so that was going to use the bait bring the base dispatcher in for me loops Sorry, I'm rushing something out because I've gone over time and so If you want to loop It's really hard with that callback based approach where I return the thing I do the dot on I didn't really explain it because it just looks like line noise this here. I think is pretty clear While I haven't got the line I want Ask prompt the user ask for the line back and at the end of it print out another question was your favorite color So this is obviously not a complete program. I haven't written the full quiz here Again, I use a hypothesis test to generate test data for me the bits I really care about they're gonna be static and it's gonna say what's your quest read the line. I'm going to deliberately give it a Different quest that's the random thing coming from Hypothesis and then I print what is your quest again? I give it the line It's looking for to seek the holy grail if you go back to my Loop here to seek the holy grail is the thing I'm looking for and we can continue And returning from generators. This is a bug. I'm gonna discuss this with Chris and length. I haven't time yet You have to yield do return which it just looks for as a rapper to say oh, that's not an effect to pass on elsewhere in the program Actually, I think we can fix this so it doesn't need to do it There's a comment in the code and the docs that says you don't need to do this on Python 3 The code actually requires you to do it on all versions of Python So I believe the intent is there and there's just a certain amount of Figuring out that we need to do And this is the Newest testing thing he had is a performed sequence. You just give the same sort of structure, but you don't need that With sequence dot consume and you don't need to worry about closures It returns the result of the full thing out the end of it for you So you can see you know your program return this value or the function act like a function and gave you a value back And then you can write regular asserts against it. So I think it brings together all of those things and fairly nice and straightforward fashion Yes, I have got three minutes for questions Wow, that was certainly a very interesting talk Thank You Robert We do have a few more minutes for questions. So I'll just be coming around How is it used in paper? How have you found using it for real? So this is currently not used in PIP, but it is something I think might be a good fit for some of PIP's problems because PIP is all about IO and Some of the bugs we have are all about IO. This is used in Otter, which is a rack space Auto-scaling thing. So you have a heat cluster that's a cluster that's been defined and managed by heat But you want to throw more machines and more capacity at it when you have lots of load coming in So it's heavily asynchronous It's got external state that it has to synchronize against which is the actual state of the cluster and IO that can go wrong in every single way you can imagine because it's a distributed system and They're using this so slowly increasing the areas within the code that they use it Because it's got the sync perform and the async perform and that the idea of these Interpreters you can embed it in an existing program in a very small way like you can put just this little bit here Beside if you like it and then Glom things on around it or put it in another island over there and just add it and add to it as it makes sense Any other questions you guys should use the time that's available come on Well, let me ask a question of you guys does this sound interesting or are you regretting choosing this room? Yeah, right, so I thought the question was a bit harsh provocative Okay, um, no, definitely got me thinking about you know, Python at scale and what you run into and the awareness of it and things you have to think about and try out and so I Wouldn't use it because there isn't kind of immediate need but definitely interesting input cool Yeah, it's an interesting talk a little bit Fast for me to actually take all that and actually ask you any sensible questions afterwards But I'm sure it's something I'll have a look at in my own time later I'm here all day and tomorrow and I'm happy to be approached in the hallway track There's another question up to the right behind you. Yes, I think I had one question up here first I was going to ask you if you will commit to running a tutorial next year on This because I think there's lots of people here who are interested but maybe need a bit more time and some practical examples I'd be delighted Yeah, that's in one up there. I think probably the last one is there any facility for separating different kinds of effects and supplying different interpreters for different families of effects for example IO state manipulation and so on So you want to have two different interpreters and you want them to cooperate with each other When some an effect of one type is encountered go to that and yes the composed so the interpreter is the dispatcher So a composed dispatcher takes an arbitrary number of dispatchers and tries one and tries the next and tries the third and tries the fourth and so on so We're beeping So all yeah, all you need to do is to have one Dispatcher for each family of effects and then use a composed dispatcher that's appropriate for your your combination to join them all together very straightforward Very last question if it's a quick question Yeah, my question was more Isn't that creating a new language and shouldn't you use a functional language in the first place if you're solving problems where that makes sense No, so So yes and no yes is creating a new language. So is event lit twisted async IO tornado So DSL's of any sort you create new languages every time you create a language to talk about something whether or not the syntax has changed So you're absolutely right. It's doing that Is this one one that lets you mitigate some of the things that the Thereby default language in python gives you and that gives you good bridges to thereby default language I think it is I think going to Haskell for an existing code base is a very expensive proposition and going to Haskell for an existing Is a very expensive proposition now will it pay off and how long will it pay off? And is it the right thing to do a hard question to answer is separating out your IO and your global state manipulation from your Code that you can look straight in front of you to reduce your connaissance and to give you better Results from your testing and less unexpected surprises in production. I think it is So I think this can be a good thing without any impact on the question of moving to a different language I completely different interpret, you know, I guess color. Okay, anyone who's going to the pie ladies lunch The group is already meeting outside. So you would have to leave Right away Other than that, please join me in thanking Robert again for his talk