 Austin is going to tell us about refactoring, which is awesome and tricky in Python. So please welcome, Austin. All right Thank you. Okay. That's all right Again, my name is Austin Bingham and I'm going to talk to you about Python refactoring with rope and thrall I'll explain what both of those are. They're Two technologies that work together. I work for a company called 16 North part on it as well. So hi Oh This wrong button there, right a little bit about myself. I'm originally from Austin, Texas and my name is Austin So I'm Austin from Austin when I like to show this This this slide to give a sense of the differences in places I've lived Austin has statues of Stevie Ray Vaughan downtown a big guitar god and I went to the University of Texas So hook them horns about five years ago. I moved to Norway so vanger Norway there on the west coast Our statues there are different. We have three swords rammed into the ground So there's like 30 foot tall stone swords to commemorate the unification of the Norwegian kingdoms And this is this a few or the most beautiful fjord in the world. I guarantee come visit Stavanger sometime It's really wonderful And this is something I added at the last minute I was in the Hague the other day and we tend to think of Python as very much being used for web stuff and being Used for a lot of science and engineering, but I saw an art exhibit. This is one of the most famous Mondrian paintings here rendered in Using Python I I saw this at the bottom of the display and I had to get a picture because you can't really read it But it's import numpy import tk enter from rent import random art parts So they're using Python to generate really interesting art for currently a very overpriced painting But the Mondrian exhibit in the Hague is really really worth seeing and you get to see some Python So little side note. So what is rope rope is a Python library for doing Python refactorings It's it's that simple. That's why I kind of have the snake eating itself It's um, it's a Python module you can import but then you can have it look at your source code And it can do things like renames and and pulling out of functions and all that kind of stuff So that at a very high level that's all that rope is I didn't write rope rope is something that exists Independent of me. I've never met the people who do it But they're very very bright because it does a really hard job refactoring Python because Python is so dynamic Rope has a lot of very powerful IDE oriented features like code completion Finding definitions and occurrences. These are things you'll note by the way, which aren't actually refactoring They're just things that rope is also able to do Organizing imports a good refactoring looking up doc strings, etc. Etc. It's really designed to be used from IDEs and not used as a module per se It's not wise for you to import rope and start trying to use it yourself because There's a lot of ceremony involved in refactoring So you want your IDE to take care of all that for you and that's kind of what I'm getting to withdraw later on So what are the elements of rope? I'm gonna take a pretty high level view of rope just to give you a sense of the parts of it Well, how a refactoring looks and things like that. I'm not gonna get into too many details The highest level concept is the project a project is really just a directory with a bunch of stuff under a bunch of files and other Other directories containing even more files And this this is the tree of stuff that rope is going to manipulate for you when you ask it to do a Renaming or something along those lines So there's a project class and you instantiate that with a directory and that's like the base of all the things you do You create a project just like this. I think that's fairly readable, but it's that simple there That's created a project and it creates a little file for you and it's parsed a bunch of stuff and it's ready to go The kind of things you can do with projects you can walk all of the resources resources are files and directories And you can get the children of resources and continue walking. So there's this is just It's not important that you understand this is to show that kind of the flavor of the API It's very straightforward and really easy to understand if a bit under documented There is a prefs object that you can get and this thing is basically a dictionary of Keys to values and these values are all sorts of settings related to You know how much stuff should it keep in cash or where I Forget all the different pressers hundreds of them And this is if you look in your rope project on disk You'll see a dot rope project file that is essentially reflected directly in this prefs class So this is something you can interrogate at runtime if you need to if you end up doing ID embedding of some sort with with rope Right. I just said all that Resources again, those are things like files and directories and that this is a lot of what you end up doing when you work with rope is Walking these trees and finding specific things you want to work with and then applying refactoring to specific resources Again, not a lot of detail just something so you have a sense of how it all puts is put together What's more interesting from my point of view Sort of aesthetically and technically is is how refactoring is done So you have these concepts of projects and resources and so forth But how do you actually what does it look like to do a refactoring with rope from the code point of view? the first thing you do is create a Renate or in this case we're creating a rename refactoring. So I've imported the rename refactoring class. I've constructed one This can fail of course if if rope says that that's not a resource that exists Or you've like you know gone off the end of the file it can complain And so it's possible for the construction of this to fail immediately You can see this is only partially bound and what I mean is that we've created a rename And we've told that which project to work against and we've told it the resource to work against in this case a file called state Dot P y and we've told that the offset which I want to do the rename But what's conspicuously missing from this of course is what you want to rename it to so trod has this notion of these partially bound Incomplete refactoring is it's figured out if you if you then tell it to do the renaming It's figured out all the places it has to touch so interestingly You can use the same refactoring multiple times if you want to for some reason rename something to six different things So that's the first step in a refactoring with rope The next thing you do is calculate the actual refactoring what you would actually do in this case I'm changing whatever was at point 42 into the word taco copter if taco copter doesn't ring a bell It should because it's one of the greatest business models in the world. It's it's Drones delivering tacos to you. It's it's brilliant and hopefully one of these days will be a reality I think it was denied by the FAA in the United States. They couldn't do it one of these days the future will arrive Once you've calculated the changes you haven't actually done the changes you've just figured out what you're going to do You can print out things like the description of the change the changed resources Which is an interesting bit of information if you have an IDE and you need to refresh frames or something like that That's a good piece of information to get without actually having done the Refactoring so changes represents a fully balanced change that you can then execute you just say project dot do Changes is the refactoring you calculated and then this is when things actually get changed on disk. So that is the sort of 10 cent tour of how to do a refactoring in In rope and this is sort of the full process here from creating a project to creating or factoring Calculating changes and doing the changes. So really pretty straightforward conceptually pretty easy Technically very difficult, but that's that's ropes job to keep the technical part difficult and not have you worry about it too much Right one important feature of trod is this notion of history. This is the most historically accurate picture I could find on the internet. It's a blinkin with a machine gun on a bear freeing the slaves. So It's a joke All right history it keeps track of things in this history class every project has a history List I think is what it is or a list like I forget the exact class But it manages the changes that have been made and also the changes that have been undone So if I've made a change and I want to undo it I can do it But it also remembers the things I've undone and I can redo them and this is It's kind of fun sometimes to kind of shuffle back and forth between different refactorings to see Which is the one that you like the best, which is the one that may maybe is most aesthetically pleasing or whatnot To undo changes you can just say project of history undo very straightforward or you can select specific points in the past That you want to undo so you can go maybe for refactorings back and undo that one or redo it as the case may be This is this is transitive Maybe transitive is the wrong word, but it will undo all dependent changes So if I if I undo my most recent change or if I undo my first change It has to undo all the ones that have you know come afterwards So we'll be undo all of your changes and the same with redo because if it didn't do that Of course, it would just kind of spam your code and all likelihood of you wouldn't want that Redoing changes is very similar. There's a thing called the redo list. You can look things up in that And of course, you would never do this manually You don't want to be writing code to do this you want your IDE again to be asking the history object What is there and then displaying it in a nice window so you can check boxes and stuff and decide what to undo and redo But this is just the API at the rope level Another really interesting and very powerful feature is multi-project refactoring You may decide that you have separate projects in different directories in different places But you know that they rely on each other one one depends on the other and you want a refactoring that Mostly applies in this project also be performed in this project and rope can do that At a very high level it works like this you create a multi-project refactoring You give it the refactoring type in this case rename that you want to do I've just used rename for all of my examples here. There's many other refactorings You construct this multi-project refactoring and tell it the other projects not your main project All the other projects you'd like this to operate on and remember these are just projects built on some other directory tree somewhere and then You can use the the NPR the multi-project refactoring there to construct The unbound refactoring just like we did earlier Then you create the changes like we did earlier and change it This is clear the somebody who likes Java because it's the enterprise proxy manager implementation factory factory Yeah, and then you would do multi-project Perform on the changes the multi-project API is a bit weird and it took me basically a lot of experimentation to figure out how it worked But it does work I can say that with with with good confidence it works as well as rope works on single single projects Which is not perfect all the time We talked a lot about refactoring things. Thank you refactoring things that rope does They had a lot of obviously non refactoring related things like you know finding doc strings and finding definition points and so forth These are great ID tools as well. Frankly. I use Jedi for a lot of that kind of stuff now but I still use rope and trolled for pure refactorings I Said this many times, but rope is not intended for manual operation It's really tricky to get right and let your ID do the heavy lifting for you Let an integration tool do the heavy lifting for you and that is what trod where trod comes in it's trod is Okay, there's the definition of trod. They say it's not a few bit of a bubble. It's Norwegian for thread I thought that was kind of witty You know rope thread, okay Little pro tip if if if you need to find names for your projects learn a foreign language and just use the foreign languages word for That I'm not Norwegian of obviously, so I just I just picked trod and nobody in the world that uses trod for a project name So it was wide open What so what is trod trod is a an HTTP JSON server wrapped around rope? That's all it is all it did is put an HTTP API on top of rope so that you can call it easily It also includes I say various clients for driving the server right now it contains one and that's for emacs So I use emacs and so I needed this tool basically was why I wrote this So the idea is that your ID them emacs visual studio, whatever consent HTT basic HTTP JSON requests to trod and it will work on the file system and do the work for you It's great that the embedding is really really easy the embedding of a refactoring tool in emacs is really simple because It's just so simple so why trod why why did I go through the trouble of writing this there are existing tools that wrap rope for emacs One is simpler integration with editors everything in the world now my watch probably this table speaks HTTP JSON right so it's easy to send these messages back and forth So I didn't have to worry about embedding the Python interpreter in emacs or something along those lines Which is technically feasible, and I know how to do it, but it's more complicated than it needs to be HTTP is way more than fast enough for this kind of work people will say oh, but you know you're sending messages over HTTP What about response times no no HTTP is way faster than we can blink so of course it's fast enough for this kind of work The real time involved in refactoring is the refactoring themselves, and so I don't have to worry about overhead Or sorry latency This is really a great way for me to support multiple versions of Python in one emacs If I was embedding the Python interpreter in emacs I basically to pick the one I wanted or jump through incredible hoops to have Python 3 and Python 2 embedded in emacs as it is That's all separate processes It's really easy and really emacs knows very little about the distinction now between 2 and 3 and knows a little bit because I have To tell it but very very little and this really simplifies to 3 concurrent development, especially over multiple projects It's it's an architectural win in many many ways It's the proper level of abstraction the the the job of trod is Python refactoring not being embedded in an IDE and so this it's allowed me to pull them apart and really work at the proper level of Abstraction you can see I've solved the most heinous religious war in the world now emacs now loves vi because they both use the same tools If we all wrote our tools like this, we wouldn't have to complain. We'd all use the same tools really under the hood We just had different key bindings right so It's that simple believe me So a little bit about the architecture Good on time here This is like to talk about the Python goodness for the most part. I use bottle for the HTTP handling and bottle is wonderful Anybody here involved with bottle? Thank you. It's it's brilliant. It really is I just copied bottle.py dropped it in my source directory It's there. I don't have to worry about dependencies or anything like that It just works and it's awesomely awesomely easy to use. Thank you very much if you need to do HTTP handling use bottle Or at least consider it another one Pika or Pika who knows how to pronounce this? Pika okay, he should know because he wrote it. I believe right. Yeah, okay Thank you. Pika is an actor model implementation in Python, which is hence the the award here This is that's an Oscar. It is awesome as well It I started the implementation of trod without that and I had to handle my own concurrency because you need concurrent Asynchronous handling of requests because summary factorings can take a long time, especially on large code bases Well, once I switched to actors basically, I didn't have to think about it anymore This is another huge architectural win I knew that my state in this actor was Distinction my state in this other actor and I didn't have to worry about locks or anything It just works it just works and it it the biggest win is not even concurrency, which I don't really get It's the conceptual clarity of the program itself So if you've never used actors and you have concurrency nightmares or concurrency headaches Really make a branch for Pika on your project and try it out and I I almost can guarantee that you're gonna love it It's Even not pick any actor model stackless is essentially an actor model try that stack of this is incredibly cool for many other reasons as well So this is another big win. So thank you. Thank you for that The architectural diagram you if architecture has to has box has to have boxes and lines So I was required by Euro Python to do this You have bottle at the front receiving all of the inputs and two actors the project which to think the kind of okay Thing that kind of handles all of the actually talking to rope So it you know it takes the requests for do a renaming do a Extraction of method etc. And you can see that it works directly on the file system Which the editor is also looking at by the way, which is a bit of a trick sometimes The other actor is the state which basically is a thing that keeps track of the request to the state of that request And that could be queried through bottle as well. Hence the line obviously The tasks eight like I said, it's just a mapping of integers Which are just increasing integers for every request mapped to whatever the status of that thing happens to be failure or pending or success It's sort of a freeform thing. I don't use it much except to keep track of what's going on right now It's simple The state actor the state object the task state object is an actor and thus I'm call it thread safe. That's a reasonable term to use for this really just means it's asynchronous safe and multiple asynchronous tasks Can be accessed at any time and I don't have to again think about locks or anything It just magically works because there's serialized messages at every port Yeah, I've got enough time to talk about this the important part here is for the asynchronous operations intro So some request becomes in for a rename That's asynchronous because it might take a long time the the request just spits back immediately a task ID And then that's done from the client's point of view the client can then come in later and ask for task 23 and get the status back, but all the all the renaming actually happens asynchronously Synchronous things happen in a much more simple way find definition. We want that to happen as fast as possible There's no reason to continue without it's no reason to come back to the ID until it's completed So this is a synchronous operation There you go UML The distinction there's a couple of asynchronous things renames extracts organized imports And there's a couple of synchronous things things like find definition get history find the call tip that kind of stuff. Those are all synchronous I Love emacs, so I had to sneak this in really I the trod emacs is the only editor or the only place where I use trod in the only place that I know that's used at all Actually, so the code for using emacs or using trod from you max is there's very simple trod open and trod rename It's really that that's straightforward because emacs because emacs knows where my cursor is or vi for that matter knows Where your cursor is trod rename? Ragingroid which is from an asteroid's clone that I wrote It works because you don't have to tell it where to do it from it emacs figures it all out And this is this is letting your IDE do the heavy lifting of figuring out where to do the refactorings and stuff That's I keep hitting that point over and over. This is what I mean and this is what the HTTP messages look like as they come through Probably all knew that anyway Right, so how am I on time I have about okay I Could do demos or we could take questions the demos are just me using emacs doing some refactorings So if there are questions, let's we can start with that and I can do demos later if people want to see okay Yeah, I'll repeat the question. So the question fundamentally had to do with I have rope doing work on files And I also have emacs looking at these files potentially doing things at the same time right now that interaction is very dirty and crude I just I Let me pop the stack up a little bit here. I'm the only person who uses to draw it as far as I know So it works for me. That's that's what I know I I have my I have my emacs set up to once a second refresh or revert any buffers that have changed on disk So I know before I do a refactoring I just save my file and do the refactoring There are ways around this of course because one of the things I can ask rope is what are you about to change? Right and it can tell me here's what I'm about to change and then after the refactoring is done I can automatically revert all those buffers. I just haven't written the logic to do that I'm accepting patches. So of course if you want to do something like that I'm more than happy to consider it or build it in but you do have to be a little bit careful about That kind of stuff and you can of course get into weird situations where you've Made your own edits in emacs and then the file system is getting changed two or three times That's just a race condition. I'd say don't do it. Yeah Other questions. Yes I Don't fully understand what what it is you're asking about a me a Mechanistic renaming of arguments. I don't know what that means Okay, the answer is maybe I don't know rope the trod the question was about something. I don't I don't know So but trod has a lot of functions that I don't expose. Sorry rope has a lot of functionality that I don't expose right now In in trod. So if you are missing some feature We can explore whether or not it already exists in rope or if it needs to be added or what the state is. Yeah Any other questions Okay, well, we've got a little bit of time so okay you asked for it All right, so this is this is emacs and I'm going to do a quick refactoring So the first thing I need to do is start the trod server. So trod open and I tell it the directory Right and you can see of course that okay emacs is kind of managing this process this trod server It spits out some information. So that's that's trod running under emacs's control now Let me make that a bit bigger if I can You know at my age I should be able to do this I Don't know how to make the font bigger. I'm sorry Well, you can kind of see that the point is this on the right is my unit tests for my collision detection increase text scale Text Scale increase. Hey, look at that. All right, I'll do it over here Probably yes. Yeah in case you watch watch the nine cat. Oh, isn't that great? They make you feel better Okay Yes, this is how I spend my time Okay, so we've got We've got this function calculate collision vector Which is defined in collision dot py collision detection stuff for my my asteroids game and over here is the unit test and say I'm in my unit Test and I for some reason decide I don't like that name anymore, right? So I can say control XTR rename that to calc call back And I'll think about it for a few seconds and this this is kind of what you were getting at earlier You in the second row about the timing, but you can see okay now. It's it's changed Actually gave it the wrong name and that's okay It's changed it everywhere that you could you could find it So it's changed it not only in the call here in the test, but also in the definition one shortcoming I don't know if there's a great solution as it does this it leaves all these things Incorrectly indented that seems like a Rope shortcoming that I would like to get fixed. So if I don't like those I can I Can display the history now and I can see I've got this one thing that I can undo Which was renaming calculate collision vector to Calc call back I do want to undo that because actually not the name I wanted to give it so I'm gonna try to undo and It will think about it for a few seconds, and then I promise it will Yeah, now the history doesn't automatically update. So let's just ignore that for now. That's another fixable thing, but that's um That's sort of the the main things that it do. Yeah It's it's just in memory. So if I kill off the trial server, it's gone as far as I know it might be serializable But I've never tried I've never looked into it. It's never been an issue for me basically One one kind of cool thing that a lot of people might appreciate when they're starting to work with new code bases It can clean up rope can clean up your imports in certain ways and so in trod I've I've created the command trolled imports super smackdown, and it just runs all of the organizing and cleaning up of imports And I tell it which file and it does it and a second later. It will Boom. Okay. See isn't that much nicer than it? There's no leading in import dot dot all that kind of stuff It's all done and of course I can Undo that so That's it for the demo if there's no more oh more questions, okay, I'm can you can you say that a bit louder? Renaming decorators that's a really good question. I don't know Yeah, like I said, I'm the only ones who uses this so if I've never had to do it Who knows if it works or not? I Would think so because rope is pretty darn smart rope does incredible things well beyond what I would have imagined would be doable with Refactoring tools and a language as dynamic as Python, but we can we can try it out later. It shouldn't be hard. So yeah Anything else? All right The last slide there's some links to all this stuff the presentation is on github github slash a Bingham You can find it links to everything else and thank you very much