 Okay, let's move on to the next talk of the day. Let me introduce you to Stefan. Afternoon everyone, thanks for taking your lunchtime to come and listen to me talk today. When I submitted this talk, after I submitted this talk, I realised that the title of the talk was pretty generic. And in theory, I could come in here and talk about pretty much anything I wanted to. So I sat down last night and I decided, right, I'm going to start telling the story of this little boy called Nigel. Who had a dream. I'm not actually going to talk about Nigel, I'm not going to talk about Brexit. Thank God, it's off the table for today. Instead, what I'm going to talk about is versioning. And how do you manage versioning in a Python project and in some cases in kind of projects in general? This is going to be mostly focused at people that are packaging their own projects. So libraries that you're submitting up to PyPy or whatever. But some of the things I'm talking about here today will be useful for basically anyone working on software, whether it's a web application or a library or something else entirely. For each of these, I've broken it down to three different sections and each of those I'm going to break down into two sections. I'll get into that in a second. Before I do that, I should probably introduce myself. I'm Stephen. I work at Red Hat. I've been working there for about three years. I still don't have one of those cool red hats. I'll get one eventually. I work on OpenStack and have done since before I joined Red Hat, for anyone that doesn't know what OpenStack is, it was the cool kid on the block until Kubernetes came along and stole our lunch money. As a result, one big difference though between OpenStack and Kubernetes is that OpenStack is almost entirely Python. The contributions that OpenStack has made to the Python ecosystem, I'd say, are pretty significant. As a result of my work on OpenStack and the spare time stuff, I've ended up contributing to and maintaining an awful lot of Python projects, and I've used a lot of the tools that I'm going to talk about here today in those projects, either by choice or it's been forced upon me. Before we dive into the actual tools themselves, a quick recap on versioning and what do I mean about versioning and managing change and stuff. So you probably should care about versioning if your software has users, if it's something more than a script sitting on a server somewhere or on your local PC, and if it's something that you're expecting to change or evolve over time, so you're planning to add or remove functionality. And from the Python perspective, versioning, number-wise we're usually referring to this PEP from a while ago which specifies version identifier strings like this. Most of the time you're not going to use all of this. Most people talk in more generic terms in terms of things like semantic versioning and calendar versioning like Ubuntu and that kind of thing. Basic idea behind versioning, of course, is that your user should know when you've introduced changes into your code, when you've gone and removed some functionality that they may have relied on previously, or you've added some cool new whizbang feature that you want the world to know about. So I promised three different kind of focus areas. The first of these is packaging. So again, this is mostly useful for people that are packaging up their own libraries and publishing them on PyPy. I'd imagine that most people that have written packages have ended up using setup tools to do that. Obviously a replacement for the legacy district tools project. There are cool new kids on the block here with regards to things like poetry and flits, that kind of thing. I'm not going to talk about those today. Instead, I'm going to talk about two kind of plugin projects that exist for setup tools that you can use to kind of help make your life easier. The reason that you'd want something like this is the way that setup tools normally manages its version is you put in a static string into your setup.py document or your setup.config document and every time you update that, based on the new version of your software, there's an infrastructure management track here, there's an automation track here, and there's a couple of tracks in Janssen with the idea being that developers don't like doing stuff manually when you can automate the process. And as far as this static string stuff goes, it's usually duplication because you're usually managing your version information somewhere else, typically in the form of Git metadata, Git tags, Git branches, that kind of thing. So the first of these two tools I'm talking about, setup tools, SCM and PBR are plugins for setup tools that allow you to pull out that Git metadata from your repo and automatically inject it into your package. But they work slightly differently. One thing to note, the first of these is a kind of broader Python ecosystem project. The other one is an OpenStack project that's used across pretty much every single OpenStack project and anyone else that wants to use it, I use it in my own projects quite a lot. Setup tools, SCM is by far the simpler of the two. You say that you put it in as your requirement for the setup stage of your package and you set this Boolean config option saying whether you want setup tools to do your versioning for you. And as soon as you've done that, it will go and it will figure out what version of your package is, inject that into your thing, and then when you go to build your tarball or your wheel or whatever, all that stuff will happen magically for you, which is all lovely. PBR does almost the exact same thing, but it also does a whole load of other cool stuff that I'm not going to go into because that's a talk in its own right, but it will do stuff like it will generate changelog files for you, it will generate an authors file, it will manage release notes, it will do basically everything under the sun for you. Whether you want it all or not is a different matter, but very, very powerful tool. And the main advantages, like I said, of using these tools is that you can just completely forget about versioning from the perspective of actually having to hard code some strings somewhere. You just tag your package, you build your tarballs, and you push them up to PyPy and you are done. All very nice and handy. The second of the three focus areas I want to look at is how do you actually declare individual changes in your package or your library to whoever's consuming that library, whether it's someone in your team at work or some random or on the internet that just happens to consume that. And the normal way of doing this would be either you don't document it or you document it in some giant changelog file that people have to go down and skim through to figure out what the hell has changed and why does version two of this library no longer contain these APIs or has changed the behavior of them. And developers like the idea of hard coding a version string into your package, they tend to hate writing ducks and if you can get them to write ducks at all you're doing pretty well. So what you'd hopefully be trying to do is get the documentation to live inside in the code itself. And this is what projects like Django in the previous talk and other tools like Sphinx do themselves. They have their own custom tooling built around the warnings framework where you import something and it's deprecated in Django 2.2 and then removed in Django 3.0. You'll see a warning pop up when you run try to use that API saying this still works but it's going very soon to switch over to something new. And you can go and reinvent the wheel you could go do all this yourself but why would you do that when there are tools out there that will do it for you? So once again we have two tools first of these is one that's from the border Python ecosystem and another one has been developed within OpenStack to serve a similar purpose. As with the last one I have a clear preference which I'll try to not highlight too much. Deprecation is the simpler of the two it gives you decorators that you can decorate random functions with and you can say that the thing has been deprecated what version it was deprecated in when you plan to remove it, if you plan to remove it and some details for the user to actually figure out why these things have changed and maybe what they should do instead. You go and you run this code and you'll see warnings like this popping up with your code, exact same kind of thing you'll see if you've ever worked on Django or if you've ever used Sphinx from an extension perspective. It's very simple, very easy to use. Debt collector initially looks very very similar again a version, a removal version some messages to help inform the user as to what's actually changed and the behavior looks almost the same but it also lets you do things like move modules between different places, move classes that kind of thing so you decide that you want to move your compatibility module under the utils module you can do that, you can put in an alias essentially in the old place, move all the code across and then eventually remove that alias once everyone has had time to switch over. Again, super super powerful tool and a very useful way of avoiding reinventing the wheel and it looks almost the same when you've actually gone about it. The third of these then is possibly the one that's most useful to almost everyone here because this doesn't necessarily require you to be working on a library this is also useful if you happen to be working on a web application or some GUI application or client, whatever. You want to be able to expose not just API changes but any kind of change back up to your user and say I've gone and removed this thing or I've added this thing and the way that people as I referenced earlier normally do this stuff is they write this giant ass change log file or news file or changes file or whatever you want to call it and they pump in all the information that they want either iteratively as they go along through the cycle or some poor fella has to sit at the end of the cycle and try and figure out what the hell has just happened which is not fun. The other thing you can do is try and pull this information from, for example, your Git logs just do a diff between two different versions but that means you have to be very, very regimented in terms of what your Git commit messages look like. If any, I, because I work on OpenStack I tend to use Garrett on a daily basis so our change log commit messages tend to make sense and then I switch over to GitHub and I see pull request and it's like I'm going to type or fix up another type or fix up another type or they don't make very nice change logs. And one other big issue with having this giant ass changes file if it's done through the course of the cycle then if you have multiple people contributing to a project you end up having merge conflict hell where you have three or four features going in in the space of two weeks. The first guy that happens to get their feature in is a log update in everyone else has to re-spin their patches and keep re-spinning them just to update trivial changes. This is an example from Sphinx. In this case I just gave up writing change logs. I figured that the maintainers were going to do it for me. Thankfully, if you're not Sphinx there are two different projects that you can use to do this for you. Once again, one from outside of OpenStack and the other one from inside OpenStack the one from inside OpenStack is super, super powerful maybe too powerful but Towncryer, the first of these are ultra easy to use. You write these news fragments you dump it into some predetermined location and the fragments have a specific format. They have a ticket ID whether that's a GitHub issue tracker number pull request number whatever they have a type of release note so if it's a feature, if it's an upgrade impact that kind of thing and then they have the suffix and this is all restructured text based might support mark down I've never tried it go figure that out but you write your change log into one of these files and at the end of your cycle when you get the side that you're releasing my package 0.9 you'll go and run the Towncryer command and I've passed this with the draft flags so I can actually see what I want on the command line but what that will do is it will read all of those fragments and it will go put the order them accordingly squashing into one giant news file and dump that into again a pre-configured destination you can then version control this add a release 0.9 commit and include that as part of your tile ball that you push up to pipe out very simple again very nice and easy to use and this is what projects like talk use possibly pie test and a few others as well Reno is a whole other level of complexity so from the high level it looks pretty similar to only instead of having to manually create a command it provides these sub commands that will create the abstract for other fragment for you again into a predetermined place and that's a YAML file with restructured text snippets inside so you can have a release note for a particularly large feature that has for example it adds a new feature it deprecates an old feature or has some upgrade impacts that people deploying this library need to be aware of which you don't actually need to go and combine this thing at the end of the cycle because Reno is smart and Reno is aware that you're using a Git repo and it's aware that or it knows that at the end of your cycle you should be tagging your commit message or you should be tagging your chosen release thing to say this is release 0.9 if you go and run Reno you can specify a particular version and it will skim down all the way through your Git tree and it will figure out which of these release note fragments you have map to a given release so I added five release notes for 0.9 to 1.0 it will take just those five and it will put them out into a nice restructured text file that you can include in your documentation and it provides stuff like Sphinx extensions so you don't need to actually worry about having to go and generate that file manually you just put in the directive somewhere and it's just nice and it just works the only time Reno breaks down is if you do what Sphinx does and they have stable branches that they decide to merge back in on a periodic basis then Reno gets confused with circular dependencies and that kind of thing but generally the way that I'm used to projects working you decide that this is going to be my stable release and you branch off and you just back cherry pick bug fixes as you're going along and those bug fixes just happen to include release notes so your stable branches get all the new release notes for all those bug fixes that you're back porting and like I said this is what we use across the entire OpenSec ecosystem now so this is an example of release notes from Renova maybe something else and as you can see we didn't write this single page this is all a whole load of release note chunks that people have contributed as they've gone and worked on these features and at the end of the cycle the thing has just been automatically generated and will continue to be automatically generated every time we build our docs and that's Towncry and Reno and with that I'm going to return to us regularly scheduled broadcast and get back to talking about Brexit so yeah, quick wrap up version 101 three kind of focus areas how you can use your packaging to automatically version your software through your packaging how you can indicate to consumers of a library that you have made changes to the API of that library and then finally how you can go and document these changes to both API consumers and everyone else and with that, thank you very much for coming and I'm happy to take any questions here or afterwards yeah, we have time for some questions I'll take questions, yeah yeah, we have time for a few questions if you're interested just raise your hand and please exit the room quietly okay I said quiet, please thanks for the talk do you think there's something missing something you haven't do you think there's something missing some tool you some process that is still broken and some tool that you feel like you haven't written yet so I think the question was I think the question was is there still something missing even with these probably the biggest thing I notice missing is probably testing and a lack of understanding around things like semantic versioning so it is quite common the way that we do release of version package management in OpenStack is we use constraints we say this is the upper bound on every package in OpenStack that you can use so talks cannot not talks but deprecations cannot exceed version 2.0 and at some point we will decide to bump 2.0 up to 2.1 and quite often for random packages 2.1 removes a feature and it shouldn't because 3.0 should be the feature removal version so I guess lack of testing and lack of understanding but that's softer issues that I don't think libraries can really help you or that much with if you know how to do semantic versioning properly and you remember to make sure that you bump your version when you remove something and whatever then I think this gets you 95% of the way there to what you actually want I'll just repeat what you say so so the question was for tools like PBR do you have the ability to get the version elsewhere in the project instead of it being statically so from within the Python code itself PBR provides a couple of essentially wrappers over package tools or something one of the setup tools sub projects that will give you the package version all the information about your installed package the only problem with that is that it relies on the package actually being installed so it won't work for example if somebody just dumps the tar ball out somewhere instead of pip installing it if you do that I'm not actually sure, I haven't done that with a Python package for a long time so if not thank you very much enjoy the rest of your conference