 Please welcome Maximilian Wiener. This is working, yes, nice. So welcome to a practical pie builder talk. Yeah, I'm Max, obviously. I'm on GitHub. I don't use Twitter, but I do have email. So if you want to get in touch, feel free to send me a lot of emails. I do speak German and French and a bit of English, so you can use that to get in touch too. I'm from E-Mobile Scouts 24, which is the leading German real estate portal. So if you want to find a new apartment or move to a new house, we have services for that. And while we're mostly a Java-based company, so most of the web development is done with Tomcat and Spring and so on. So really Java-based, although we do have a few teams who actually use Python for web development. These are the cool guys, but where we think that Python really hits the spot, that system automation and infrastructure as code. So we do everything in Python when it comes to automation. So what I actually do is I'm a system developer. We're in the production department, so we're operation sets and so on. And we automate stuff with software development, metrology, so we test things and so on. And the most high profile tool we're working on at the moment is what you can see here. It's called YAT. What you see here is the YAT shell. Basically, it's a tool for systems orchestration, which is a lot like Ansible in many ways, except that YAT shell is older than Ansible. And the difference essentially is Ansible is orchestrating systems over SSH based on rules like this playbook that you have to write. And YAT uses the fact that our entire software platform is built on top of RPM packages. So everything is an RPM package. Configuration is an RPM. Software is an RPM. And YAT builds a dependency tree with all the packages. And from the dependencies between those packages, it's able to infer what needs to be done in order to update the server. So basically, the idea is unlike Ansible, where you execute a playbook, with YAT shell, you have some hosts and you just tell updates and then it does everything that needs to be done in an automated fashion. And there are a lot of things for error recovery too, so it can heal itself and so on. It's pretty cool you should check it out. So we scout as a company. We're not a small company anymore. So we have like 700 employees, about 150, maybe 200 in IT. So that's a lot of developers. What you can see here is the amount of virtual machines we have on one of our data centers. It's like 600. In fact, we have more like 2,000 in total. And what the slides basically tell you is that we're in environments where there's a lot of stuff going on. So we have a lot of developers and lots of software. And in that kind of environment, the worst thing that can happen to your software, being a software developer, is legacy, which comes in many forms. Personally, I think legacy means you can touch it. So maybe it has no tests. So you don't notice if you break it or maybe it has no documentation and you don't know how to run the tests, which is just as bad as having no tests at all. And basically you want to prevent that. You don't want your software to become legacy. And so what you do is you use the Python ecosystem, which really provides a lot of tools to avoid writing bad software. Test frameworks, linters like Tarek, Zjades, Flake 8. These are all really cool tools that you can use to avoid your software from slowly rotting to its death. So that's one thing to keep in mind. You really want to enforce all these best practices and use as much of these tools as possible. But the challenge basically is that we as a team decided that we want to empower our colleagues from operations. We want them to use Python instead of things like Bash and Perl because we think these things suck. So we want them to use Python instead because it's simpler, it's more readable, it's easier to get productive with it versus a rich ecosystem. And so the challenge we face essentially is that we do want these colleagues to use Python and we want them to use best practices so that the software does not become legacy. So they should test it, they should not write on the scripts and so on and so forth. And so the problem is you can't show up and tell them, yeah, you're gonna use Python. This is very simple, it's a cool language. And then I have this huge list of things that you need to know and understand in order to write good Python code. So you need to know how coverage works and you need to use it, you need to know how Flake8 works, you need to use Pylons and so on. And it's really overwhelming from a beginner's perspective. So basically PyBuilder is the solution to that problem. Sneakpeak, what we're actually aiming for is you check out a project from Source Control and you CD into it and you have a virtual length with PyBuilder installed and basically you just say PIP, which is short for PyBuilder, and this starts running so it does a lot of things and when it's done, everything that was important has been taken care of and you know if you can check in or not. So you run one comment and you know whether your challenges are good or not. So that's what we're aiming for here, what you can see is it's running integration tests, it's running unit tests, it runs two linters actually Flake8 and Frosted, it's measuring the coverage, steaming it sufficient. So really it's fully automated. How does it work? Well basically like every other build tool under the sun, it splits the builds into tasks. So you have these small building blocks of logic that have maybe dependencies on each other and you run PyBuilder and tell it what tasks it should execute and then it makes a plan with the dependencies and the required order and it does that, but that's still not enough actually because in that slide you still need the knowledge about the tasks that you want to execute. So that's still something you should have to document in a wiki or something, we don't want that. So the solution basically is in the PyBuilder configuration file, you have this default task thing here in the middle and that's what PyBuilder should do when you just run it without tasks. So when you run PyBuilder by default in this case it's gonna run clean and then analyze and then publish. And that's what we use to make it just work because we want people to just use one comment and then they know if they can check it or not. So what I'm talking about until now is mostly should sound like make probably. So that max you say, first already make and make is much better than PyBuilder. Yes, you're right. I think the strength of PyBuilder lies in its plugin ecosystem because basically everything PyBuilder does is a plugin. So take for instance coverage, it's using the coverage.py API. And the cool thing about these things being plugins is that you don't just measure the coverage and it's not just you run the build and it tells you, oh, the coverage is 50%. It's using the API so it can take decisions based on the coverage and you as a developer can say for example, yeah, I don't want the coverage to sink below 50% and then automatically it's gonna break the build if someone like removes all the tests because the coverage is going to be 0%. And that's something that gives you the guarantee as a developer that people are following the best practices that you deemed necessary to contribute to the project. So another good example is we have all these linters built in so we can use Flake8 or Pep8 or you know, whatever. And if you ever contributed to an open source project in Python, usually you make this contribution there's this pull request. And then one of the guy with Comet writes comes up and says, oh yeah, this is full of Pep8 errors and I'm going to just make a comment on each of them and you're gonna have to fix them. Well, when someone contributes to one of my projects like PyBuilder for instance, I just direct the people to Travis and tell them, yeah, look at Travis, it's red because there are some Pep8 errors. Please fix them and when it's green, we can merge. And the cool thing is he can run the full builds on this machine because Travis is also just running PyBuilder without anything specific. So one of the other highlights I guess is that we use a key value based configuration. So basically the idea is if you can understand English and read it, then you should understand what each of these lines does for instance for the first line of configuration here tells PyBuilder to run the integration test in parallel. You have two lines like here that tell PyBuilder to ignore some specific frosted warnings. Also like you probably noticed, this is Python. So we think it's pretty cool. A lot of people have criticized setuppy for being a Python file because people actually abuse it in pretty bad ways. Personally, I believe that with PyBuilder this is not a problem because we don't have the same problems as setuppy. I don't really have time to get into details but essentially it has to do with the fact that setuppy is packaged when you do a source distribution and the PyBuilder file is really just for development. And when PyBuilder packets something it actually writes a setuppy file. So you can be pretty sure that the file is going to be extremely clean. Another thing is we have requirements built in. So in your PyBuilder config, this build.py file, you can actually write what your project depends on in a declarative way. So you can also separate between normal runtime dependencies and build dependencies. So that's pretty cool. And one of the nice things I think is because PyBuilder is using pip under the hood, you can put specific pip configuration inside your project file. And for instance, like here the last line, you can tell pip, yeah, for this project you're not going to use the public pipy server but instead you're gonna use our local installation of the defpy server. If you don't know defpy by the way from Holger Krickl, you should check it out. 2.0 just came out. So you can do all this instrumentation stuff on pip. And this is really something you can do with requirements.txt because requirements.txt is just a new line separated file with requirements and you can't tell it to use another index just doesn't work. So this is the last slide. I still have a bit of time. So I'm going to switch to a shell and show you how to get started with a project. Maybe this will also answer some questions. So I'm going to a temporary directory and actually I don't trust the wifi here because it dropped a few times already so I have a virtual length which is already loaded with everything I need to show you. So the most important thing is that pi builder is installed. This is something you should always put in a virtual length because it needs to see the package in the virtual length and I'm going to make a project and name it EP14. So basically now I can run pi builder. It's going to fail because it's not a project yet. So we have this nifty little tool which is called start project. Yeah, let's start start of course. And this asked me a few questions. So mostly the idea is you can just press enter all the time because it's going to suggest defaults. So we have like this default project structure which tells that the sources go into source main Python, the unit tests go into source unit test Python, the scripts go into source main scripts and so on. I'm just going to stick with the defaults here. And the scripts, this is I think pretty important to note the scripts are separated from the source code because there was this talk yesterday about building awesome comment line applications where the referee basically said, yeah, scripts are like vampires because they live forever and personal observation in most cases, they also look like shit. So we don't want them to be in the sources and really keep the scripts to a minimum. So that's the idea. And now I have this project and I can run it and it's failing because basically I have 0% coverage, I also have 0% sources. So I'm going to disable that. And again, a key value based configuration. So really it's easy as saying set property and the property is called coverage break built first documentation for that. And I just set it to false. And now I can build and it's going to pass. So it alerts me that the coverage is actually really low but it doesn't break the build. That's a choice I can make as a developer. So now what I'm going to do is I'm going to create a script. Yeah, no, a script, bad, but this is the easiest thing to do right now. So source main scripts, I'm going to call it EP14 and start with a Python shebang. User bin and Python. You should also all the time use that shebang unless you specifically want Python three or two because this works with virtual length two. And I'm just going to print hello. So I'm going to run a full build again. And what happened is it built a source distribution and this target folder. And what I can do is simply install it. Target just, so in case you didn't know you can pip install target sets, which is pretty cool. I know, looky-looky, I have this EP14 script which has been installed in my virtual length. Just like that, unlike with setup pie, I didn't have to tell the tool that this is a script. I just dropped it in the right place and it just works. So now I can pip install it. So next thing I'm going to do is move a bit of a script into a source file and write a test for it. So source main Python. Yeah, let's start with a unit test. EP14 tests, I don't know. From unit tests, import test gains. So this is boilerplate. Note that you can also use a pie test or nose tests or whatever. So if you rather use asserts with a pie test then it works too. And I'm just going to write a simple test that asserts that, yeah, I don't know, assert equal. I'm going to call it, yeah, not EP14 but like EuroPython and I expect this to yield a string which will be EuroPython. So I also need to import it. So now I have a test which is obviously going to fail because I don't have the EP14 package yet. So I can check it out. And it works because I named the test wrong. Yes. So there's a convention for test file names too. And it's a glob and the default is EP14 is a star wildcard underscore tests with an S. So I have to name my files like this. This is also something you can change but again I'm just sticking with the defaults. And yes, no. So I run the tests and it says, yeah, it can't import EP14. So I'm going to create that module and drop in and source main Python. So not where the scripts are. And I'm going to put everything in an initpy and basically it's just this function EuroPython that's going to return something else right now. So return, I don't know, none because I want the test to be read. So I can run the tests and you can see I didn't have to tell PyBuilder where my tests are. I just run it and it just works because it knows where the tests are. No, the test is read, I have to fix it. So now I can return like EuroPython and now it still works. Yes, so basically the problem I have here is I called my package exactly like the project. So essentially what you see here is that the package conflicts with actual source distribution. So I have to fix that too. I'm just going to call it lib probably. So I have to rename this in the test too. I know it works. I have 100% coverage, which is not really difficult. And yes, but that works. So I'm mostly done with a demonstration. I think I'll take questions now. Maybe you have questions which are related to what I just did. So feel free to ask away. The question was if we have Sphinx integration, not right now, no. It's probably just a matter of writing a plugin because I think Sphinx is largely automated so you don't have to do a lot of things. But the answer is no, not right now. Yes, just a plugin. Yes. So in our company, we use Tim City, which is basically Jenkins with better CSS and it costs a shitload of money. But we have colleagues that use Jenkins with it. We also use Travis with it. Essentially the cool thing is that when you decide to use a CI server it's most of the time it's difficult because you have to find out what your project dependencies are, how to get them on your CI server and so on. And with PyBuilder really you just run the PyBuilder comments and it's exactly the same as on your workstation. Yes, so the question was how the reporting works. Basically you have two choices there. One of the things you can do is on most CI servers you can see the output of a comment. So you have the output of PyBuilder which tells you what's wrong. The other thing you can do is you have this target directory and inside this directory there's a reports directory and there you can see, for example, in JSON formats or plaintext, for example, if you're interested in coverage you can see only two statements and 100% coverage versus also the same thing with JSON. So you can reuse that. Yes, so on most CI servers also, actually I only know of Tim City but I think Jenkins can do it too. You can like upload artifacts after a build. So what we do with Tim City is just put up all these reports on the build so that, you know, if it fails you can look at, for example, the coverage or the flag 8 and you see what's wrong. Does that answer your questions? Any more questions?