 Hello and welcome to my talk about PyTest Pac-Man. Generating Pac-Man fixtures for fun and profit. So, what we'll discuss in this talk is I'll introduce myself, I'll introduce the project, the Pac-Man database layout, the differences between the different databases it has, what PyTest is, fixtures to do, and a conclusion. So, I'm Jede van de Wa. I joined Arcinix in 2010 and recently, three years ago, I am now an Arcinix developer. I'm involved in the security team and the DevOps team, and I maintain several Python projects in Arcinix, such as the Pac-Man bindings in Python and the Django-based website. So, our main website is written in Django and the ArchSignal utility and more. So, why did I start this project and what is it? And I started it basically a month ago or so. I had a problem. I took over the maintenance of the Pac-Man Python bindings, and I wanted to create a proper test set for it. And I started relying and to test it, I need a local database. I need, and basically I need a normal ArchSignal. I need to mark this environment. So, I used Pac-Man for this because it is, well, it is possible to add and download repositories and then install this all without root permissions. But, well, this adds the dependency of the network. It's slow because I have to download a packages database. And I wanted to circumvent this. So, I tried to generate the local database format. And it turns out this is pretty easy and also doable. And so, I created a small script to do this, then I turned it into a Python fixture and finally it is used now in the Pac-Man bindings. And then I thought I could reuse, it would be awesome to make this reusable because more projects could use this. For example, ArchWeb, which is the website, also uses, it reads Pac-Man databases to import the packages state into our website and display this. So, why don't I make it by this plugin? And that's how it started. But first, I will introduce the database layout. So, what we need to mark here. So, Pac-Man has a local database. This contains all the packages you have installed. So, and it's really, really simple. So, as you may have noticed, there's this fairly Pac-Man directory and it contains a local directory. And this includes a special file and this file is the database version. So, it was introduced in 2013. It's currently version nine. And when this was introduced, you had to actually update, upgrade like your database. And the current version nine hasn't really changed. So, the version isn't bumped with the ALPM. So, the C API version, which I think is now 11 and this version is still nine. So, this local database, it contains a directory per package, which includes, so the package version and the package rel and also an epoch, if that exists. And it contains, it can contain four up to four files, but two of those are optional. So, the first file is the description file. This contains the metadata of the package. So, in S-key, you have these strings in uppercase, which are between percentage delimiters and they contain like the name and then there's the value, which is just a string. If you want to have multiple values, you just append the value two times. So, if you have the multiple depends, which directly translates to the depends in your package build, you'll see that here on the slide. Then there's the files file. This contains the installed files and just as a list. And then there's the backup entry and this is exactly what you specify in your package build. So, it is an array of files, which requires special behavior when upgrading or removing and they contain the file path and a hash of the file. Then there's the install file. This is the install script, actually, which you have exactly specify in the install is directive in a package build. This is not really relevant for testing as as of now, I'm not testing any installs or upgrades. Then there's the entry file. This is also not really interesting for testing but I added it for clarity. So, this is basically contains a description of the package archive. So, for example, if you take the Linux package, then you look at the, you open, it's a gzip archive. And if you look at it, you'll see a tree of the package with any files with hashes of the files and permissions. And this is an archive format, basically. The next thing Pac-Man has is these sync databases. They are located in the Farley Pac-Man sync. They contain the, yeah. This contains the repository databases and these are synced with the Pac-Man minus capital SY command and this syncs all the repositories you have defined in your pac-man.com. Then there's recently this files databases. These were added in, I believe, Pac-Man five and these allow you to search for a file in the sync databases. So, everything you have synced, you can search and see which packages contain, for example, a user bin echo. And these are synced with Pac-Man minus FY and both of these files are, you can see them on your mirror. So, if you go to your mirror repository, you go, for example, to the core directory, you will see a core.db.tar.gz, because that's what RGIS is as their compression format. However, this can be except or is that standard, for example. And locally, these files are renamed to core.db and core.files and they are both gz. Formats seem to be, well, this contains all the made data for which packages are in the core repository and it's similar to, it has a similar structure as the locally be, so it just contains package name, package version, package file, and then directory and then a description file. This description file uses the same format as the local database file. And the same goes for the files. db is also uses directories, packages, and the files file, which contains the local, which contains the files to look for. True. So, now that we have all this knowledge, we know how to generate these kinds of files. So, what did I do? When I started adding tests to the Pacman Python bindings, I started with PyTest because it's one of the best testing frameworks for Python. It contains fixtures, so you can give a function to a test function you have and this can be set up some data for you. For example, a database, MySQL database, for example, or an HTTP client with your REST API, key, et cetera, configured, or some random data you need in the test. Or, so the fixtures allow me to create the required files I need for testing. So that was nice. And then PyTest also has this plugin framework, which can extend the Python runner, for example, so it adds coverage reporting to it. And you can also, well, the plugins, you can do a lot to extend. So, you can also share the fixtures. So this was something I wanted to do because I wanted to have the fixtures I created in the Python bindings repository. I wanted to share these with other projects where I could use them. And if I may create a PyTest plugin, it will allow me to install it locally, so in usually a Python tree as a package and PyTest will automatically pick this up as a picture, which it can load. And it is really easy to create one, so you simply add an entry point to your setup to the Py and ship provides the required files in your PyPy, so you basically provide a library with fixtures and then you're good to go. So currently I created two fixtures, so this is similar to what is now provided in the Python bindings for Backman. GraphStory, so I have a fixture which can generate a local database. So this fixture is a function and one that can generate a sync to be and this is also a function. So both of these functions, you have to pass a dictionary with packages in a specific format and then it will simply generate them. So the uses are following. I have a JSON file here in my repository, my test setup, which contains key value, so exactly the fields in the description file, so basically name and then the value Linux and then the version and then what is it for, 5.8. I create a fixture out of this just for convenience. Then I create a new fixture which contains the, which is my test local database. And this generates the generate local database function, so I return this fixture and it creates a local database in a temporary location. Then in my test local database, I provide both the test data and the local database. And I simply assert here if the required structure is created. So this is a bit of a silly test, but it's just for demonstration purposes. So this works. This is not included yet. It's not yet finished really, so there's still a lot to be done. Currently there's no, it's not published yet on PyPy or as an arched package. So there's still my to-do list to do soon, hopefully this week. I still have to then make use of it in the Python Bindings for Pac-Man project and there's a lot of need for documentation. Of course, how you can use these fixtures or how the data you provide to these generate functions should look like. I want to add more tests to the repository itself and I want to add more flexibility. So currently it can create a file to be, but it cannot create a file to be. It cannot create a file to be, which is something I don't necessarily need yet. Another thing which is missing is the sync to be with the required zipped extension. So for example, if I want to make core.db.tard.gz, it doesn't allow me to do this yet. I need this for Artrep because it reads databases like this, but I'm not sure yet if this is something the plugin will provide as this is also something you could do in your own project. Something I have to determine. Then I want to integrate it in Artrep as it now uses hard-coded fixtures, which are pretty, which is the whole point of making this my test plugin because I want to generate fixtures, which is a lot more flexible and easier to do than hard coding things. And it also allows me to create, to test more scenarios. So in Artrep, for example, if I want to test, if packages get upgraded correctly in the database. So for example, if there's a new links version and I import a new core database, I have no test for this at the moment because I have hard-coded one database and I would have to hard-code or basically store two databases in the project. So this isn't really manageable. Yeah, that's it. So the conclusion, yeah, it's really easy to test. It should be really easy to test anything which requires some sort of Pac-Man database in a Python project. So if you're looking for less, this should really help. And the source code is located on our kit lab if you're interested. And that's all. And thanks for everybody involved in organizing this event. It is really sad to not see you all, but it's awesome that there's still an online event.