 Thank you guys for coming for this talk. So today we're going to learn something about talks mainly but Before I learned it's actually a tool which has the bold aim to actually standardize testing inside Python And we already have tests you may say everyone heard about pie test But if you like a lot of the one of the guitar repositories for example for the pie to source with itself You see that by down there. There is a touch that any file meaning it pytas itself uses stocks Or you can go it if similarly does that or there are other tools such as for the example request also does Similarly uses stocks So if you come and start want to contribute to this project you might get the question So why is this talks all around and what actually this purpose is? Hopefully by the end of this session you will have a good understanding of what stocks tries to do why it's needed And what can it do for you? How can you use it how it works? Okay? So let's start with the first question why it is needed Yeah, just to brighten up the mood ever so often I put in a few pucks here and there so That's for that. Okay, so let's start with like generally you have a your Python application. What kind of parts Python application nowadays has so first and foremost you have your business logic This basically boils down to your actual source files This is what your manager writes for you to actually or pace for you to do Okay, then we have the test which are basically for yourself So you can actually ensure that whatever your manager asks actually doing whatever you wrote Then you have some packaging because usually whenever you create an application or even like a library You actually want to package it and deploy it to the where it will run or to the other users So you need to have packaging then you need to also have documentation Because someone else when comes back still need to understand the project then you have nowadays also Quite popular type checks for example my pipe. I try to actually check if anything's okay or not in your source code And then you have also like stated code analysis. This is think of tools such as flake or pep8 or piling and Finally usually when if you especially if you're writing the library This is true. You want to support that library for more than one Python versions so that everyone can Be happy and start using it and therefore you actually want to test that you whatever you're creating Works under multiple pythons occurs and then there's another Another dimension which is mostly used for example the Django world where you have multiple versions of Django So if you write a Django plugin you want to ensure that it runs with all types of Django versions Okay, so these are basically Complications which means that instead the like beforehand you maybe had to write one test now You're that one test instead now you have like basically 10 tests or 12 tests or similar a lot more tests which you want to basically run and Additional thing which is all usually nice whenever someone comes to your project and they want to like quickly Get started with the project. They don't want to read two pages of how to get started developing They usually would be nice if they can just have a single comment which they can use to actually set up the entire developer environment, so to say and Obviously you want to ensure that all this kind of configuration and all these kind of Steps and all these kind of checks in your application actually pass after each commit So the next people who come and start contributing something to your project It's actually you can start with a clean slate and doesn't have to first appear whatever is broken in the project Luckily, we live inside the great ecosystem of python This means that we have a lot of people who are eager to open source their solutions for various solutions for various problems And what this means that for example, we obviously have a tool for everything for test We have pie test knows test for packaging now this is quite for libraries there is setup tools fleet or Poetry but for applications like pex or ser or things which are now this quite popular Or the documentations both things and make docks is widely used inside the python ecosystem As I said before and the static analysis tile checks like black Piling flint also make it Then we have all the ways we can create parts of a virtual environment Which basically have to test multiple python versions and then you also have we can use this also to test the virtual Dependency types for example, you want to test again Django one and Django two in the same time Okay, so now basically You can set up for your project use one of these tools and make sure and you use one of these tools and basically Once you use that tool everyone can be sure after every comment that everything runs Okay, but the downside is that now basically you introduce to your project 10 tools and Everyone who wants to basically just maybe Contribute even a little might need to actually on all those 10 tools and understand how they need to be used to actually Be able to work with the project. Okay, so This is not really a Good place to be because basically now you mean that whoever comes to your project You will have a really steep learning curve to actually join the project and actually understand why all is why all these tests are Needed and making basically you will have some really sad users when they want to contribute to the project because they Will not actually understand why something for example my field some checks my feeling in the CI For which he locally did not fail for example stat Quite common example is that for example He messes up the formatting or some static checking and even to locally test past It will still mean that the CI will broke on the remote and this generally leaves Contributors needing to come back all the time to whatever they started to create so The idea would be that it would be nice if you could somehow Bring down the bridge or build the bridge between the CI and the local development environment And somehow we could basically be possible to run the same thing that we're on the CI run also locally. Okay, so As I said beforehand GitHub Advertisers creating the contributing file which basically means someone has to read it and understand it and then actually use it Then you have the option you may probably at one point you will get fed up with Copying around comments. So you will start writing shell scripts to actually contain all this kind of test Or if you're really hardcore you go and write a make file, which basically does the simp. It's more like an advanced shell script. Okay So here's just a good example how one of the shelves could look so you basically you want to basically start with Just checking that if anything fails stop don't continue and then basically start to packaging your project installing it creating a Virtual environment running your test and if everything wants all these comments are run You probably everything will also pass on the CI side. Okay now As I said beforehand this in this example It's only for one python which is like a system python would be nice to add virtual environments Because what if you need to support both for python 3.7 and 3.6? so also You want to run all checks regardless and actually print maybe at the end of the report because they're on profile here I did not end that I did not actually address the point of maybe would be nice in the end we would print out all the things which failed and because that's what the usual actually will need to address and Would also be nice if you could actually run this in parallel Because basically a money many of these tools can run in parallel because they're not depending on each other okay, so as far as going to actually testing multiple pythons, you can use the virtual amp or the VM which is added to python core libraries and Basically, you can create your own lightweight Side directory and then keep installing your package into it We this you can use basically by activating but you don't have to actually activate it if you really want you can and you can Spell out the full path you can use the full path to actually run Things inside a given virtual environment. Okay, obviously this now requires you to actually remember. What is the pet for virtual environment or? This basically Once we had this just for one virtual moment this kind of our best trip now becomes this 10 lines which makes All still does not actually do any reporting when also just hand there's one single python versions. You probably want more than that So one can see that this point that this kind of writing this kind of magic scripts Which run every test basically every kind of tools starts to be hard to actually maintain It also looked error prone because if you can do anywhere and type of the other if you want to add another environment It's like you have to type a lot of other it probably will Incorporate a lot of copying around so this sort of seems like There must be a better way with this and And what we want is happy users. We want them to be jumping around in our project So let's try to do something like that and this is the the talks of dream Talks was actually started out by the same developers who started out pie test They envisioned to the testing the test runner more or less to be talks and pie test to be actually the test frame the testing framework, but you can they Isn't those to be the tool which actually drives which People start interacting with Okay So how can we achieve merging the CI and shell based testing? Okay So the idea is that we should be able to run locally everything which we run on the CI This is all especially important because in anything breaks on the CI You want to be able to reproduce it locally because that's how you that's where you can actually For example start debugging interaction understand what's happening. Okay? So Talks is basically a lot for a Python project to define all tools in one place Meaning that all kind of tools that you may use you may add checks for it You can define it in one single configuration file with their actual configurations Furthermore you also have a central way of invoking them when we talk about central way of invoking them think of it like make targets that's probably the best approximation or generally just build targets and Finally should be ensuring should be creating an environment which is isolated and reproducible So to say so when you are and then it should be and by isolation. We don't just necessarily mean virtual environment You also mean for example shielding you from whatever you have in your user operating system environment Okay so It is basically a CLI tool meaning if you actually to actually use to talk you don't need anything You just need Python pip and then just install it and Then you can invoke it with for example pi to minus and talks and that if the user should work it and should be able to run All the testing without actually remembering. What are the comments? Okay? And it basically manages creates and runs the repressible environment So this is what talks actually does for you it basically will ensure it has the has a look at your configuration It will try to recreate your old your recreate a Testing environment for a given target exactly as you defined it. Okay, so let's see how it actually works This is like a flow diagram which shows what talks actually does it initially has like a simple configuration phase So this is basically just putting together everything you have in your configuration file And whatever you passed in from the command line plus whatever may be inside the operating system environment Then there is does Some of the steps are optional So you don't have to do all the tabs for example the packaging is an optional if you have a library You also want to for example package your library because when you want to test it You actually want to test the package version whatever the end user would use not what your local file system contains. Okay, and But as I say this is basically packaging so you can also use it for your applications So once the packaging is done it will for each of your targets it will Define a virtual environment and this virtual environment will face of a three phases It will have an environment creation install phase and finally it will have a run comments face and Once all the comments have run it will have a reporting. Okay, so this is for example a simple configuration I'm not sure how good it is try to do me Shows a bit better. So for example here is a simple project. I defined the Three environments for example one of my environment is basically just running the test on their pike in 2.7 For example here, and then I have a linting a documenting and a publishing target also this target For example the linting because you can use a spiling 2.0 requires python 3 plus So I specify to you like python 3.7 my documentation does the same because maybe inside my conf pi I actually want to use the fancy f strings and all that things So it is possible to specify that for the documentation is a different python environment or a different version and then I also Defined in the end a publishing for example, I don't have to actually remember how to publish everything. I go just define there that For me publish basically means have an environment which has both setup tools and wine and then just basically run The setup pi as this baby still and then use twine to actually publish it Okay, so this means that now basically whenever I need to do publish I don't have to anymore remember whatever I just call the publish target and it will work And it will create for me and make sure that I have the correct twine versions and that kind of things, okay? The first thing which I have there in like the first time when I see embolist that is basically the ambulance Which will run always so the user doesn't specify what should run it will run those environments in dent order more or less But as I can see things which I don't need to run always I can define it as an additional one and they are for me to be run as on demand For example the publishing is something we probably don't want to do it after every comet So that's something that is not actually defined inside the default environment so to say which are run by default. Okay So Yeah, yes, you can see each of them are like separately defined They have their own section where you can define not just the dependencies You can also define their actual Arguments that you want to pass on to those kind of configurations and this allows you for example You can now have all your configuration in one file because most of the tools provides command line environment flex and you don't no longer necessarily need to have Custom configuration files laying around like five four in your project for each a different kind of project. Okay For example, this will be the actual documentation building and you can see it even generates a Unique build directory for it will the user can actually then see the output if they want to see it and Yeah, now, let me Okay, so Let's go the next one the packaging so the packaging is basically It creates an installable package. This is the only that's when you have a library the build dependencies as At the moment they are not actually Meaning that from wherever the environment you're actually invoking tools needs to have all the build dependencies now, this is a temporary thing because Long I wrote your long term, but probably is more like in three months or so the plan is that one's people will have PEP 517 and for 518 support we will actually be able to specify our project or build dependencies inside those files and We'll be able to actually ensure also that you can actually install like people insh ensure that you have all your build Dependencies like build dependencies are for example the setup tools that you want to use But also like additional ones which are next actually doing the build but are required to do the build for example setup tools SCM is one of the popular switch you can use to like automatically version your project. This is also like Build dependency, but it's not actually a build. Okay So The next step environment creation It basically wants to be optimized meaning that if the environment already exists you already run the comment beforehand It will not recreate it from scratch. It will reuse it Now as I just you can remember beforehand like here I had here at the start section where I defined what kind of Python I want so this means that there must be a way somehow the application or the tool needs to figure out What pythons do you have on your computer and what actually interpreter where it lives? so for this one we actually use the System path on linux. So basically we try to see if there is a python 3.7 executable in your system Okay, obviously this doesn't work on Windows on Windows. You need to use the py exa for this kind of Interpreter discovery. There are two peps which is basically detail how Python discovery should work on different platform basically like a Unix and Windows and the tool follows basically those two's ideologies and We use virtually of the create virtual environments for the given version and Then at the end of it there is an install dependencies meaning that you basically we just whatever you specify You can specify explicitly dependency one of the things we install the dependency specify here This supports whatever pep 440 supports meaning that we can also specify like requirements takes the file They're basically just get forwarded your installer, which by default is peep but for example someone can use peep amp or whatever other they might want to use and Then there are those other dependencies after we install the project dependencies or the ones that specify this for the tool Then there's another set of dependencies the project itself and this also means everything specified inside the Insta requires you'll also be you installed in the second phase. Okay Now there is a slight Yeah, the difference is a peep but can be changed real tools Okay So once you installed your project you have it available in a fresh environment Just as it would be installed basically at the user end if they would use it You then the following things you actually run your comments. Okay, so for this kind of comments for example here We do tea things or three things. They are tea comments to be run. The first comment is basically Let's just build the documentation. The second thing is basically Let's see if all the links are valid inside the documentation And then the final one is just print out for the user where he can find the documentation and view it. Okay One of the important things that we do for running the comments as that we basically strip away most of your operating system environment variables And this is quite Helpful in a lot of the cases especially when you're relying on some environment variables or your project has some environment variables Which drives its functionality. It is often the case that you set it locally But then forget it that on the other users machine that not maybe set so this ensures that you can already fail on this kind of Environments not set case locally and not have to actually wait until other users reported that by the way on my system That doesn't work. Okay, and basically the chain of comments to run it one after another and whenever one fails We stop and if any of the comments fail that the environment is marked as being a failure Okay, and We Stream basically whatever your tool puts out on the Output this can be configured basically but by default we stream out this allows you to actually see What the tool output was so you can then take a take action based on that? Okay So and the final thing is we just printed nice report For example in this case you can see that two environments failed Basically the test and the linting but the documentation generation finished with success Okay So basically this is like the workflow of tux. This is what it does you it basically ensures you that you have a virtual environment Which is as specified inside your configuration it automatically sets it's up it will package your project It will in the end Allow you to actually print out whatever happened when you run it the idea is that for example If some new user joined the project They don't have to say like so how do I run the test basically they can just invoke tux and that will print out And that will do everything for them and basically have a local CI if they need it Okay, let's see some use cases or how it actually works in practice Okay, so for example, this is a default angle you can see that this actual column might take some Time because this is creating it on a fresh state you can see that we have three Targets we have basically the same the testing the linting and the documentation phase And we will start creating this phases one by one once we just invoke the tux And it will invoke pythons with the correct argument It will ensure that the environment has been created and whenever any error happens We immediately displayed for the users so they can act on it. Okay now the thing is This usually takes a lot faster the reason why it now it is a bit slower is because basically we have create operations there And creating a virtual environment installing dependencies because network is involved and I was involved is not as fast as many of the things So at first time it could takes longer But for example in a when you invoke it the second time then basically we don't have to do anything Then just re-invoke the comment that you have because all the dependencies has already installed and are in place Okay So it is obviously possible that you can run also just substitute environments For example the cases you might only want to test what actually modified the source code for example documentation generation You don't really care what you actually developing or for example another popular use case of this is that you want to run your if you have for example testing against two Python environment and You want to ensure that for example you run the test for Python 2.7 and 3.6 and then you want to have another step after it which basically Calculates the aggregated coverage report that you can do we are this kind of way because you can specify that run these two Python environments And then display an aggregated An aggregated coverage for those two, okay Okay, another thing that Is quite nice is that you can actually document whatever the targets are so the users don't have to actually guess So you can add descriptions to it and then the user run for example, this is for our own Toxin if I'll basically when someone says okay, what can I do with this project? They can just list whatever the possible targets are and they can read if they want to run any of them or not Okay, you can see that there are default environments, which I'll always run and then additional ones which are on demand basically another option is that you We support the pause arcs So basically if you testify for your comment that you want to pause our guns and this means that if you use the double dash and After double-dash you can basically forward arguments to our underlying tool This is very helpful for example in this case I wanted to run the test under Python 2.7 and if anything fails I wanted to enter into the debugging for example PDB Okay, so I did not I could basically just pass them on to the invocation After the double dash as additional arguments and they automatically get forwarded This is also a nice way for example You can have more verbose test reports and all this kind of things Which are basically more like you want to understand more deeply what something broke and not just run the default test It's so to say Okay Another option Which we have there is basically we have another a project which is like detox is basically there to run Things in parallel for example, it is possible to run all the environments in parallel and then display an aggregated output Now obviously the limitation of this is that we can no longer stream the output in real-time Because if you do basically you would mean that five environment five tools running parallel and all five tools would Print the character on the screen. It would be quite hard to actually see what's on the output. Okay So another quite to you can actually create a developer environment When I say developer environment and this is especially true for libraries It means that we don't actually install your package as the user would install it but we install your package using PPS in developer mode, which basically means that people not actually install your package by Putting it into the site package but creates a sim link and say that the source code lives in your wherever you're developing and then the Whenever Python is running it will actually pick up your local files and not the remote another site package files. Okay Okay Now just one thing once you start creating this kind of environment or generally it is quite healthy for example in your Environment you can quickly select one of the environments to be the one for developer Ecosystem to use for example in this case for Python There is this pymb managed plug-in, which is quite nice Which allows you to basically just right click on any of the environments We may talk for any other virtual environment and say but by the now change the Interpreter package to this Python, which is inside this in this folder Okay So another thing gives is quite popular is like obviously Especially like big projects care about 45 Python versions you might not have all the Python versions locally on your machine So it doesn't make sense to force failing those environments with you. You don't have pythons. You can specify that only you run environments, which For which we can actually find the correct Python version for example don't try to run Python 3.5 if that's not on the system because it will fail But it's not actually a failure that the test field is just that we could not we have dependently missing for it Okay Another helpful thing is and this is especially helpful for like Bigger projects or like a Django project is that Django projects Historically have like a lot of versions and you want to check a lot of Combinations and ensure that all the combinations actually work It allows you to build equally specified and you can see in three lines all that kind of Django combinations And there for example, you can see that for example the Django master actually you can specify even like github links And it will install the latest from github and try to test it again that okay, so as you can see basically we just 10 lines of configuration now you can test against like 12 combination of Dependently that your users might run into okay now obviously you might not necessarily want to run all this locally when you develop But this can work something in the CI and whenever something blocks in the CI You locally you don't have to manually recreate the environment. You can just say by the way on the CI I saw that the pi 37 Django master field Let me just recreate that and run the test only in that and try to debug what actually is the underlying issue for that Okay So CI integration as I said the whole concept of it is that We would prefer to not be to be CI agnostic meaning that we want the CI to basically don't have to actually Specify all these the CI should basically just invoke the same thing as the user would locally basically just invoke talks And this means that your CI configuration if you use stocks beside the user actually having the option to easily set up Locally any kind of environment now the CI you don't have to use the property CI environment But you can basically just say that oh by the way make sure that I have these pythons in that in that CI and then finally installed in with Install for example my talks and then just run talks and that's the entire CI configuration Okay So Let me sum up a bit what I Try to present here today and what we learned So the first thing is obviously there are weaknesses of all the tools the biggest fitness is or what people often get confused is that dependency updates the way people works if you say to peep that install peep 11 and Unless you specify that by the way, please make sure that you upgrade if you have any things the package is not there If you'll actually not install it this will say I have peep it doesn't even check what version it is It says I have my version peep is already there. I do not actually evaluate that you wanted peep 11 So that if I you need to specify the dash Capital you which basically means to upgrade and then it will force make sure that it Upgrace whenever installs that peep version another thing It is which but this is not the true for example dependencies For example, if you install a library and the library's dependency is already there people not actually upgraded and because we use peep we have this kind of Mistake on our side or like this kind of vulnerability on our side, which bottom line falls down to that whenever you add new dependencies to your project Talks will not actually upgrade your or pull in your new version of the dependency it will add new dependencies But we'll not actually if you just bomb the version requirement. It will not actually ensure that the bigger version is installed okay, so The work around for now is actually just force the recreating the environment You can pass in the dashboard flag and this basically means by the way you recreate the environment from scratch and that ensures it the medium plan goal for that is that once peep will actually support Installing or like having dependency graphs and evaluating that kind of things this should be also solvable okay Another thing is that the build dependency I said beforehand this probably should have some solution in the next few months and Let's see what strengths we have so what we can use basically you can make sure that you can add new tools to your project and To do users can run those projects locally without actually needing to teach them the new tools Only if they actually want to run those or like they actually want to debug those new tools Then they have to learn about it It allows you to run your test it under multiple Python versions without having to manually create all the virtual environments for the Python versions Plus it allows you to run locally exactly what would run on the CI Okay, so this means that if and as I said before like if anything breaks on the CI, it's easy to reproduce it locally and Finally, it is just a Python package meaning is runs on any environment So forever your contributor or a user may be it's easy for them to get on board because now they for example Compared to make they don't have to install something which only works on unit system okay, and It has a hopefully a readable Configuration instead of some ugly only configuration, okay So finally one thing which we have like similar as py test has we also have a plug-in system So if you want to do anything which doesn't Work out of boxing talks you can do it by having the plug-in for example, you can see that They are plugins out there and quite popular which basically swap out The Python setup I think for example to using CMake to build in the project or use the run you running ducker as a To as an environment where you can have virtual environments So ducker is also kind of like a virtual environment So it's easily to plug in those kind of things, but there are a lot other things for example There's also in development the package You should allow you to use conda if you're more like a conda developer and Use the environments from there Okay, thank you very much, and now I'll take your question What tools do you have any thoughts about SQL integration or integration with databases? Or is it just a purely Python specific thing can you just write your own tool to go off and populate your postgres database with Test data, etc. And how do you extend it? I guess the question is the broad question Depends on what you're using. I would say probably the easiest what you're speaking about is basically I suppose database dependencies I Would say probably what you're looking for is you can set Create different kind of fixtures which set up different kind of databases and then you can specify for example That different environments generated use different kind of fixtures which set up the database and That could I think fairly reasonably work and you can do like that for example if you want to test both with Oracle and MySQL That would make it possible I'm not sure that you could probably also do it without having the texting direction You could probably also just have parameterized stress in PyTest Depends I think it's personal preference which you prefer I Thanks for the talk Would you know if during the environment creation stage? Would you be able to specify requirements.txt file or something like that? Yeah, I feel like in the dependencies you can actually specify both the requirements takes the or constraint takes the which is another Thing which we introduced as opposed to what's in setup.py. Sorry as opposed to what's in what's specified in setup.py So setup.py is only required if you have a library Usually for libraries, I think the best practice is to use setup.py to the specified dependencies Requirements takes is more popular for applications like the requirements takes were developed for applications If you have an application, then you don't have a build process You only have the dependent environment creation in that case you can specify the requirements takes it easily Just pass it on as instead of dependently I think I think at one point you mentioned running things in parallel Doing multiple environments at once. I don't remember seeing talks do that before. How do you it's So at the moment it's not built in meaning that you need to run detox Which is also lives under the tax organization, but it basically is just a driver which Allows you to run multiple text environments in parallel. It's an higher abstraction So, yeah, it's not built in but if you look up detox should be what you want Will detox go as far as running a couple projects in parallel or is it still strictly one project at a time? It is one project at a time mostly because your different projects might have different configurations That means say there is no real reason why You we could not like have parallel Configuration parsing and running. I don't think anyone actually raised the need for that So strongly that they actually wanted to develop Thank you for that Just a question you mentioned you can make a variation on the jungle dependency in your project there In our case, we have dozens of dependencies Usually we ship with every dependency has been fixed to a specific version and what is interesting is to see for Many of those can we upgrade Because they're fixed in time and we realize three months or six months later You know, maybe we should upgrade to the latest and not be stuck in the past and we want to explore the Versions we can now use How could we leverage that kind of tool to explore how we can upgrade our dependencies without breaking completely? Because sometimes they you know, they collide and it doesn't work anymore so So in my view that would be a Requirement that is fixed to specific versions and one other requirements that would be more flexible and say okay Yeah, I mean if you can see this example, for example, you can see that Django 1.8 is actually fixed to Django 1.8 But for example, the master is actually master. I could easily just specify Django latest which would be like What I have without any dependency specification and I need to install the latest. Oh I Mean technically speaking you could do it with similar like like this kind of factor expressions That because we called in factor expression. They're at the top They are put there even like the dependencies you can put it in So it allows you to do even more deeper kind of this kind of specifications But just a word of caution if you have like 10 dependencies, you're talking about factorial of 10, so I'm not sure you would actually Run that in real time probably usually like in practice what we saw is usually People want a few major things and maybe over latest and then when that breaks the according to that you may be altering your Like for example specifying that by the way our Package no longer works with because it broke on the old when I installed all the latest It broke with this packages latest So I'm just pinning that to a given version until someone takes up the task of fixing what why it doesn't work now with the latest Because otherwise you just have too many the space is too big and even if your test run five seconds You're already looking at like Probably next year when one of the running finishes Okay, any more questions, so we don't want to wait the next year, so we have break now and have lunch