 Thanks and thanks for coming everybody. So the title of the talk is smashing up PyTest, CoveragePy and ASD.py to take TDD to a new level. Let me say just quick things about me. I've been a freelance programmer since beginning of my career, like 20 years ago. A couple of times I took responsibility for the whole project and hired a few subcontractors to deliver it. I chose on one project I chose Python for the delivery. It was in 2008 and I didn't have to touch anything else since then, fortunately. At the moment I have five Python subcontractors in one office in Bratislava working for one client on a long-term project and I'm looking for more clients. Okay, I would like I have a little survey, so a survey time. So how many of you did write at least one automated test? Cool. Who has tested you longer than two minutes? How about longer than ten minutes? Okay, so some hands. Okay, eight hours. Okay, two hands. Who is getting a broken build too frequently? That's cool. Who's using nose? And how about PyTest? Okay. So these are three user interaction limits. A hundred milliseconds is the limit for user to feel the system is reacting immediately. One second is the limit for users flow of thought to stay uninterrupted, even though they will notice the delay. And ten seconds is the limit for users until they think the system is broken and start doing something else. So computing adapted. Developers have done a good job in making it all quicker in recent years. There is very few activities in today's computing where you have to wait for more than ten seconds. But how about executing a test suit? It takes minutes or hours. It's 50 times slower than most of the other computer tasks. It brings intensive load on the computer, delays with other tasks, fans screaming. So what are the consequences? Developers hate waiting more than anybody else as executing tests, tests interferes with the workflow so much. It's no wonder that under pressure or distraction, some developer doesn't run the test or doesn't notice negative results and commits a failing build, which makes the lives of other developers more difficult and sometimes starts a downward spiral. Broken test suit means error lifespan increases. The developers valuing, using and maintaining test suit the most are most punished and waste the most time. I actually think that the test execution time is the single biggest flaw of automated tests idea as a whole. But how about running just affected tests? Majority of code changes are local, so it's a waste to run the whole test suit each time. There is, of course, a solution. Most of us have used the developer thinks I'm changing just this module, so let's just execute the related tests. However, it's quite cumbersome and reliable. Good luck being correct in this hand picking when the dependencies look like this. And also, one of the properties or one of the purposes of test suit is to discover a failure which you didn't think of being able to cause with the change, right? Influencing something you wouldn't think of. So let's explore the idea of affected and unaffected tests on a very simple project comprised of one Python file. Let me have your utmost attention and please have a look at this grandiose project and try to grasp it completely. For those who don't know, Python discovers and runs any methods which are called test underscore somethings, so this constitutes a valid test suit, including the code under test. And here we have a grid of dependencies between tests and methods of particular project. Whatever you do inside the subject method body, there is no chance you will influence test underscore add. Have a look at the source code again. You can hack all the inside subtract and test underscore add is not going to be influenced. It never calls subtract. For test underscore add to start calling subtract, any of the methods it's already calling have to change, which will trigger execution and will create an updated dependency metrics. So back to the dependency metrics. Out of six positions, we have four crossed ones, so almost all of them. But on bigger projects, the ratio is going to be much smaller. So there is a lot of methods that can be changed and only influence small ratio of the tests. Maybe you're suspicious about how could we track the metrics in dynamic language like Python. I remember feeling the same way when I, when I'm hearing about coverage reporting for the first time, I thought that it would be fragile, slow and unreliable. But no, it's very good. It's stable and widely used project. Creating a metrics on the slide is just a little addition to coverage by itself. It has the same limits. Obviously, at the moment, it doesn't work across technology stacks. You cannot track execution of C++ code or JavaScript triggered from Python. It also doesn't track data file changes. If that's input of your tests and it changes execution path, you'll get wrong results. But for the circumstances where it works, it works very fine. Now let me show you a tool which automatically executes only the most effective tests on every file change. Three tests. So if you want to be evil a little bit, you can join two methods. So it was screencast so that it doesn't go wrong on the presentation. And look how it went right or everything went as expected. So the idea transferred into a tool is Testmon or PyTest Testmon resides on Testmon.org page and GitHub. Let's go briefly through the libraries which it is based on coverage Py. It's a giant which allows all this after some initialization and executing of this code snippet. We get the file names and line numbers of executed code. Coverage Py is mostly used as command line reporting tool but the features of coverage Py which Testmon uses are almost documented and almost part of the API. It doesn't sound very good but really there is just one or two undocumented attributes used and also not Bachelor who is the author of coverage Py reached out when we started with Testmon that he would like to know if there is any obstacles and probably also fix them if something makes problems. And also, Ned would like to add to coverage Py the functionality to feature to track which tests or which methods are actually executing which lines. So it's exactly the information which we are tracking and this way it could converge or there could be a joint effort. So after executing each test with coverage we are getting file names and lines of codes triggered by the tests. From there we need to get methods which are executed and ASD from standard libraries could tool to do that. That was my first contact with syntax tricks when I implemented Testmon. From the name you can imagine what they are the syntax trees. I would say they are much more they are not much more abstract than other things in programming. So for me the word abstract is actually quite distracting. Testmon only needs to parse the Python source code and understand it enough to know where the line boundaries of methods body are. So ASD library is a little overkill for this but it's ready and it works in all versions of Python also future ones and I was glad to learn the basics. If you would like to be if you would be interested in learning ASD a little bit more I recommend the two resources there to study on the slide. Obviously Testmon is a PyTest plugin so far. PyTest seems to have a more active community recently and also here I was surprised to see more hands raised when you ask about which test runner you use. But I know and I have the feeling there is a lot of projects using GnOS and not switching anytime soon. So I'm interested in also porting Testmon to GnOS and if anybody has experience in doing some advanced stuff in GnOS I would be glad to talk and maybe we need help with that also. One interesting aspect of the whole project of Testmon was that I thought that this is a valuable tool which many people could use and it would save them a lot of time so I asked for money on Indiegogo and I didn't have any followers or blog or any other open source tools I used before or developed before so it was really difficult to get the word out. But still the modest amount got collected so that was a nice and inspiring thing and I would encourage you if you have done something good for the community and have a blog or followers and would like to dedicate your time to do some valuable tool don't hesitate and use this route. I think there will be more of it in the future. So a little sneak peek into the future. I hate the way the tests used results are presented on the console. I hate scrolling on the console and whittling through the stack traces so I was thinking about some better way to represent the errors and I think the best way would be or the best I can think of so far would be to present the errors inside the text editor you use. So something similar to a linter for example and this is a screenshot from atom. It has a very good way and easy way to add a linter and some code checker and in the combination with the interactivity of testmon where you can get the results really quickly I think this is the way for the future to make it as a linter. Of course it's not specific to testmon. This would apply to any testrunner but I think in the combination of the quick feedback and good presentation would be great. Okay so this is the conclusion. Testmon is awesome. Use it please. Give me feedback, share, tell your colleagues and so on. This is contact again and it's time we have enough time for questions so I hope there's there is many of them. Please raise your hand if you guys have questions and we'll get the microphone to you. Come on, don't be shy. Okay. Hello is there any way to force some tests to run all the time? For example I have a test that tests database trigger and I wanted to run all the time. Can I do this? Not yet, not yet. It doesn't have many features so let's see how it goes and it's also kept it's been kept minimal so that it's easy to study and to prove the concept really works and then adding features is like the next step. You show testmon running kind of as a process that monitored a bunch of files but can it also produce a report that I can process on my own or does it have an API that I can use to tell me how source code lines map to tests so that I could use that information myself? No, not yet. I guess that's the like that's the addition to coverage that I mentioned. Like because if you have some I don't know what would be maybe can you can you give him the microphone back? What would be the use or maybe yeah we talked about it but maybe you can tell the use. The use case I have in mind was for my lightning talk yesterday the the mutation testing right one of the assembling blocks for that is it takes a long time to run all the tests inside this these loops so if you can determine exactly the tests you want to run based on some you know git diff or something like that then you can drastically reduce the runtime for these kinds of things and make them practical so that would be where something like this could come in really handy for that kind of work that's why I was wondering and I can't see how I'm just trying to I'm trying to envision how I could use this to to make to make my stuff work better. Yeah well I said there is not many features and there is no API so but it's really really a small tool so it would be easy to add any of those okay at this point. Yeah hi how do you test test them on? Good question it has a test suit but I haven't there is a problem in calling coverage by recursively right so I can have a test suit but I cannot use the plugin itself which is a little bit of a bummer but but yeah there is also a solution I could on on the parts which are which rely on the coverage the next feature would be to test not to add would be to manually specify methods and tests to manually specify the the dependency so that even if the tracking is lost in the recursion it would still work. Does it track changes that you make in PIDO test fixtures as well so if you update the fixture will it detect that as part of well if it's data files fixtures if it's some JSON and like it was jungle fixtures right then no that was one of the things I mentioned that it doesn't work when you know across across technologies text and it doesn't work when you have data file inputs or some inputs from external services so ideally if you would have a test suit which constructs everything in python code also the fixtures then it's then it's then it's a python code and it's tracked execution and could you could you go back to the little example you gave with the subtraction addition stuff yep so what you did not show in the little screencast was what if these subtract methods gained a call to add how would it deal with that so you add a new dependency basically to to one function right how would it know that would it know that after running the test once or when would it notice if you add a call of add method to subtract method right the subtract method is called by two tests right so all of them would be both of them would be executed the test subtract and test both yeah the test add wouldn't be and it's now a new dependency in subtract it wouldn't be and it wouldn't gain the dependency also right so there's a new dependency because subtract now calls add and the but test add doesn't call subtract okay got it okay my fault thanks also that's a good remark like we've been using the test on on the project and most of the bug reports have been like this fortunately so so we found out that test one is right which which is quite quite a good surprise because because yeah I also was afraid that it's gonna be fragile and and stuff but the biggest problem actually is that tests are dependent in dependent even if you don't know about it there is some fixtures which you know that doesn't get re-initialized or something like that so then you get failures because test mode you know always runs some other dynamic subset of your test suit which doesn't happen then many in any other circumstances so so that's like the biggest problem in adopting okay next question if my test calls a helper function sorry if my test calls a helper function and the helper function changes will it notice yeah because it's a part of the tree yes no it's not part of the tree it's part of the execution well the the syntax tree or the ast parsing is only used at the end of the whole analysis and it's not that important that the important thing here is that when you run the test and you the test calls the helper function it's in the list of executed lines and executed methods then of the of that test it's in the dependency matrix it will appear as a dependency so that probably answers florida's question which was about pytest fixtures not about data files about text files pytest fixtures in other words it was a question about staying in the python world not a question about jason fixtures yeah that answers the question yeah i think fixtures are are created by a method all right so so it's gonna get tracked um does it how does it restart the test runner as in it does it restart the entire process or does it because testmon is implemented as a pyto test plugin right so how does it restart the test run when it tries to run new tests as in how does it affect a session scoped fixtures or do you restart the entire process or exactly how does that work well i i'm not sure about the session session scoped fixtures it really depends on the changes or are you asking about the runtime itself like i can about about the fixtures set up and tear down basically so after it's run a couple of tests will it tear down all the fixtures and restart them again next test or does it i'm not sure this might be well not tested yet i'm not sure how it would behave so that a session fixture is you know always run if i understand currently always run just once in the beginning of the test execution right um and if you change anything in there that probably it's not tracked because the specific test doesn't execute any of the session fixture right so that wouldn't wouldn't get caught wouldn't get registered as as a dependency do you understand correctly that if a function does something nasty for example change some global value and other function depends on it on this global value so we kind of can broke can break the function which you haven't edited and we the tests will not run for the the other function for the old one it this is quite difficult to explain but you know if that evil function changes some global value it will appear in the dependency metrics either it's called by a test or it's not for every single test you can say that and for the tests which do not call the function it doesn't matter what the evil function do you know the world can explode in that function but the tests are not executed the tests do not execute it so for example if if my function depends on the state the function function a depends on the state and then i have function b that starts to modify this state and makes in runtime makes my original function break how well that's either either this is a test dependency right because the evil function gets called in test one and then the test two relies on that on that on that global value so that's what i called test dependency and i talked about that would be a problem but if there is you know if the tests are independent and the test two doesn't rely on that value and resets it again or doesn't use the value it doesn't matter it gets registered or or it it works as it should okay that seems to be it thanks