 Hello everybody. Thank you for coming. This talk is going to be about Testing building and releasing add-ons and trying to automate it somehow. I Thought that maybe it could be interested for interesting for you because I've started working with blunder just two years ago mostly developing and helping blender kit with their add-ons and I came from this background Where I've studied like fine arts and then worked for a while as a quality engineer in in red hat So I'm not like Extremely experienced coder, but I like testing somehow and When I first met with blender I I had this Strange feeling that it reminds me of Python, but it's not like Python like pure Python. There are some some catches and the environment is a little bit Strange in a way and when I started when I started Searching for ways how to test test add-on I've realized that like setting up unit tests and like other kind of tests Is not gonna be the same as when you are testing just Classical Python module or something like that After two years, I have figured some ways. Hopefully and I'm not sure if these are correct because I found that like sources are limit limited and I Found just just a few like threats on the blunder Blunder developers forum and In the other sources, so this talk is gonna be like my sharing of device how we do it in in Blender kit and also I would like to Show you some of the github actions We've set to maybe help you with just basic Basic tasks on github if you are using github for for development of your atom So first thing I will show you is Really a simple github action for automating the builds The second one will be like expanding it a little bit more to automate the release process and then We will look at the vase how we use unit tests and Then we will inter-corporate it into github action Yeah, so one one thing or like one question could be why to automate anything or Why to test anything? I know that Probably I have this feeling that there are two kinds of developers of Blender add-ons one one kind of Developer is this 3d generalist who somehow started to be interested in like Creating add-on so these people come from the 3d world and another kind of person could be somebody who is more interested interested into programming and Python and She he is just exploring The the beautiful words of Python in inside Blender And I know that For some of us or like for most developers it could be quite boring to to write tests and we all love to develop new features and deliver new new stuff and Like the automation some sometimes is like takes a few hours or few workdays to set up and we Instead of like spending this time on automating it we choose to like develop more and more more features Which can be a trap some sometimes so Yeah One one thing we know is that like beautiful is better than ugly but also like explicit is better than implicit and I believe that when like we We we've write our processes in code we make them somehow explicit In the beginnings when I started in Blender kit We did some priority Pre-release and at the time you are just like bundling the zip file manually and we did this terrible terrible mistake that We zipped it in in a wrong way, so I don't know how many of you could make if if you If you want to deliver the blender add-on and you have multiple files You should you zip them, right and when the user unzips the zip file should there be a Directory named in the same way as the as the add-on or there should be files directly who thinks there should be like just files Without like being in the directory Will I'm great another person and who thinks that it should it should be like in the Blender kit directory or like My add-on directory and then in this directory there should be files Yeah Yeah, I'm not sure you know Every time I think about this question. I just download some add-on which works and I take a look, you know but like as you as we are like releasing Every every beaker every month or like once in a year It starts to be a repetitive task and that some some day you will just think like yeah I remember, you know, like I do it this way and then you deliver Pre-release in better case or just the release to users and somebody finds out that it's broken, you know It should be in directory. I think So Being explicit is better and like trying to automate things because like once you have the build file Like or the script which builds it for you then you can be pretty sure that It it's the correct way, you know, unless somebody created the evil people request, but you should check your pull requests. So before merging Yeah, another another reason might be that that should be just one way one obvious way to do it. So I will later show you Development script we use Yeah Yeah, and I want I want to like Share this talk with you because I'm not I'm not Dutch. So I know that it can be It might be obvious so Yeah, I'm I would like to show show more people like what we have found and maybe we can talk later about Maybe better ways Yeah, so let's let's know like close the presentation and I will skip to visual studio code. So I have Some pretty Simple I don't it's it just basically does nothing, you know, it just like says hello on register says buy an unregister It's a set it's I've set it in in tool Like more than complicated way because I have like this other module called nice Which actually does does the stuff and then there's another sub module code and panel, which just creates a button and then and panel and And there's some another operator which prints another Text so it's pretty basic, you know, it's useless basically, but the structure it follows The structure you might want to use in your in your add-ons Because sometimes you don't want In most cases if your your add-on is a bit more complicated, you don't want to just have one huge file You sometimes you don't want to have multiple flat multiple files in In what in one directory? Sometimes you want to create a sub directory or something like that and structure the code a little bit more So this this structure follows like The structure you might use in in in like medium-sized add-on sometimes It will be like later Important when we look at the unit to test things So This could be your add-on and you could be just creating the zips manually As we want to I wanted to like help help With the with the bills And make it as fast as possible to create automated builds. So we have created a Get-up action called Blender add-on build You can check it on the Marketplace On github and I will show you how to how to set this basic how to use this action to To create automatic builds on pull requests and merges to to your main branch So I'm not sure how many of you like our no, no, no, no Blend no github actions or how many of you are using actively like github actions Some of you. Yeah Nice Yeah, so And we want to create a get up get up action. We first need to set up Sub directory called to the github and in this subdirectory we need to set another Sub directory card work flows. Yeah Better, right? Thanks. Thanks So We will have a subdirectory github the github and then then another subdirectory work flows and for for the beginning I will I will use this simple built file Which I copy from from the the the the repository which I will share with you in the end I Should probably like rename one of those so it's like Not sure how to do that Okay Let's let's live with that Okay, so so I I've added the simple built file and this is pretty basic basic Get up get up workflow There's some name and then I define situations when this workflow should should run It should run on on pushes to to our main branch that's It's like after managing PR or if you are like committing directly, which is a bad idea then It will it will run on every commit to any push to main branch and then I want to also like to to run this this action on any pull request Created in your in your repository. This this really helps because like when when somebody else creates pull requests to your to your agent or When when you are just like creating a pull request for yourself You can see the see the zip zip in there and you can like download it and send it to to the reporter of the Buck and you can get the feedback quite quite fast Also, there's this work workflow dispatch, which means that you can trigger it manually That can be handy also sometimes and then there's definition of this of this job Which is quite easy. I just we named we named the first job build We will run it on you wouldn't do the latest That's the runner available at on the GitHub actions And then we define steps and there's just one step It's it's build Adam and it uses the our GitHub action blender. I don't build In production you probably should like define a stable version here, but for now I'm using like the latest at main and then I Just specify how how I want the artifacts to be to be named at the end in this in this case I define like sample item this get up action also supports excluding files, which is a good idea sometimes you can see addons which Have like all the other other files inside by mistake So it's Like what I'm doing here. I'm just excluding dot gate I'm excluding that github because I don't want my workflows in the addon. It doesn't make sense I'm excluding with me MD and probably I should I could like a exclude Get a gunner if I if I had any Yeah, so so this is the This is the action and I will just name the commit some some somehow nicely and they will push it to to main branch Which is a bad idea, but Yeah, and this as you can see it runs and now it will Fail as on live demos happens every time Yeah, I got lucky Yeah, so and I have here I have this the build addon so I can I can download it and And use it in blender And it's like that the first first thing you you might want to set up if you are not using any Automation or you are not building the addon through any any any script That can help you also This is this is great because it makes life a little bit easier, you know Now I have the I now I have the bill just with the one PR if this was a pull request That would be just like the one or two commits fixing the problem and I can easily Send send a zip file to to do a user who was reporting the bug Yeah, I I think that Like when when when things are easier we tend to Like develop more and do the stuff sometimes we can I just think I will I will do one one or two features And I will send the fixes tomorrow, you know This this helps me a lot Yeah, so one Like if we want to expand it expand on it expand on the automated builds a little bit more We then could use the GitHub action for for releases Which just basically does that it builds it uses the build build action and then it automatically creates the the GitHub release it It can be also like if you're really really think more often This might this might help you a little bit more and once we I will copy it again and show you Yeah, so so this This workflow is a little bit expanded It it's workflow with which is just being dispatched manually because we want to to set the Releases when when when I when we want to to make them so it does some some inputs the inputs I use are the Yeah, there's one required Input which is a version and by default it's one point point zero point zero Important thing is to to adhere permissions to contents right because otherwise the get up get up action Wouldn't be able to create a release on the on the GitHub page And then there are there are two jobs one one of one one job is like the build we have used before And the second one is bit is the release release job Which has this definition needs built so it invites for the build build workflow to to finish and Then using the the blender and release action. I just defined the artifact name which Should be the same as the artifact name specified in the step before if you are using Your custom custom script for making Builds you can you can skip this this part and define your your own workflow and and build a build The add-on in in the way you need and then just upload the artifact and Remember which artifact name you use and then just define it in here. So This this release action is not tied to to only this this one Get up action. We've created but you can use it also like with with other other workflows For sure Yeah, and then I just defined the release name which which will appear on the release page and the version which I take from the from the inputs the the action has some more tweaks and Preferences which could be set, but these are just like the most simple simple things. So I will just commit it once more Now It will run again because I just pushed into into the main branch But now in under the action step I can see I have a simpler release and I can run the workflow and Create The release for me it will It will take some time Is it gonna fail or not? Yeah, there's still no release because it doesn't like create the release directly It just creates a draft for the release. So this is the this is the draft. I have The zip file named here with the version You can tune the naming as you like but we mostly use this or I like this format like the name of the addon and then then the version Yeah, and now you can just like put the text and and release the addon so These these were like two actions like two basic actions to to help you to automate a little bit of the workflow Yeah, and Now what we what we would probably what we should do now as we Progress a little bit more with the automation in the addon it could be Starting with testing some basic tests For for this reason I Realized that yeah, I will now I will now skip to the to the complete repository Because the coping would take some time So this is the the final repository and I will show you how the things are set it up I use this this file called dev.py which In some cases like in case of Blenderkit and some other agents. I want to have a little bit more control over the Internals of the addon so we use we use custom custom Python code to control the process a little bit more in this do build function that they are just First of all, I'm not like I'm adding just just some Some files which I which I which I want and I also ignore some files as was seen in the In the action before One one difference is that because I want to run tests I have a switch here or like if statements here so I can Choose whether I want to add my test files into the into the addon or not This is a little bit controversial, but because like in in real world you want to test the The same artifact which goes into production or like to your users It could be it could be done I think By creating another addon which would be installed in parallel to the to the to the original belt but For us Like like the easiest way is to just back the test files into the addon When we need them so basically I we just use this switch Called like include include test files which which adds all the tests into the zip file Yeah, another thing which I Which I have in this in this file is this run test function which basically calls the Which opens? new new sub process and Calls blender with with these optional flags. We don't need need any any visual outputs or so Only in We first the blender to to run in background. We don't need audio. I Start blender also with factory startup. So if there are any other Under other addons installed It will just ignore them and loaded the the factory factory Version of the of the blender and then I also Define exit code for situations when the when the Python which I want which I will execute in the in the blender fails, so I said the the exit code to one and then I defined The file with the script which I want to to run in the blender and this test tests that by file Is something which imports all the unit tests? Other than then that In this file. I have just some some basic parsing because I would what I want to do is to I Want I want to have some nice at least nice somehow nice interface, so I Have likes two sub commands one is called Python dev that one is called build. So when I run the build Subcommand, it will it will create the the zip for me and This is probably even better than using the github action because you can also you have the same script Which you could use in in the github workflow by not using the action, but just like running this This script of yours in in the workflow directly, but you can also build with the same code the addon locally, so It's not Two different codes to different processes. It's just one one code which you can run on different environments and resources And another command I have here is the test command which will which will trigger this this run tests Which will call this run tests Function and it will start the start the tests. So how the tests are defined I've put them into the tests subdirectory and Like the the main file is is this one first I'm importing unit tests and then I import addon utils I import addon utils because I need to enable the addon because I want to start the blender with The factory startup, so I need to enable my addon manually And then I create like all manually automatically through the code Then I create a unit test runner Suite and test loader and Once I have the suite I Load the test cases from from another from other files. I could Like you also can have all the files inside this this Script, but at some like If your addon is bigger, it's probably good to organize it a little bit So this example shows ways how to load it load the test cases from another files Yeah, so I have one one one test file called test underscore nice Which focuses on testing the this terribly stupid File in my addon What what it does it basically just checks the checks the the the STD out in in blender and checks whether The the hello and goodbye is printed. So for for that for for this a first call setup method in the test case class and I and I said I said the STD out To shrink how and I also use the teardown method to To like clean after this test case And then I have two methods one for for testing the hello function and one for testing the goodbye Goodbye function Yeah, so Like these tests are like totally basic because like the whole addon is basic But what what took me a lot of time in the in the beginning when I I tried to like realize like how to set up the unit test and stuff like that was this problem of of paths that It's sometimes behaves a little bit strange in in in blender environment, so I Try even though the addon is really basic. I try to show Like how how things could be like structured that we can have like multiple Unit test based on multiple files and how to import them correctly and stuff like that Yeah, so this is simple Test file for the nice module and then I have Test file for the like n panels module of the of the addon and Here I have two test cases one test case tests the the hell word panel and another test Tests the hell word pop up I Think I think that even though I'm using unit tests I found out that just testing Like making writing unit tests in for blender addons or for operators in in in addons I don't have this feeling that these are like really unit tests. It feels more like integration tests or something like that because It's really hard to I Don't know. I think that I need to test the integration with with blender with with the whole environment, so I would so I sometimes Like write the test conceptually as it is these were more like integration tests so for example, I like to Define the or like be in control with the With the flow so I often defined the The test classes with numbers, so I have So I am explicitly saying which which one will be first and then which one will be the second Etc. If you look at the test case, it's really basic first first first method test just if the word panel was registered like Successfully and the second second test checks whether the The panel can be it can be found in the in the Screen areas Yeah, yeah, so how does it look when when when we execute this it's Patent of pay test Oh great So who's the problem here? Yeah, finally it great Yeah Then I will show you how it works in in the github Where it still works because I I didn't broke it one hour before this presentation trying to improve last details Until I broke it all down Yeah, how how to integrate this this setup into into the Into the github What I use often is is this Longer workflow Which Which builds and tests the the add in on multiple versions of Blender Release released versions and also versions which are alphabet our release candidates So this is the workflow I use It's called build and test for now It runs on the pushes domain and on all pull requests and also can be triggered manually and first I build the I Build the add-on and now as I have created the nice def pi File and I wrote my own ways how to build add-on. I'm not no longer I'm no longer using the Build action I'm now directly calling the the def pi file with the build sub command and then I just store the artifacts Store the artifacts in the After the workflow ends Sometimes what's also like good idea is to name the artifacts a little bit more than just Name of your other than zip What what I like is to To name it for example if you build on pull requests You can take the number of the PR and put it into the zip it It helps you with with the organization because sometimes you You send the file to multiple users and it could be a little bit I Miss leading when when you have just like simple as in one simple in two three five You know, it's better to to have the pull pull pull request number in the name So when you open your downloads directory, you see which which version should go to it which user Also on the on the main branch. I like to for example put there the shot of the of the last commit so These zips are like Identifiable somehow Yeah, and then what what I what I do is basically I call I run Another script and this is gonna be the last script. I will show you and This script is called make tests metrics by and it's a it's a script which checks the the blender Resources and checks for the the latest Available versions This file looks like this because I also want to define The versions on which I want to to test I have I start with this this list of Versions I want to test on I like to keep like the first first patch version of NA and in minor releases so so I have like three 3.0.0 and then I have the latest Or like the highest batch version, which is like in this case 3.0.1 but in cases of long in the case of long LDS releases it's three Three dot three dot zero and then I have the latest or like that the highest highest batch number, which is three three twelve Recently I don't want to test on all like patch versions. It would be just like not so ecological So I think this is a good compromise and then in the end in this in this script I I have this small function which checks The build a blender org Downslash download slash daily and it it it like Extracts the the alpha versions beta versions if there are any and it adds those into the into the Test metrics. Yeah, and in the end it just prints this metrics out and gets used by the By the next step and next job in the in the workflow and This workflow then looks like this it Basically, it loads the the the metrics from the from the previous previous Script I've shown you before and then there are just few few simple few simple steps I check out the I Check out the the repository I do a cached download of the blender binary which I I got the URLs in the in the in the script before I Download the blender I Just create some some repository for it and extract it and stuff like that all the boring stuff. Just just like untart the untart the blender and Then I run run the the tests and it's It looks like this. It's just patterned of pie test and then I there's this Flack install at which defines the location where where the other should be installed Which is maybe also the thing I forgot in the demo and the reason why it didn't work locally Yeah, so Okay, finally So how it looks how the test looks? locally The unit test looks like this It ran the six cases. It was quite fast it in the background It opened the blender it started the the test file it loaded all the test cases and executed them one by one and As the the Python script Exited with zero then also the blender exited with zero otherwise it would exit with one and It will it will be shown in the in my terminal and it it it would also like fail the the tests and how it looks on the github I Will open the the final version of this sample I don't It runs on pull requests, but I will show you that I can also I can also run it on Manually so I just choose this billion test workflow And I can choose the branch and I will I will run the workflow and let's see how it goes so one thing it it it makes the build and it uploads the build as the artifact after this this whole workflow ends and Beside that it also runs this make tests metrics workflow, which which gets all the versions and or URLs of recent versions of blender and Then in the next step in this metrics it loads load this this list of versions we want to to test and it creates one job for every for every for every version of Blender we want to test Which then looks Like this so this is my final final run for the I My final run of this workflow I have tests for Blender three dot zero zero tests for Blender three oh one also for three three twelve and I'm testing also on Blender alpha 4.1 and and beta for point zero point zero And this is this could be quite useful I've managed to find some problems with with Blender for Point zero point zero and our add-ons so we catch them Like before the official release, which is a good thing. So If you are interested in maybe like expanding or text test metrics or like testing on more more versions of Blender This could be a very or like inspiration for you how to how to achieve that and Yeah, I if you find any problems in my in my Concept or or my scripts I would be more than happy to to get feedback or like email or pull requests because Yeah, I found that the resources for the inspirations are not that White in in this Python and Blender Environment as it is in pure Python for example Yeah So I think this this could be The end maybe if you have any questions you can ask me now