 Oh, he's coming. Oh, he's here. Okay, good. Hi. Does it work? Hmm? Ah, ah, great. Okay. Hello. Thanks for coming and goodbye. Well, first of all, I have to apologize. I have cold, apparently, and my voice is doing weird stuff, so I'm going to sound awkward during this talk. Okay, my name is Sebastian König. I have a little studio in Leipzig where I'm doing a little bit of VFX, a little bit of Blender stuff as a freelancer, so I'm using Blender professionally, so I'm a professional. Maybe, who of you used the tracking tool in Blender already? Okay, who didn't? Who has no idea how to use it? Okay. Well, you're out of luck because even though this is a workshop, I'm not going to work that much, and I'm not going to explain a lot about tracking. I'm more interested in showing a few new features and some workflow enhancements and some tools that I wrote or tried to write. Okay, so maybe first as a little introduction, VFX is usually, and unfortunately, not really about placing robots or spaceships in clips. It's more about removing crap from footage, like the actor had a pimple or something in the face or the crew forgets to put the lighting equipment out of the shot or some crap in the shot. That's very common and that's what usually, at least in my case, what VFX is all about, removing crap from footage. It's very unexciting and ideally, you don't see it in the end. So, if you're going to show off your work and you say, here, that's my work and you say, yeah, there's nothing in it. Exactly, that's the point. Yeah, but okay, so the thing is that for these retouching jobs, there's more or less always the same or at least a very similar workflow. And if you're doing a lot of these things in a row, then you start to think, yeah, this could be a little bit more streamlined. Blender is nice, but it's also very crappy if you do the same thing over and over again because Blender is so versatile and so broad that it's not optimized for specific workflows. So, if you don't optimize Blender for your own workflows, then you're doing it wrong. And I was doing it wrong for a very long time. So, I finally started to get a little bit into Python to be able to make Blender a little bit more streamlined for certain things. So, in my studio, there's a colleague of mine. He's working on a Da Vinci color grading suite, you know, the nice buttons and balls where you can do the color correction and it's super awesome and the software works great. It's really streamlined and optimized for workflows. Like color grading, drawing masks, attaching a mask to a clip, moving the clip around, doing things very efficiently and in a very nice manner. And Blender, if you want to assign a mask, for example, and then you have to go to the compositor, choose the mask, get the mask, and then go back to the movie clip editor and do all that stuff that is really annoying. So, part of my talk will be showing a few tools that I'm trying to work on to improve the situation. First of all, though, a little bit of the new developments in the Blender tracking tool in the movie clip editor. So, I'm just going to demo a few things. For example, how you can not open a shot. Oh, yeah, by clicking the right buttons, you can come to this panel and you can open up footage. For example, this one, for example. So, the first thing is the new full-screen mode in Blender, which is very nice, thinks Dalai. So, everything is gone and that's really nice because you're not distracted. And then, the thing is, let me go out of that again, starting the track, you always have to do a couple of things like prefetch it into the memory, set the number of frames and all that kind of stuff. So, you're always fiddling with all these things and all these buttons and they are all over the place and now we have the tabs. So, you have to go into the tabs and search for the stuff that you use. So, this is so much nicer because you don't see anything. But thanks to the new Pi menus, thanks to Psi-Fi, we can just do that and have a very clean interface and if you want to use a tool, then you have your Pi menus, at least if you have to write them yourself currently. So, the functionality is there, but it's actually quite easy to put tools into the Pi so you can use it from there. I've been working on these Pi menus for tracking since a few weeks already and I realized that optimizing workflows can take a lot of time by trying to optimize the tools really for certain workflows so it's an ongoing project to do that. But anyway, so that's Pi menus, it's very useful. If I want to track this shot, maybe let's first have a look at that. There is this thing in the middle and it would be nice to have the geometry. Maybe we want to light a fire there. So, the first thing that we need to do is to track, obviously, the camera. So, now that I'm in this clean setup, I don't have my tracking buttons and I can't remember the tracking shortcuts, but I can remember at least three letters like W, Q, E. I can remember those. So, I have all my tracking tools on these buttons in the Pi menus so I can track this marker now starting from this frame. Apparently, not really. There we go. Okay. So, these tracking pies that I've been working on, they are especially for tracking like really, I think they are very useful because you have to track in a certain direction, track forward, go to frame one, go to frame, go to the end frame. So, the Pi menus, by the way, who used the Pi menus already? You try it. So, the way that these work is you launch the Pi menu and then you can just, while you launch it, you can drag with your mouse into a certain direction and that will then trigger this command. So, I don't have to click there. So, if I know I want to track forward, I just press E and drag in the direction that I want to track. So, it would be, if I go to frame one again, like place, this is such a tiny interface. Yeah, click and drag in the direction. If I want to track backwards, I click again and drag in the opposite direction. I can very quickly place my markers on places where I want them to have, where I want them to be and then they pretty much do what I want. So, one of the things that I also usually do is to switch between certain types of motion models. So, if you have like an easy shot, you probably get away with the default setup if you only track the location of one point. But if you have a moving camera, then usually the feature that you're tracking is deforming in space. And in that case it would be nice if the marker would do the same movement so that the feature is looking the same to the tracker. In that case you can use affine tracking. Sorry. And usually you would have to go to the track panel, do the tracking settings, go to the menu, click there, no there, and then choose the option, which is great. I mean, this is a tidy interface sort of. So, everything is put into little boxes and if you know what you want to do, in that box and then you grab the tool. So, you can find it. If it's a little bit logical, organized, then you can find the stuff. But that's not great for a quick workflow. So, if I know that I switch between location and affine a lot, then I can just put these two options into a pie menu and can drag to the left, then it's location-based. If I drag to the right, it's affine tracking. This is now using, if I do that, it's now using affine tracking. So, I can track this forward. And here it's not so apparent, but if I track something on the floor and set it to affine, then it's going to deform with the footage. And that's going to stick much better to the floor because the feature is very precise, as you can see here. So, it's looking all the same all the time because the feature is being distorted together with the footage. Okay, currently I have to do this over and over again. But it would be nice if I could say, this is my new preset. So, what I can do is to launch my pie menu and say this setting is now the new default. So, I can just drag in a certain direction and now this will be my new preset. It's a little bit too big though. So, I can now track that again. And it's using affine tracking. Another thing that is something that you commonly have to do is to clear a path of a certain track because looking at this one, you can see once it touches the border of this thing, it starts to slide away because there's something getting in the way and this is not what you need. So, I can either go to the frame and then go out of full screen mode, go to the track button, clear in that direction, or I launch my track pie and drag in a certain direction and it will clear the path from there to the end. And this is still too big. So, this is very nice for workflow and it makes everything faster. And once we have at least eight markers, we can then start to solve the shot which usually would mean that we have to switch the tab to the solve tab, then search for the tools, or we just say shift S for solve and then we have the solving pie where we can first enable refinement so that blenders automatically figuring out the focal length or the distortion parameters, whatever we want. We can launch that from the pie menu and then solve the camera. First, we have to set the keyframes and this is also in the tracking solver pie. So, solving the camera will give us, in this case, a solve error of 0.3 which is really nice. But in 3D view, I still can't see anything of that. So, therefore, I have my reconstruction pie. So, I can first grab three markers on the floor and maybe these ones and say set up a floor plane. Maybe track even one more marker here. Quickly track that. Not taking very long. Then after that, I have to solve again. And now I can say this is the origin. This is the y-axis and now I'm going to set up my tracking scene. If I now look through the camera, everything is there. If you want to do this very quickly, you can do this in a minute or so. I can even render that and have a working composite because of the way the set up tracking scene works because it's setting up an entire node setup for me. So, that went well, sort of. Now I could modify this even more so that the geometry also fits a little bit better to the scene. We don't need that. So, we have a floor plane now and the next thing that would be nice to have is a model that looks similar to this geometry. Now we can model that or we can let Blender model that by tracking a point cloud. And the point cloud tracking, we could just place markers there and track those and then solve everything. But if you solve a scene with many markers, then the solving would be very slow. And if the markers are not doing what you need them to do, then it will mess up your entire solution. So, you would have to do it all over again. But if you use zero weighted markers, you can just very easily place markers there. They don't even have to be accurate. They can do something. It's not that important. They should at least track a little bit of your footage, but it's not going to mess up your solution. So, because I'm now going to add many more markers, I'm going to lock these ones so that they are not tracked again. And I'm going to hide them with H and then I'm going to first set up one marker, maybe very small, even with a smaller search area so that it's tracking very fast. And in the current blender the pre-pass, which is a certain way of tracking a feature, pre-pass, it's a brute force algorithm to try to match the marker to the previous frame. This is going to take long. At least if you have very much, a lot of markers, I want to turn it off. And since I'm still ideally working in the clean full screen, I want to do that with my tracking setup buttons so I can just launch the pie menu for that and disable pre-pass. And you notice that it's that the pie menus are very flexible so you can also place your own icons in them and you can make the icons respond to the setting. So if the pre-pass is disabled, it's going to show a different icon then if the pre-pass is enabled. So now it's on and you can see that the checkbox is activated. Okay, and then this is taking way too much screen space here, the marker. So now I'm going to change the marker display with a nested pie menu that I have here available in my display setup. We can just launch a second pie menu and disable the pattern area and the search area so that I only have this little dot which is enough for me because I know that there is a marker. So now I want to automatically set up a bunch of markers and I can do that with which is maybe I should quickly enable the pattern so that you can see something. So Detect Features is something that you can use to automatically place many markers in your footage and that's a nice starting point. However in this case I only need them to be here in this area so I can use the last operator and modify first a few things. So with a lower threshold I have many, many, many markers but I only want them to be in a certain area. So actually I have to do it again because first I want to draw a grease pencil line around the area where I'm going to I'm sorry for that that's unintentional. So I'm quickly going to Detect Features again and then say damn it just do it inside of the grease pencil and I can even lower the parameters even more that's really many markers. It would be nice if it would also detect them here so there could be a constant or a contrast threshold thing. Yeah, so I'm going to add them myself so I'm going to add all these features because I want to have a point cloud there. Okay, so that's yeah go ahead. The thing is I forgot something because all these markers they still have a weight of one and we don't have an operator to copy the settings from the active marker to the other markers. We only have a setting to copy the settings from the active marker to the preset. Well we do if you know Python then you can do everything you want so I scripted my own operator to copy the track settings from the active track to all the selected tracks of course I still have to set the weight so if I do that then all of them have now a weight of zero so I can track them which is going to take a while because there are quite a few and it's not working so that's a little bit unfortunate it can be because Blender sometimes sucks with movie files sometimes it can help to just reload your clip and reload it into the RAM and sometimes it's doing it better but not in this case yeah see it should do it automatically anyway it's quite slow so oh and they all use the pre-pass sorry so I turn this off and copy the settings and track again and now it's faster okay that's going to take a little bit not too much I don't know really it's weird right? it's really strange ah it's because it's master master is just slow there's no magic inside this okay maybe this is already enough so the thing with the zero weighted markers is that you can just track a few frames solve this again and solving is very fast you still have the identical solve error point cloud here and you can use this point cloud by reconstructing a mesh from the markers and I have to hide all these ones but there I have a mesh unfortunately it's just points and I had to search a little bit through the internet to find a script that will convert these at least some of these to something kind of resembles a mesh or like with faces so first I have to rename this yeah but that's crappy I could drop a plane from above and then project to vertex but it's not really working great I'm going to try this oh and there we go I have no idea something that's the oh thank you yeah it's not working that's the demo effect at least it's easy to blame the demo effect for it I'm sorry I can't demo it so imagine you have a mesh there oh you have a tiny mesh maybe we can do it like this just yeah yeah yeah you can still save this I can still do this and convex yeah no convex I can't do cavities or like of these things I have to make a little bit yeah it would work with points only I think so you have a smoke and you composite and you're happy and then everything's fine so that was just one demo of how you can optimize your workflow with Pi menus but then if you can use Python to write Pi menus then you can use Python to do other things like for example I'm going to just open a new file and quickly track one other shot which is this one so another very common problem with the feature detection is that not only does it hang for some reason okay try again feature detection okay there we go tracking many features will inevitably result through jumping markers you see that here and somewhere else and that is very bad because jumping markers will destroy your camera solution and then I mean you can clean this up by hand like going through all these things but then I thought yeah if Blender can if I can see the spikes and probably some math can also see spikes and I really suck at math I'm really like dumb as bread when it comes to math but yeah like just comparing two frames and then building an average between some values that's something that I still managed so there is one tool that I wrote also for the tracking pies is that you can just filter the tracks and erase the bad tracks and then you're left with only clean curves so you can quickly set up a camera solution with an solve error of 0.2 and set up a tracking scene and then you're basically done at least if you orient it maybe okay well that's like you can have a camera track in 20 seconds with composites although I assume that in a few weeks or months we will have this automatically done by Blender itself okay in the beginning I was talking about that VFX means that you have to do retouching a lot so there's one more thing that I want to demo and that is a little bit optimized retouching workflow so let's say you have a shot with some people in it you want to get rid of at least some of them no it's crashing oh no 50 okay so the lady with a turquoise shirt can stay in the shot but the other people should just go away they don't fit there so you want to get rid of them so first you need something to make your camera fit to the scene like usually you do and in this shot even though I was moving the camera this has been shot obviously from high above with a zoom lens or very much a zoomed in a long lens as you would say so it doesn't matter if there's parallax or not you can just use a tripod solver so I pretty much only need to track a few spots in the scene like so and that should go very fast and one of them this one stops a little bit too early because it touches the frame border but I can offset it to another place and then keep on tracking that and since I want to see if the markers are doing the right thing I quickly mute the footage with my display pie mute footage that looks alright then I will set up a tripod solver by using my solving pie and then I also want to in this case I want to set the focal length myself so I'm just going to set it to 200 because this is about the actual length of my lens so I'm just going to solve this and the solve error is 0.4 which is good enough so I'm going to set up a tracking scene and I have to zoom out a little bit and if you have a tripod shot then there's no 3D information there's no way to align this automatically but a tripod motion will stick to anything you can do whatever you want rotate the camera to around the 3D cursor and try to somehow make it fit a little bit to the scene and it's not really that important it just has to fit a little bit maybe like that and because it's just like a tripod motion it will stick to whatever I do with the camera ok so maybe the cube can go into the hidden mode maybe also these crosses are a little bit too big so I'm going to go into the motion tracking panel and make those a little bit too smaller so and what I want to achieve now is to get rid of the people so I need a clean plate a clean plate means a plate that's clean like an image without the stuff that I don't need and I'm going to use the background layer for that so at first I'm just going to align this with the motion or with the scene so in all these places where I have people walking around where I don't need them I want to have a plane that is covering them something like that sorry so usually what I would have to do now is to unwrap the plane assign a UV project modifier oh first I have to subdivide the plane then I have to project this bake it load it into new material and if I do that three times in a row I'm really pissed because it's why I mean it's pointless so ideally you it's not a pie menu but it's in this case it's a button so first I'm just going to project this onto a plane and after that I want to get rid of the people so I'm going to into texture paint mode and can then just paint them away oops and I have to be careful if I paint while blender is animating it's going to crash someone might want to look into that it's not a bug it's probably not even designed to do that okay there's going to be a report then and then for the rest I can just switch to to stamp mode or to another clone tool where I can maybe go into another mode I'm going to background so I'm just going to clone from itself as clone from over there and this shot is I mean it's of course that's very nice for that because it's you can just clone from everywhere because it's looking all the same and oops whatever you get the idea like that something like that and then okay that's good enough okay oh I forgot to load the correct script so I have to yes what I was I was projecting the movie and then I switched to clone from self there's the button clone from image and the image is the footage that is running in the background yeah and by the way this is on my first DVD I was oh no actually it was on tears of steel DVD I was talking about how bad it is because you have to if you have the background images set to movie clip and you want to have you want to paint in full screen mode you can do that because the image is not updating turns out if you set the move the background images to image then it is forced to update so you always have the correct frame I think I did even it's not a bug it's just by design but don't if you have an image an image sequence loaded somewhere as an image data block and you play back then of course you don't want it to update on each frame because everything would slow down so you need something that forces an update of the image data block so if I'm in full screen mode and have no open image with that sequence it's not going to update but if I set the image to background or the background images to that image there's a user of the image that is forcing the update so it kind of makes sense I was complaining to some of you in IRC and you told me no no no that's design maybe even ton I don't know anyway it's working so it's fine I was going to say something oh yeah then of course if you want to render that you would have to modify your note setup and of course you can just do that automatically with python so that you can just render that and it will do everything for you but we are not done yet because I said that I wanted to only mask out the people here and not this woman so I want to mask out her with a mask that is obviously following the footage so I can go to mask mode now and draw a mask around the area that I want to replace everything but if I play back now it's well going to slide away so I could either parent this to just one marker which might lead to bad result because there might be some perspective shift maybe not in this shot because it's tripod but what you can do is to have a plane track that is tracking any plane in your footage and if you use a plane track then it will deform with the motion of your tracks so it will be actually like a plane on your plane in the footage which is to parent this mask to so now the mask is deforming with that plane track and I can even go to preview mode so this is the mask and then you want to feather that a little bit then all you need to do is to well all you need to do is a little bit you have to do a little bit of things you have to add a mask in and you have to combine it with the alpha which is now at this point a little bit annoying because it's not elegant and now you have to multiply that mask like that you have to invert that first like so and then you have to set the alpha and you have to pre-multiply ok but then it's working so then you can use the mask to to do that kind of stuff I have to re-render now every frame to see the correct result because the texture that is masking out these people is coming from the 3D viewport so it would be much more convenient if I could use the plane track to paint on the plane track right Sergei just have a plane track and make it possible to paint on the plane track because it would be amazing hmm well that's a good question maybe you could map it to the 3D viewport like have a plane track and make it automatically be a plane in the 3D view see? thank you ok ok but the thing is that for these kind of shots you also want to have some color correction usually so in that case you have like RGB curves and quick re-render so to correct something here and often you only want to have these color corrections only in a certain spot of your footage so you need a mask again well I made it easy to just press CTRL and to add a mask to the node and then I can double click that mask node and I'm already in the new mask data block where I can now draw a mask around the area that I want to highlight with a nice feather and you would see the result immediately could also invert the influence you notice that currently I have to always switch between the mask editor and the compositor which also is annoying but if I in the mask mode switch with my masking to the image editor then I can see the result here in like here because this is now the viewer node and the image editor has masking so I can edit the mask right there with my effect in the image editor node and that's basically it for this workflow and I think the whole point of this was to try to encourage you to learn Python basically that's the I'm going to release these on the blender cloud as part of my track match blend 2 course I'm going to work on them still polish a little bit but at some point I will publish these tools because I think the workflow can be enhanced so much there's just different workflows for that's a good question why not? I have no commit rights maybe I should ask for commit rights that's not finished enough that can be better, I'm still working on that first it's going to be cloud exclusive right? we have to do marketing for the blender cloud or not, maybe not it's a blender so everything has to be open no but really I mean this is made for very specific workflows the point is that it's very easy to learn this kind of stuff you can copy paste from the entire source code from blender most of this stuff was already there in some other form you just have to copy paste change a few lines of code this is really awesome because I think by default blender cannot be made for specific workflows I think the good thing about blender is that it's so broad and that the interface allows you to form blender into the tool that you need but you have to do it, you have to do it yourself or if you have a studio you have to have like Bartek he would do it for you or ideally you have your own technical director who can also write your own tools speaking of writing new tools in the last minutes I want to show something that Zerge has been writing recently with Kier and that's quite awesome I'm going to just quit the scene so I'm just going to demo how crappy the current version of blender is because it's incredibly slow like for example if you take this shot here I'm going to set the frame range to 800 so it's 800 frames and it's very long and it's also quite shaky especially in the end so this is very very shaky so we need a very big marker and because it's so shaky ideally we have a very big search area and ideally we use affine tracking and if we do all that this is going to be really slow like if I started to track that we are already at frame 4 wow this is only going to take like 10 minutes or so I don't know this is current masses, this is what we have right now that's boring so instead the new version of blender which I don't know when we are going to have that exactly as soon as possible so again I would set this to 800 frames and refat it into the RAM and set up a marker that is incredibly big like really large maybe not so large and it's going to be affine tracking and if I track that now this is like really faster so blender is using predictive tracking so blender is able to look into the future and no it's okay so if I'm here on this frame on the next frame I'm probably there so let's watch over there for the feature so I mean that's awesome so the tracking is actually faster than the playback so thank you and I think the time is over so thank you other questions so you want us to learn python and create this wonderful new as you've shown can you show us your code I have no idea what code for a py menu looks like there's a template and if you go to the scripting layout in the templates there's a uipy menu and you basically have just a class like you would have for any other panel and you have layout and then py everything that's inside is going to be a py menu that's like very easy so in my tracking pies well I have some of my own functions there that's the special tools kind of so somewhere there are the pies so it's just that you have a class for the py menu and then you put your operators inside of that yeah exactly welcome piece of pie thank you