 For me everybody, hi, I'll be shortly starting that people just roll in so slowly I pitched doing a live mocap Session here to the blender community community, and they kind of scratch their head. I show you want to do this I went yes, I just believe that the technology and hardware and software is at a point that it's actually usable I've been Teaching 3d for the past 15 years to students professionals educators Anyone who wants to learn 3d they kind of go with me and they learn along the way So my session here is to just show a real time How would I work in a studio environment or just at home because this is a one-man setup, right? I'll be doing everything while I'm trying to record also the The mocap right so I'm hopefully unplugged in and I'm just it's gonna just magically work And I'll be able to demo whatever I have prepared here for the session So I'll jump into my short presentations. You'll have an outline of of what's going on here So yeah, my name is Bartos and I'm passionate about about how to learn 3d and just teaching 3d Even to instructors or people who I give feedback to instructors who teach and Epic Games hired me to help them on the fellowship and this is kind of where I'm coming from with this whole thing So I help other students and instructors to just learn how to do 3d Efficiently and just get the job done so real-time Technology is revolutionizing entertainment and gaming and one of the most You know things that people want to try out try out is mocap because mocap is so intriguing and interesting to work with And you get the real-time feedback So I usually work in Unreal Engine But today I'll be working in Blender and maybe showing a little bit snippets of what you can do with Blender Going into Unreal Engine if you're thinking about can I get a maybe a raise of hands who has dealt with mocap suits or Markless suits or anything to do with mocap All right, there's a few hands cool. Not too many hands. So that makes me happy. So I won't be Because when I went into this industry, I learned in the past year That this is a very established industry for the past 25 years people have so much experience with mocap Well, I have maybe want one two years of actually diving into this and when I dive into this stuff I go really into it like I I try the most DIY things and up to the most Expensive mocap suits I can get my hands on and I'll try to demo a few of these On videos or just in live, but I prefer to do it live So just a quick introduction This is the breakdown. It's a five-minute session of setting up. I'm actually set up here I'll do a demo of a virtual camera because my main focus is here to show you how to work in real-time In blender because that's something that not not a lot of people do working in real-time, right? The real-time pipeline in blender what it is the demo of the mocap getting set up in mocap and Combining all this in a real-time pipeline, right? You can see here. I'm demoing I mean, this is a picture from a studio that I had the opportunity to work with so I went through Even the most expensive mocap. So I kind of have a large Good good picture of what you can get out of mocap, right? And what are the downsides of these suits and upsides of them because there's you can get a lot out of them But you need to really know You really need to know what you want to get out of these things, right? So I'll start with something very simple. I love this a virtual camera virtual camera by tracky, right? I'll just jump into blender Now virtual camera is an app that I actually absolutely love because let me just quickly connect to my Wi-Fi Now I carry you around a Wi-Fi router because I would not be able to do this on the network here Because it has to be configured with firewalls down. Everything has to work 5g So the suit will connect and also the phone will connect So we'll be doing two things at the same time a real-time camera demo live demo I'll maybe ask someone from the audience to film me while I'm be doing something in the suits, right? so this could be a collaborative thing and Collaborative is the keyword here because I'm inspired by what epic games is doing with Unreal Engine Fellowships and Just a quick question. Is anybody familiar with the term exquisite corpse? Wow one hand awesome. So exquisite corpse just to just to give it a rundown And Actually, it's the best to just show a video, right? So to now me here is a great example of exquisite corpse where one animator knows what fit what what starts and When the frame ends or there's a particular object? I'll just talk over this so it won't get in the way. So basically, this is a great example Done years ago of an exquisite corpse where one animator starts working with They basically agree that transitions is the main focus of an animation, right? So but it can be anything right? It can be an object. It can be a character in this case That just gets totally transformed in each artist's mind, right? And I'm fascinated by this because this is a way to get a lot of people working on one project and Get a unique thing out of it And Maybe MTV does it with low-goes Rick and Morty did this also you can check it out There's there's some examples of the exquisite corpse idea. It comes from the French language I'm not I can't pronounce it but it means basically where somebody takes an image and Continues working over it, right? So this is this is a great example of an exquisite. It just keeps on going going. It's insane It's just keeps on going and going and going and the main theme here is the character that gets transformed into different Positions and the animators work on the transitions So I was supposed to do the demo of the tracky demo, but I got of course sidetracked So I'm just gonna all get plugged in now launch my camera app Be sure I'm connected to the right network. So I just plug in my So I'm gonna be plugging in and plugging out because this thing doesn't have internet So it will be a bit a little bit of a plug-in plug-and-play situation here All right, we're in the correct network. Let's just go virtual camera Just start serving right? So I'm gonna quickly scan my code here. So it should just connect all right, cool, and It just connects to the to the app. I See the view same from here from blender and it should be able to work with just just by scrolling It was really cool. I use this all the time. It's like a second monitor, basically But as except for being a second monitor it has actually a tracking system So if you have a robust iPhone works best with LiDAR Should be able to just move this around enough There we go. All right. So this is kind of where we're at I have this set up here in case the whole mocap suit won't work, right? on the live demo with a lot of iPhones and Just a lot of network these things work best because these are inertia suits They don't know what's going on in the whole environment So if there's magnetic interference or stuff that that can affect it So it's best to work in the place where you don't have these kind of things, but I'm hopeful that it's gonna work So yeah, I'll just play it back here. This was just a simple Way of setting this up where so this is all working in real time right now So I can actually you know start filming it do a shot and all that cool jazz, right? So this was just a quick demo of the virtual camera and I'm gonna try to combine this, right? So let's just disconnect it for now And I'm gonna ask somebody from the audience to hold the camera while I try to perform the the mocap right and I have a set up here Actually, I'm gonna go, you know what I'll try to do it just from scratch Why not because this is a class. I mean why would I just show a complete project where you can go from scratch, right? We'll delete the cube, right? All right, so I like to start with a plane because it just gives me the ground where I want to work with so I'll just pull this to the left screen and Open up Rococo studio here that I actually have already an animation set up here That I recorded before at home So I'm gonna just put this get this live streaming into Rococo And this is the setup, right? You go to the Rococo plug-in It's provided by the suit company and you basically start the receiver Oh before you start the receiver you need to also activate it here Make sure your ports are set up and everything it should just work So yes, it worked it detected that there's a incoming data stream from from Rococo and From here the next step is to work through Actually attaching a prop or or a character so it needs a character to stream to right, so let's grab a character here Have the why but I like the why but because it's just so simple to work with and it just doesn't get In the way, right? All right, so if we have the character set up here now we should be able to select the actor here hit bar and that's my name of the character and The next step is to click auto so that populates the whole bone structure, so we don't have to deal with that which is really really nice and Set as t-pose and when I play back, I think we're misses so yes So this is kind of working right so we're streaming live from an existing animation now you can see that the Processing here is working with footlocks and it's just doing a bunch of things that do not work in real-time on these suits For a reason it needs the processing to get that to get that to get that ahead and So remember whenever you work with these with these it's a good idea to stream it But it's even a better idea to actually save it and then export it as a fbx So you get all the fancy things with like with footlock and you can all the control and precision that you'd leave that you'd like to have right All right, but this isn't fun. I mean, I'm I'm just playing back an animation, right? What if we want to do this in the suit? Well, let's try it Make sure you have a big beefy power bank. This actually works I Wasn't able to discharge this suit even working for hours with it Which was a pretty amazing the pretty robust you can just go on and on and on with this stuff So I'll just plug it in. Oh, we're we're green which is a good news and yes gloves ready. It just connected magically Yes It didn't even work at home. So that's awesome. Um, all right so yeah, I'm a bit mangled but the the way to actually calibrate it is to And I'm gonna need some help here So this is the only part that doesn't that needs assistance right to get the headband out So I'll put the headband on Let's follow the actor here. Oh straight pose Calibration complete and I didn't even put the legs on yet. So they're all wonky. Let's get the legs on So I know So I normally work in socks here, but for a reason I have shoes now All right, let's try again. We'll just follow the actor here Yes, so it doesn't get in the way and just hit control B We're in blender. Yay. It actually works. So Let's just do some tests. I'll walk around and see if how much it actually works And I'm gonna just record it because it's working All right recording So we see what's going on. Here we go actually doesn't work what it doesn't like It depends on the the system that it has to have foot placement. So if I do something like Let's see if I jump on here. You can see what's going on, right? There's a way to fix this right because there's a system where it actually can calculate this elevation But you'd need to know how far you stepped up. So that's one thing to take into consideration Next thing is just putting your your feet up Very based on the fact that it needs that foot placement, right? So if you know what you're doing with your foot, you'll be fine, right? So I'm just recording this I can keep on going and going. This is like a 30 minute take But I'd like to also have this, you know, it's visualized in blender right now But what should I do with this? Right? So You can do a bunch of things right because you can upload the character from From any software like character creator blender Autorick pro works works really well with this system that I might be able to demo here and basically you will get IK here Which would be amazing All right, I'm gonna stop the take Just stop it and that's processing now with the footstep footlocks and just play it back As you can see the footlocks do a world of difference now This used to be really really hard to do and annoying right because it just it just couldn't handle Footlocking back in the day you'd had stuff go in it You'd have things like going all around the place and for the past I'd say three months last three months the innovation of the software has caught up to the hardware here That's why I'm presenting this because you know half a year ago You would have like weird things going on here and that's that's kind of the the it just gets the job done here And I love these this this product in the in the sense where I used to say before That mocap suits are outdated because I just looked at all these AI tools and you could do it with AI I mean AI is amazing right but the problem with AI is that you have to upload it somewhere That's one thing or process it right and that takes a lot of resources and stuff Well the suit here you're in business you already have a take that you can work on and just do a bunch of things on So I'd like to now try to combine the two things that I showed right let's try to do the virtual camera right so I'll be just doing a performance and somebody from the audience will grab my phone and try to record me And at the same time we'll try to capture that camera data into blender See if it will work right Let's try it so While the receiver is still getting all the data here Let's just jump back back into the character mode in real-time and Let's just get here to the virtual camera Connect. Let me just check my Wi-Fi here. That's good Scan QR code Sorry for that just technical technical difficulties. Yes connected cool. All right, let's get that on and connect it Move around the camera Yes, you're moving now. All right. Cool. Try to position yourself just looking at the you should be able to see on the viewport Look around maybe Let's see if I can grab this camera Um, if you could just so I can just move up here Come on. Come on. So the thing with offsets, right? So you're here. I'm trying to just right All right, that's about right here and now hit that little button. You should be able to get control back back Yes, the same button Cool. All right, awesome Oh, but my character died. No, what happened? Let's go back to the character It is actually receiving. Oh, it is. All right. Cool. So I think we're in business Let's just get more frames in here. I don't know like 4 000 frames And so we have some leeway get rid of these two key frames for the Just get the camera going do a auto keyframe, right? And Is I think we're set up I think we can trigger a recording here, which is really nice Straight from the plugin and the plugin will tell rococo studio to start recording and then we have everything in sync and that's really nice So kind of works like on real engine with the virtual with the vcam It's the same kind of deal. We're we're getting close to what unreal engine is doing Um, just by using these two things, right? So I'm just going to start recording And let's see so All right, so try to record me here as I talk around So I'll be just, you know Something like this Get some close-ups Oh, I'm not actually recording, right? I need to hit play here boom Okay Let's try one more because the actually the app has a feature for recording if you just hit this button here with With this, I think the red one and the play button. So that will actually play it back three, two, one and Is it recording? No, did it trigger? Um All right, it's recording. It's okay. Keep on moving around So I'm just walking around here, but My guy stopped but anyway, I think it's the ports may be trying to fight each other start receiver. Oh, there we go All right Stop now There we go. So the camera got recorded. Yay. It worked. No way All right, so yeah, um Thank you. Um, this was stressful. Um Yeah live demos are always this kind of weird, but it Uh, but it actually works. I mean that we're getting previous here in blender We could record this this character here already the keyframe, but that would overload the system So it's kind of easier to just use This, you know, recording here plug-in and work from here rather than and you know, switch and back and back and forth with programs I'm actually running at 200 fps, which is boost mode here with all this wi-fi going on here I'm pretty impressed that the suit is able to handle You know All that right and it's actually and it just works, right? I'm just gonna stop it So that's processing the recording now. All right, so we had a recording. We had our vcam demo Now we can have a kind of a look of How to export this data and get in get it into into characters, right? Because that's that's actually interesting We're just two ways you can you can actually import a character straight into rococo studio, which is Very useful. Let me just go back here characters And it supports a bunch of things you can use an unreal item character a mixamo character most blender characters Ready player me icon characters character creator. So you're pretty much Set with this with this stuff So we'll use the the y bot You should be able to straight import here and this is just a drag and drop situation Where you basically drag and drop And you can see the character already performing here and this is really nice And i'll i'll tell you exactly why right? Because you can do adjustments on the fly because when I go up here There's these there's this thing called Elevation tracking which is this stuff, right? If you want to deal with that But also what I love here is adjusting right adjust arms adjust that so if I need the right wider spread I can do it. It's all happening in real time I can adjust this this this and this is really cool like this is just amazing Right that I can do this in real time on any character And I've even pushed this system to upload Meta human meshes So I mean it can handle a lot or a character creator mesh And just generally a very heavy character. You can already see what's going on right and record this Directly into the fbx of this character with with these setups right this really I mean I think it works But just in general I think this is one of the most like this is what you pay for in this stuff Where you want to work and just do minor corrections here so kind of retargeting retargeting is a big issue and and and It's it's not an easy thing to do retargets right and this rikoko studio just really helps you with that And I just want to say that it's amazing that you can have it So I'll just jump out of this character So I'll disconnect him now So he pops back into into my guy Now I'll just go briefly over the exporting. How much time do we have? 50 all right We have about 50 50 minutes. So let's Select the clip. So let's select our last clip here that we recorded double click And we get a bunch of presets here now presets are really important when it comes to Just dealing with characters, right? I personally work with mixamo rigs because they're just they've become the jpeg of motion capture, I would say Um now the next one is on real engine characters, which I would say are the ex rp and g's of of of of this and And human ik if you work with Maya iClone and so on but I really like mixamo and on real engine characters because they just work right I can get The data really fast in and Converted and do a bunch of things that would be really really annoying and hard to do so let's let's just export this As a mixamo character And just work with it. So just go here I like to name these clips. So I know what they're actually named And just export Now let me stop this So we have already the keyframes for our camera here Let's save it. Let's save it. Save it. Oh, I think my phone is still controlling. This is really cool Actually, is it is it still working here? Yes. Cool. I can use still the phone here and just focal lens This is really really nice. This is I mean because you can just Work on the on the go on the go with this I usually have this propped up here on a on a tripod and work on a bigger ipad, right? So I just have a second screen and then I can pull back and add another View so I can have different views, right? Um, let's pull up a scene that I've actually prepared here Recent So this was the the the the last scene that I did. I wanted to export this character actually, right? So I think we're done. Oh it exported So we can grab this fbx now Now I'm going to go over just a bunch of Quickly software that that I use here on the daily basis to just get these things done, right? one is Let's go to the plugins So number one is rococo you can use it for just the whole thing and also retargeting the plugin is free so you can retarget to any mesh and one thing here worth noting here is Is that let me just find it quickly Auto rig quickpro here we go So auto rig quickpro is an amazing plugin. Um, let me just get my Bone structure. Let's actually open that last file. Sorry for that We'll work on what we actually recorded on the session Oh, not this one So quick rig, um It's a paid plugin and it's totally worth the money. Uh, I'll explain why No limbs found add limits. Uh, let me just Oh, here we go. This is the button that I wanted to show so you can see that you can import and Get ik working in all of these characters and these systems Which is the ue 4 mannequin ue 5 many many A dash character all of this will work, right? So in this case, we have a mixamo character Let's try it. So We'll clear the current limbs. All right And this should automatically detect What we need, right? And Let's just go over here quick rig animation We can bake the animation, right? And let's do ik ik And hopefully fingers crossed this will work. Do we have an animation in here? Oh, we don't have an animation in here. Um, so let's actually just use the file that already have the animation imported So we just don't waste more time Uh, all righty get it back here And go to uh auto rig pro quick rig Select mixamo all right Uh You can import the bone mapping from file, but we already have that. I think it works spine leg Yep, I think most of it is there if I'm missing something we'll see on the retarget Um, so we'll just select this quick rig Uh leg leg left is missing required input foot So does it have a search? No, doesn't have a search. Let's just go here Nope, I don't want the hips. All right. It's actually attacking my hips here. Uh, let me just hear Right, let's not waste time on this. I'll direct to the videos that actually rococo studio did and They use this to do to you to basically get i k working And there's also two one more plugin that I'd like to mention that basically turns blender into um I clone essentially you can do that you can use animation layers in such an easy way which is a lifesaver for this stuff um So yeah, we went over the the the the system Just a quick recap of a typical animation studio pipeline that usually works. Um It's the old way of doing things. Uh, we've been I've been learning about this all on the on relengine fellowship and it just shows Um, how much Has changed, right? Because this used to be the typical animation where you go with the storyboard modeling rigging all that all that all that all that fun jazz But we're in real time here and the real time kind of works like this. Um, it's a bit Different it's set up In it in a way where we pull from the traditional way of making things But we just use on real engine to to visualize everything and and get get working with the with the With the pixels already real time with the ready pixels, right? Now I asked on the on real engine fellowship. How many people work with blender? I'm on the european. This is the european animation fellowship And half of the people raised their hands and i'm like Awesome because that means that blender is Just something that people use daily as a driver to get stuff into unreal Because it's just so robust and just works so well if you know both tools Now I I kind of Try to get people out of maya And they they they feel that they want to make the leap And but they always ask me how will I be able to do this and that and that and the answer is yes You can it's just getting a few plugins But the same as with maya if you think about it you you need A vanilla maya Is not that interesting it's all to do with the plugins that are the the interesting things that that make it all work Um, I'll just run a quick video here To let me just disconnect the suit here so we can get back online I just have the videos here. All right. Let's just go back to my videos I'll talk over this because this is obnoxious music um So I I'm fascinated by motion capture suits and I was able to work With the most high fidelity one which is the opti track and it gave me a big perspective on how this stuff Needs to be set up. You need an operator with this system. You need the 10 gigabit network. You need 48 cameras, right? Um, so my idea I came to a workshop to just show students how to work with mocap in the similar fashion that I'm doing here But without the the complex setup Now the I quickly rigged up the prop that I 3d scanned on set And then this is the same concept that I'm kind of showing you on the blender, right? A rigged up character working together, but in this case we're actually holding physical objects that are tracked And recorded and take recorder, right? So I asked this so basically that the whole idea was how to teach Skateboarding in mocap So I would as you can see I would run around try to get cool angles and as a as a dp And we would switch between this all that I think this was fascinating in the way where I was able to all without like in this case without unreal engine It would not be possible for me to actually visualize this on set and have it working immediately, right? and this is where the like the the the font part comes because All of this is Practically zero cleanup. There's only just some minor adjustments that you need to do But it's usually to do with just making the animation a lot a lot smoother Or but you can see how much fidelity comes into these motions and Comparing this to something like inertia suits That kind of figure out what's going on on They kind of estimate what's going on so you can fool them, right? While here it's like 48 red cameras looking at only infrared spots and Yeah, and the engine was the the whole system was able to to figure to To pull it off and actually record it in real time so this was I'm really proud of this and this was a way to Just show that you do not need different motion capture. I mean you don't need to be Sorry, I wanted to say this that that you can combine different types of mo cap into One production and it doesn't really matter what suit you use if it's a rococo suit Or if it's a move AI thing or it's or if it's any other solution that you could find It happens to be that The animator is in charge because there always will be cleanup even on the opti track system They deal they have problems with finger tracking because they use Manus gloves and that's a separate company. So you need like a separate software to just use that So this is where you know where I pop this all in and I just do some you know Magic and software and I kind of I'm already there while they are still configuring the system So it's a really really interesting way. So this was the exquisite corpse. I'm just an explanation As I said and so yeah, and I I take part in these fellowships and teach students about motion capture Recording this live and you can see my beautiful wife here Karina and to this is a professional actor that I also invited on set That I asked to Can you help us? Teach a bunch of animators and people who just are you know, really interested in mo cap on how to act for motion capture and this this This happened to be the most interesting class that I As I I wouldn't say class, but it was an experience where she would put all her energy into showing what it is to actually do mo cap and She did not have any experience working with mo cap, but she's a professional actor So her her experience what with with this was that this is like theater, right? You can I mean mo cap suits Because she worked in series in normal film production TV classical production and one thing that she told me Manu, which is the actor actress that usually when she works on a set There's the dp, right and He has a camera. There's a second camera and they usually have like markers They do a scene and they can't really you know use the whole Body of of of emotion to to work around this because you have to think about where the camera is pointing But in mo cap you figure out the camera later if you want to right and that's A whole different world of of just working with this stuff and it's even it pulls It takes different kind of emotions to work with this stuff Then working in traditional film production and that was really fascinating to me Um, I'll throw a few snippets here if you'd like to take screenshots or ask me about the software Character animation is hard, right? I've been doing it from connect from markerless in this case, uh, this is a screenshot from glycon 3d, uh, which is a software that just You can use vr any vr hdc vive and also the quest the quest to uh, essentially get motion straight into Unreal Engine blender or any other Program that supports skeletal meshes, right? I've talked to the developer and he's just also pushing the boundaries of how to democratize this stuff because this If you think about it, this is democratizing it. I know how much It costs to set up a robust opti track, you know System it's beyond anybody's budget probably around, you know, normal people Well, this these things are are becoming more accessible Um Yes character animation is easier now. Uh, this was a yeah, this is if you want to go and check out move one It's been just released yesterday. I was on the beta and this is also a Just a wonderful piece of software that you just point one camera You get a take and it actually deals better than some mocap suits with stuff like touch So if you want to clap or do things like this I would use move ai right because it just does a better job or there's more There's more It just doesn't really really well compared to suits, but if you're working in a suit You can get so much more takes and you don't need to upload it and you work in real time So it's a whole different story, right? So do do we really need to go this this high end, right? Well, if you want to capture dogs, of course, right? Um, this is probably the next frontier of motion capture suits, right? How do you get our pets into this which would be fascinating? Um, I know that auto rig pro is is on the is really really pushing the boundaries And they have some amazing rigs that you can all work with With this stuff and just get mocap into it um I didn't want to go much into this but this is What I am passionate about 3d printing and I try to invent my own helmets Headcams and my idea for this was to well if you you can use a broomstick and it will work, right? You can use anything But once you work for a couple of hours in these things or you put an actor and you tell him All right, can you just put that 10 kilo helmet on your head and start working? He'll get tired. You know his head will collapse and all that So I started thinking well, it's all it's it's actually a really hard problem to get um This get a phone in front of your face, right? Because the phone is usually heavy So I started experimenting. Well, what if I put the phones a center mass here use a mirror and it actually works so If if anybody would like to have fun with this I'm open sourcing this this helmet It's just open source helmet. You can print it out and just have fun I'll just ask me ask me about the link and I'll and I'll provide it to you if you'd like to print out one of these and just just use it I mean I've seen so many ideas of doing helmets, but I really think that this one is is interesting because it's just It's it's a lot of research Because if you put the phone just like this, it's fine if you do talking But if you really want to do like punching or something that's dynamic You'll get wobble and that wobble goes into the motion capture and you don't want that So I'm I'm just figuring this out as we go So hardware and software working together together helps this Helps to do this job, right? This is the backstage of how an actual I run a fellowship three monitors, right? I use a 3d connection also, which is amazing by the way Just just in general as a hardware thing that you can rely on on virtual productions or generally working in a studio environment Because you can map everything out and and have it in one place So and I invite you know This is Rococo's idea much more cleaner than my design, right? But it's it's actually really really well thought out and industrial designers can shine with this because it's It's solving a really hard problem of how to Work with this stuff and don't make it overcombered. Don't make it too hard to work with and all that, right? Um, yeah, just a bunch of software Rococo studio. I mentioned auto rick pro auto rick quick wick animation layers Virtue camera and glycon 3d if you're into vr and would like to use vr as your source of mocap data It's really good. Um, it actually works. Uh, there's also One more software that I'd like to mention. It's called animation prep studios based on unity also um, it can also do this the same thing similar to glycon works in this in this um In this realm well as I will I'm proud to be here on the blender conference. Um And if you are interested in the suit, uh, or in this case, um, reach out to Rococo Tell them blender conference 2023. That's the code use it. They'll give you Access to it. They might give you they might send you the suit They might give you a really good deal to just get going because there's super super talented people that really want to help democratize motion capture Just in general and they provide a bunch of things that can that help us as designers, right? All right, um, I'm gonna slowly Is out questions time. I know that there might be a bunch of questions I'd like to have this more of a just not me talking, uh, but actually get some feedback Uh, what would you like to know? Maybe we can record something for you and send it over Thank you, and I'm opening the floor to questions