 Hi, I'm Alex Krasner, I'm the Graduate Assistant for the Virtual Environment Studios at the Library of Virginia Tech, and I am here today to bring you a workshop to teach you how to get started with the Microsoft HoloLens 2, with developing a simple app today. We're going to be going over the basics of just getting a project running on it, how to do hand tracking, how to do distance selection, how to make it so that you can use your voice to control different things, so yeah. At the end of this tutorial, if you follow it, you should have kind of like a playground where you can play with different colored blocks and manipulate them in different ways and hopefully it'll give you techniques that will let you create your own apps in the future. So one final disclaimer also is that you can check out HoloLens 2's from the Studio's Technology Lending Desk in Torg Bridge in the library. So if you're watching this and you don't have access to HoloLens 2 and you'd like to build these things, then head there and you'll be able to check one out. So without further ado, let's move on to the workshop. OK, so here we are with looking at Unity Hub. And there's a few important things here that I think we need to go over. So the first of which is which version of Unity do we have to use for this? And if you look at the tutorials online or any different kinds of media, as of this moment, as of me right now recording this, the documentation will tell you that you need to use Unity 2020, not 2021, even though we're in the year 2022. And 2021 just went into LTS, which I'll explain what that means shortly. But they've only made this work for certain on Unity 2020. So just to show you how to get that, if we go to installs and we could say install an editor, if you don't have anything here, it's going to go to install editor. You're going to make sure you pick the latest version of 2020 LTS. LTS means long term support. What that means is that it's stable and they're basically only doing bug fixes at this point. So you know what you're working with. It's always going to work the same every single time. So I highly recommend if you are starting brand new on Unity. There's a different video I would recommend you to watch to start. But if you are new with Unity, make sure to stick to only working in LTS versions. I would not use 2021 normal. I would wait until you have an LTS, which is what we have now. You see here we have 2022 normal, but I would not use that yet. It's going to be very buggy. They're going to make big changes that will break everything you do. Stick in the long term supports. So anyway, if we go to install one of these, you're going to get a screen asking you about installing and blah, blah, blah, different things. Let's take a look. I can show you what that would look like by going into my install and going to add modules should pull up this. So it'll look something similar to this when you're going to install your version. It'll probably have a couple more sentences up here. There are some really important things for you to get when setting up Visual Studio or sorry, when setting up Unity for working for the HoloLens. And that is you need to have this box checked. There will be a checkbox next to it. Windows Build Support IL2CPP and you need to have Universal Windows Platform Build Support checked. And I haven't checked it, so I'm actually going to install it right now. You will see on the other side of this. All right, so our install is done. And now I'm going to go just point it out to you one more time. So you really can't miss it. It's going to go back to add modules. We want to make sure, again, that we had our Universal Windows Platform Build Support installed and we need to make sure we have Windows Build Support installed. You don't need Android stuff. That's for other things. You don't really need documentation. One thing you do also need is specifically this right here. Microsoft Visual Studio Community or it doesn't matter if it's community, but you need to have Visual Studio 2019. I know there's so many new versions. I know when you open Visual Studio 2019, it's going to ask you to switch to the new version. But I'm telling you right now, the HoloLens is only able to be built for in Visual Studio Community 2019 or just Visual Studio 2019. It's the only way to build apps for HoloLens at this moment. And that's just how it is. So maybe this has changed by the time that you're watching this. But it's been a long time and they really haven't been changing this. So I wouldn't be so sure. So make sure you have 2019 installed. If you have a copy of a newer version, 2022 or whatever installed, go pick up 2019. It's Visual Studio Community. It is free software. So just go pick that up. You're going to need it. Anyway, all right. Now that I've finished hammering on about that, we've got our version prepared. I'm going to go into projects and I'm going to make a new project. I'm just going to go new project. And here, let me turn on a quick feature right now. Now you should be able to see my clicks easier. When I click now, you should see yellow. And when I right click, you should see blue. But we just want to set it to 3D. When I give our project name, whatever we want to call it, I'm going to call it HoloLens to work shop video. Again, you can call it whatever you want. Make sure it's saving somewhere, you know, check up here to see that it's the latest version of 2020, which it is and create that project. And now we wait. All right. So here we are in unity. And if you're coming into this and you don't know, you've never touched unity before. Unfortunately, in this video, we're not going to be covering the basics. I do have another video. You can find that has all of the basics. I'll go over the entire interface. I'll explain everything. But today we're not going to be covering that, not in this video at least. So just keep that in mind. I'm going to be talking about some stuff that you might not know about unless you've seen that video or if you've used unity before. So let's get started. I'm just going to make a new scene just to rename my scene. Right now we're in our sample scene, which is stored in the scenes folder. And I'm going to just make a new scene. I'm just going to go to file. Save as and go into scenes. Let's make up a name for our scene. I'm going to call mine tutorial scene. You can call it whatever you would like. All right. So the next let's get things organized. I want to make two folders in here. So the first folder we're going to make by right clicking and going to create folder is one. I'm going to call scripts. This as it may seem is a folder where we will hold our script that we will use to control our objects and our world. And yeah, it just keeps things organized to be able to put them in our scripts folder. Next, we're going to want some materials. So let's make a materials folder, create folder materials. This will just help us stay organized. So next, let's set up our build settings so that we can be creating things properly for the HoloLens platform. It's pretty easy to do. We're just going to go to file, build settings. I'm going to add open scenes up here just so that we've got that out of the way for later. And I'm going to go to target device. Actually, first, I believe we got to switch this to the universal windows platform. Yeah, so this builds things for PC or for desktop. But universal windows platform is what all of the apps that run on the HoloLens use. So we're going to build for that. And let's double check all of these, make sure they're all set. We want it to be target device is the HoloLens. Architecture is x64. Wait, no, it's not. It's arm64. Yeah, so this is important. Change the HoloLens runs on an arm processor, but it is 64-bit. Build type, D3D, target SDK, latest installed, minimum platform version, exactly this. Visual Studio version, latest installed, build and run on USB device because we're going to be plugging in our headset and running it on there. Finally, build configuration. I tend to keep this as release. I think it's pretty important. It just means it's going to run better. Definitely don't set it to debug. You're going to have some performance issues. All right. So finally, all we want to do is click switch platform. And it will switch us over to build things for the HoloLens. And once that's done, we are good to move on. The next thing we need to do is we need to get the Microsoft, I think it's called, it's called the mixed reality feature tool. Mixed reality feature tool is really important for setting things up to work with the HoloLens. So what we got to do is we have to go to the internet. And I believe I have the page right here. So you go to this link up top. I know it's not really pretty. What I would do if I were to use our just Google mixed reality feature tool and pick the link that shows up here. Let's do that right now. And if we show up, not this one, but the second link download Microsoft mixed reality feature tool, go there, you're going to click download. And it'll give you a zip file. You're going to open up the zip file and extract the zip file and then run the executable, the EXE that's inside of there. Okay. When you run that EXE file, it's going to look like this. Let me pull it up. It's going to look like this. And from here, maybe you double check your settings, but you should be good to go and go back. I'm going to click start. All right, so the first thing that you need with the mixed reality feature tool is you need to give it the location of your project so that it can drop all of these assets into it. So we're just going to go into our computer. I store all of my projects on a special hard drive, my Zippy drive, it's a SSD. And I'm going to go to my code where I store it and let's find HoloLens 2 Workshop Video, go into there. And remember, you'll see right here that there's an underscore. Leave that there. That's basically telling it that the folder that we are in is the folder that it wants. I know it's kind of weird, but that's just what it is. You'll see when we click open, it shows us the folder properly. We are good. Let's find features. Okay, so for features that we need today, we need to get the MRTK Foundation and the Mixed Reality Open XR plugin. So the first one, MRTK Foundation is under this Mixed Reality Toolkit. That's what MRTK stands for. Section, feel free when you are using this on your own, maybe pick up some examples, some extensions and stuff, but for today, all that we're going to need is this one, Mixed Reality Toolkit Foundation. They even have some standard assets in here, which is really, really helpful. I would definitely recommend that sometimes. But for today, we're going to be building everything kind of from scratch, just to give you an idea of how stuff can work. All right. So the next thing we need is, as I said, the OpenXR platform, OpenXR plugin, and that is under platform support. Here it is right here. I'm just going to click the checkbox next to that, and I'm just going to double check to make sure that everything is at its latest form. Perfect. All right. So we are good to go. All we got to do now is get the features. It's going to download them, and it's going to say we actually need the standard assets. So it's going to give that for us. Let's validate. There were no issues. Let's import. And let's approve. Exit. So you'll see now when we open up Unity, it's going to realize that we've imported a bunch of weird packages into it. So it's going to take a moment, and then we will be good to move on. All right. So once we get done waiting, you're going to see this little pop up, and it's saying that we need to restart Unity. So we just say, yeah, let's restart. Okay. So we just had to wait a little bit through a couple of loading bars and to restart. And now we're back into Unity. And we need to run through this little menu. So the first thing we're going to have to do is to choose the recommended one, the OpenXR plugin here. We choose that. And we let it wait for a moment while it sorts things out. So once that's done, the next thing we have to do is we have to deal with this next little menu that popped up. And we're just going to follow what it says. We're going to show XR plugin management settings. And this should pull up when you click. It just pulled up on my other screen. It should pull up on your screen. And we are going to go to down at the bottom. You see how it has a spot called XR plugin management. Let's click on that real quick. And we want to turn on OpenXR. And specifically we want the Microsoft HoloLens feature group in here. Okay. Now you may notice that there's a little yellow warning up here. So I'm just going to click on that. And it says we need some interaction profiles and that we need a bunch of other different stuff. Click fix all. But it leaves something here that we have to fix on our own. It says we need to add at least one interaction profile. So it's time to select which controllers we will be using. We're going to go to edit. And what it did basically was it clicked on this little button down here. So it clicked, we were here and it basically brought us to this tab right below it. And importantly make sure we are always on this tab, the one that has the Windows logo on it, not this one and not this one. You probably won't even have this tab. And we need to do a couple of things here. So a first comes to first though is we got to fix that issue. It said we need an interaction profile. You can see here that the list of interaction profiles is empty. And think of interaction profiles as kind of like the way that Unity knows to talk to your controllers or whatever means of input that you have. On the HoloLens today we're going to be going over the hand tracking, the eye tracking and the voice detection. So let's set that up real quick. We need eye gaze interaction profile so that we can check the eyes. We need hand interactions that we can attract the hands. And I believe we are good from there. We actually don't need an interaction profile for voice. It comes built in. So anyways, eye gaze interaction profile. Now eye gaze has a weird bug here. Let's click on that. Requires gaze input capability, fix all, boom, we're done. All right. So next we want to go into this little top section up here and we want to make sure our depth submission but mode is set to eight, 16 bit. That's just going to help optimize this, make it less taxing on the headset. And for the HoloLens you want to keep it set to single pass instance. It's just the way that works better for rendering on it. I'm not quite sure why on other headsets you would probably want to set it to multi pass. So once we've set that up, we actually want to go back to this little window in the back that we were working with earlier and let's do some things here. The first thing we need to do is apply the settings that it wants to set up. And once it has applied those settings, we are going to click next. And then we're going to make sure all of these things are checked as they already are. We're going to click apply. And when we do this, it's going to make us restart Unity once more. So let's do it and it's going to restart real quick. Alrighty, so once we've done that, it has now finished configuring things. There's a few more settings we have to change though, unfortunately. First we want to make sure that everything is properly configured. And there's one part of that prompt we were just seeing earlier. Remember there was a window here that we're clicking buttons on it, kept making us restart. We got to bring that window back. To get to that, we're going to go to Mixed Reality. We're going to go to Mixed Reality. We're going to go to Toolkit, Utilities, Configure Project for MRTK. And this will bring us back through. We already hit the apply settings button. I think we should just be good here. Next, next. But here, we want to make sure we have TMP Essentials. Just a lot of things need this to run any of the text stuff that exists here. So we're just going to import this. And boom, our setup is done. You can click done. Now, there's still more settings to change. This, unfortunately, is a really complicated process. Next, we need to go to Edit. You may already have this menu open, but I closed those windows, so I need to go to Edit, Project Settings, and go up to Player. Right here, Player. So there's a few things we have to change here. First thing is we want to set up what our app is called. You can see here, here's the product name. Product name is what the app's going to show up as, like the title of the app is going to be on your headset. All right, your version is here. But there's one pretty important one that's not shown to us here. You can also give yourself a company name if you would like. But under Other Settings, if we go down, sorry, I believe it's under Publishing Settings. Yeah, so under Publishing Settings. Under Publishing Settings, you see here it says Package Name. You wanna change this. It's defaulting to Template 3D, but we want to call this something unique, something that no other app on your headset is called. And this is because whenever your package name is, think of that as what the computer looks for to see if you have an update to an app or just a new app, right? If it has something with the same package name, it's like, oh, you're just giving me an update. I'm gonna replace my old version of that thing. Whereas if you give it something with a different package name, now it's going to not overwrite it. It's just gonna install another app on. So this could cause problems. Let's say you left this as Template 3D and you kept building apps and putting them onto your headset, it would delete the old app every single time, even if it was a completely separate unity project that you were working on. So let's give this a unique name right now. Let's call this one, I'm gonna call it Workshop. Call it whatever you want. Doesn't matter, just make sure it's unique. You're not gonna have another app on your device that has the same title. Cool, you can set the version whatever you want. This is really all that matters here. We're just gonna make sure our package name has been set to unique. Okay, and one other thing that's helpful for me when I'm developing at least is I like to set my product name to have an underscore in it, or in this case, maybe two underscores in it. In fact, I'm gonna put underscore and then the letter A. I want our thing to show up at the top of the list inside of the HoloLens when we're looking at our apps, because it'll help us with debugging. We won't have to sift through this list. We'll just see it at the top of the apps list. So I'm just gonna put that there so it's gonna show up at the top when sorted alphabetically. Alrighty, we did our thing, so let's close this menu. Now, it's time, this is a really long process. I'm sorry, but this is just how it works. We want to add MRTK to our scene. So to do that, we're gonna go to Mixed Reality, Toolkit, Utilities, oh sorry, not Utilities. You're gonna go down here, where's this Add to Scene and Configure. Click it, boom. Here it is in our scene. You'll notice it just replaced our main camera, but it left our directional light. And in fact, it didn't replace the main camera. I believe it pulled it into the Mixed Reality play space. It did change it a little bit though. So to explain what these are, Mixed Reality Toolkit, think of it like the settings and the features of the HoloLens system. It's kind of where all the settings are held so that we can change them and tweak them. The Mixed Reality play space object is what it sounds like. It's the play space, it's the virtual play area that we will be able to stand in and walk around. And finally, Scene Content. This is just a parent object to kind of make sure that everything scales properly. The headset can't really track, the headset scales everything to this size of the world and the user when it loads. So if you put anything as a child of this, it will get scaled properly into the world. So basically, best practice, just whenever you make an object, make it a child of Mixed Reality Scene Content. Alrighty. So we've done a lot of things and I don't want to lose our progress. How do we get into Sample Scene? What? That's odd. Well, you've already done all this work in Sample Scene. So I'm going to just save it into Sample Scene. Don't know why I didn't switch that. Okay. Anyway, not super important. It really was just a name change after all. Now what we want to do is, I saved the scene, Control S. On a Mac, this would be Command S. And let's start finally making some things in Unity, okay? Where is our camera? Let's take a peek. Camera is here. Where is the camera? Are we really far away? It feels like we are far away. Well, the main camera is not showing up in the gizmos. Oh, because it turned off gizmos. Okay, there we go, this looks a lot better. So you should probably see this. If you clicked on either of these, you would see the camera show up. We can see which way the camera is looking. And I would like the camera to be able to see a cube. I'm gonna make a cube by going into Game Object, 3D Objects, Cube. It spawned it in who knows where. Let's reset everything to zero, pull it in the middle, and then drag it a bit in front of our camera. In fact, oh, one important thing. Remember what I just mentioned a moment ago? Whenever we make an object, we wanna make sure that it is the child of mixed reality scene content so that it scales properly. Just gonna drag it and drop it on top of there. It's now a child. I think I'm gonna make our Z be one. So think of these measurements as meters. You would know this from the basics, the basics workshop, but yeah, gotta remember this is one meter away from us. We've moved the center of the cube one meter away from us, but this is a one meter by one meter cube. So it's not that far. So what we wanna do next is we wanna scale everything down. Let's make this cube something more reasonably sized. Point one, I'm sorry, point two, point two, point two. I like point two, this is pretty good. Okay, we've got a little cube in front of our camera. We can check in Game View, see the little cube. But I think we still have some more stuff to do. Notably, I think we gotta make some colors so that we can kind of spice this up a bit. Let's go to materials and let's start making some quick materials. I'm gonna right click, create material, and let's call this one red, and let's do that again. I'm gonna create material, and let's call this one green, create material, yellow, and create material blue. Okay, this is kind of what our whole exercise today is gonna center around are these colors. So right now, you can see that there's no color added to these materials. It's time to color our materials. I'm gonna go to blue, and I'm gonna make it blue. What in the world, it didn't make it blue? Oh, that's because we're still in the white corner. Drag this in here a bit, that's a nice blue. I like that blue. You can really make the whatever color you would like, but I'm just going with red, blue, green, red, or blue, green, red, yellow. Okay, go to green, let's make it green. And one thing with green, green is the one where there's a little bit of a note here. I would recommend whenever you're making something green for the HoloLens to make it a darker green, because our eyes perceive green especially well, and when it's being beamed off of a laser into your eye, green can be especially bright, so I would make your greens a bit darker. So let's continue. Let's make our red. Our red is here. Red, boom, boom, big red, yellow, click, click. That's a quality yellow color like that. Here are our colors. And you know, if we drag them on to the cube, we can change its material. I'm gonna leave our cube blue. All right. So now let's take a peek at what happens when we press play in Unity right now. Change no settings, no nothing. If we click play, what happens? Well, take a moment and then we can see this. And this is our kind of our little demo thing. You see this little debug console down there that's from, it's part of the MRTK stuff we've added in. I'm gonna have us remove it shortly, but we can kind of look around in this virtual space and here's the cube. You can even move around. Here's our cube. All I'm doing here is I'm holding right click and WASD to move around. So let's make it so that we can actually kind of interact with this cube. Stop, and once it's done stopping, I'm gonna go to export this to Visual Studio. This can take a bit sometimes to stop. Here we go. So now that it's stopped, let's go to our build settings and we are going to build this project. Now, we're only gonna do this once today, or I guess twice today. I'm gonna show you how to do it right now and then we're gonna do it again at the very end. For the rest of the day, it's going to be me showing you what would happen if we built the project. I've kind of pre-baked the cakes as it were so that I can show it all live because this can take a while and you're gonna notice right now that this will take a little bit. You wanna make sure up top that the correct scene is set. I originally had us put in our first scene but it seems that some issue happened and we've built everything into sample scenes. We wanna make sure that whatever scene you see up here with all of our mixed reality stuff below it, make sure that that one is added. You can go get that by clicking add open scenes and clicking the check mark. And then all you gotta do is click build. It's gonna ask us where we wanna put it and I have a whole folder for my builds. It's under here. It's called code builds. And so what I'm gonna do is I'm just gonna make a new folder in here. Call it whatever I would like to call it. This one's gonna be hollow lens. Video. Boom. Go into hollow lens video, select folder and we will begin the building. So this could take a little bit. And I will be with you when it's done. All right. So once that has finished, you will probably see something that looks like this pop up. It'll be the same folder you were looking at and inside of this folder, you will see a couple of files but there's one that's the most important. This is one that will say blah, blah, blah, blah, dot SLN. This is a Visual Studio solution file. And when we click on it, it will open in Visual Studio. So let's do that right now. Make sure it opens in Visual Studio 2019. If it doesn't, you're gonna have to right click and select 2019. Now we wait for it to load. And as it loads, we are gaining access to this project. And I believe it has done it. Oh, there we go. And success. All right. So it's loaded our project. Now if we're gonna build this onto our headset, we need to compile it in Visual Studio. You can't compile for the HoloLens 2 in Unity. Unfortunately, that's just a restriction that we have to deal with. So what we gotta do now is just change our settings. As I said earlier, what we want to be doing is building it as a release version and our headset runs on ARM64. So now the only thing left is we need to make sure that our device is connected. You need your HoloLens to be set up with developer mode and all of that stuff turned on. So you'll have to kind of maybe do some Googling if you don't know how to turn that on. It's in the settings, in your headset. So now that we've set that up, I'm gonna go to make sure that we are building to a device. And now it knows, the fact that it even showed device is telling us that it has some kind of device connected. And from here, I'm gonna go to build deploy solution. This is going to build the code and then send it to the headset. So I'm gonna click this and we're gonna watch it build. Now this can take a while. It's typically worse the first time, but it definitely takes quite a bit. And because of this, this is the only time we're gonna do this during this tutorial. We'll do it one more time at the end, I guess, so that you can push the final version to your headset. But for the rest of this, I will be doing that cake big thing I described earlier. So when this is complete, you will resume and I will just kind of jump to that right now. So here we are. You can see at the bottom, it said the deploy succeeded and the build succeeded. Let's take a look at what this looks like. So what I'm gonna be doing here is showing you the live feed from the headset. I'm probably gonna have to refresh this page real quick when I get my headset booted back up. Let's take a quick peek at what this app looks like right now. One common thing that will happen when you first open up your project is it will ask you for permissions. Just say yes to any permission requests. Sorry if the mic audio is a bit quieter because I'm standing across the room. Okay, here you go. This is what it should look like. Basically just a whole bunch of weird meshes put onto all the surfaces around you and there's our cube right in front of us. You may or may not have a little like debug bar here. If you do, that's something we're gonna be turning off in just a moment, so. Anyway, you can see there's a cube here, but importantly, I cannot interact with it at all. You can move my hands up, I can try to touch it, but nothing happens. I have these rays coming out. I can't do anything. So, let's fix that. Let's get rid of this weird mesh on the walls. Let's make it so that we can actually play with the cube. So, back into Unity. Let's try and make all those changes we just mentioned. So, first we want to get rid of that white mesh that's on everything. Let's close this build settings. We don't need that for now. We're gonna go to the Mixed Reality Toolkit object and it's time to cover profiles. Profiles are a very weird feature, I guess, of MRTK. And basically what they are is a way to have settings stored for each of the items, for each of these different categories. So, for instance, we have camera settings, input, boundary, blah, blah, blah, all of these different settings, but we can't change any of them until we make copies. This is because it wants to preserve the default set of settings. So, what we have to start with is clicking Copy and Customize. And typically I just leave it as what it is, but you can choose a new name for the profile, otherwise it's just gonna put the word new in front of all of them, but we're gonna be doing this a lot. So, yeah. And for target folder, it's gonna automatically put it all into this location. So, let's go clone and watch what happens. Boom, now we have access to all of these new settings. But we actually can't change them until we would clone it. But we don't care about changing these settings. What we wanna go change are a few other settings. So, the first setting we wanna change is Spatial Awareness. This is where we're getting that mesh from the XR Spatial Mesh Observer. So, let's try this as we can see it some more. To make these changes we're gonna have to clone it. So, just click Clone, Clone. Boom, now we can access all of these settings to change them. So, we want to go to the open XR Spatial Mesh Observer and in here we are going to change a couple of settings. But once again, to change any settings, we need to clone. I'm just gonna clone it again. This is to change any settings. It's kind of like a bunch of cascading things we have to change. All we gotta do here is down here where it says display settings, we wanna change it from visible to occlusion. What this means is now instead of us seeing that wireframe of all those white little lines everywhere, now they are invisible but they still act as an occlusion map which means if there was an object that was tracked it would block us from seeing virtual objects. For instance, if I had a chair and it saw the chair, any virtual objects we placed behind the chair would be occluded by the chair because the headset has mapped out the chair. So, that's what we wanna do. And we're good to go now. We also wanna get rid of that weird floating box. I don't know if you saw the floating box. You may or may not have. Sometimes it pops up a little bit late. But either way, we just wanna go to diagnostics. We want to clone this one so that we can get rid of the weird diagnostic box. And it's very simple. We go here and I disable the diagnostic system. We don't need it. You could use it. It's probably useful when you're developing certain things but for all of our cases that we're gonna use today it just lags us out. It's just not worth it. Alrighty. Next thing is next. I'm gonna show you the next build soon but first I think we should make it so that we can at least play with the cube, right? Let's go to our cube which is in our scene contents. Go to the cube and I'm going to click on the object right here. And you can see we need to make sure it has a box collider which it already has and we need to make sure that it has two other things for us to be able to deal with it. We need something called an object manipulator. And when we add this it actually adds a little thing here where we can say add near interaction grabbable. Click on that and it adds another component called the near interaction grabbable. Both of these are important for letting us manipulate the cube. So there's a whole bunch of options everywhere here. There's so many options. I'm not even sure quite what to go through. You can look these up for different types of things. Right now we have it set so you can do any kind of manipulations. We have it set so we don't have physics actually right now enabled though as you can see right here. And there's different events for when you do different things with it and the near interaction grabbable lets us grab something with our hands. So with just object manipulator you could grab something with your raycast but now we can grab things with our raycast and our hands. So I'm gonna show you the demo in the Unity Client. Let's try pressing play and see what happens. Let me save, oh I can't save while I'm in here. Okay here we go and if we click we can move around the cube because we're selecting and dragging our selection. Perfect, all right. Close the flare and let's, as soon as this is done save our project so that we don't lose anything if something crashes. And the next thing we're gonna do is I'm gonna show you what it would look like right now if we actually built this because I have already built it. All right, so here we go and everything is a little bit delayed from what I'm seeing by the way. So my voice response about different things is gonna be a little bit late. Here we go, let's go to all of our apps and I made a second demo here called prepared demo two and this should show off exactly what things will look like. So let's go in, okay here we go. Right in front of us you can see there is a cube. You can see we have hands and if we grab the cube you can move the cube around. We can also grab the cube with far interaction like this. Drag it around and twist it or whatever. We can even shrink or grow the size of the cube. These are all built in functions. But you might notice this cube is just floating in space. It doesn't operate like we may want. So let's fix that. Let's make it so that our cube is able to be played with kind of like a real object. Let's give it some physics. All right, so back into Unity. Let's give our cube some physics. All right, so we have a few things to change here. First we wanna give the cube physics. So it's very simple to do that. All we do is we go to add component on the cube and type in rigid body. Click on normal rigid body. Do not click on rigid body 2D. We are in a 3D world. So that's it. By doing that we have now made the object have physics. You can change different features about it, but that's it. Putting a rigid body on object gives it physics, right? You can see use gravity is on, which is important so that we have some gravity for the cube to fall and different things. However, there's a little thing you should know about using the HoloLens 2. And that is it maps the world around you to build kind of like a 3D object of the world around you. This is good. This is really an awesome feature. A lot of other things don't even have that ability. Like as of the time of recording this, the meta quest 2 cannot do that. Now, that's an awesome feature, but it has a little bit of a drawback. And that is it has to build this creation of the floor and the walls and the objects around you. And it takes a little bit to do that. What that means is the amount of time that this cube would have to fall before it hits the floor is too little time for the headset to build a floor. So what we gotta do is make, let's make a little shelf and the shelf can hold our cube until we see a, until the headset has built the floor. So I'm gonna build a little shelf below this real easy. Let's just make another cube, video object cube, make sure, I'm gonna call this one shelf and make sure once again that it is a child of mixed reality scene content. Gonna zero that. And I'm gonna move it over here. Now let's flatten this out a bit. I'm gonna scale it. Yeah, let's make it a little bit wider, height wise, real flat and Z a little bit narrower maybe. Okay, now let's also move it down a little bit vertically. We want this to sit below our cube. So let's let the cubes fall onto it. All right, perfect. We've made a shelf and because the shelf has a box collider, the cube will lay on top of it. So also I think one cube is too few. Let's make a couple more cubes. It's really easy. Let's just duplicate. Right click, duplicate. Right click, duplicate. I'm gonna shift my two cube over here. I'm gonna shift my one cube over here and boom, we got three cubes. I bet you're wondering what this looks like if we were to run this in the headset right now. So I will show you. Let's take a peek. Okay, cool. So it did put these pretty high up in front of us at about head height. But here we go. We can see right now. Here's our hand. We can grab our cubes. We can now drop them. I can even drop one on the floor. Well, down there it goes. It's got this cube. Drop that on top of that cube. Fling them all into that wall. Replace them on top of this box. I can kind of do whatever. Yeah, so that's what we have right now. Let's add in some voice recognition features. All right, so it's time to import a script that I made for us to control the color. And what we're gonna have to do, there's not gonna be any scripting you're gonna have to write today, but you're gonna want to go to the internet and type in bit.ly slash, bit.ly slash hollow lens underscore script. It should bring you to a doc where you can download this script. So just download it. And adding it to Unity is very simple. Just drag and drop it into the scripts folder. And here it is. Ooh, lag for a moment. Okay, so now we have our color changing script. I can show you what it does real quick if I pull this up in Visual Studio. Take a quick peek. And all that it does is, it's going to be getting the materials that we're gonna give it. And it gives us a way to call a command to change the material to a particular material. And it's important that we keep track of a previous material as will make sense in a bit. So, yep, that's basically all it does. And I'm going to close Visual Studio. Let's put this on our cubes. So if we select all three of our cubes actually I think we should do this from, let's work on one cube and then we'll just duplicate all of our cubes. So I'm gonna delete all of our extra cubes right now. We don't need them for now. Let's start with just our middle cube. And on just our middle cube, I'm going to add color change. And I'm going to, we gotta put some materials in here. See, our script has a bunch of public slots for us to give different materials. I'm gonna go to materials and just drag each of them to where they belong. Blue to blue, red to red, green to green, yellow to yellow. Cool. All right, so at this point, we've now told our cube how to change color but it doesn't know when to change color. All right, the next thing we need to do is what I just said, we need to tell it when to change color. So to do that, I'm going to go to our mixed reality toolkit object up here. We are going to go to input. And in input, we're gonna clone it because we wanna make some changes. Okay, you can see at the bottom there's one called speech. That's what we want here. What are we gonna do? Make some changes, we have to clone it. Okay, now there's a whole bunch of ones that come built in here. Menu, select, toggle, blah, blah, blah, blah, blah. Not important, delete, delete, delete, delete, delete. We're gonna make some new speech commands. Okay, to do this, all we gotta do is give our commands a keyword. Think of this as, this is the words it's listening for. And when it hears these words, it will do the thing. So it's almost like a hot word for your virtual assistants. So this one, first one's gonna be called turn red. And next one is gonna be called turn red. Blue, and finally, turn green. Cool, so here we are. We have our three commands, kind of like hot words. And now it's always gonna be listening for us to say one of these three things. However, it doesn't know how to listen for us yet. It doesn't have access to the mic, basically. We have to turn that on. So to do that, we go to input data providers and let's take a look through here. We want to find Windows Speech Input. And as we check here, it actually looks like it's already here. The Windows Speech Input is here. That's all that we needed to know. By having this here, we know that it's actually able to detect our speech. So now we need a way to handle all of this speech stuff that's going on, right? We don't want the cube, the cube needs to be taking some signals, or we want to be changing the color of the cube, right? But I don't think it's smart for us to have the cube changing its own color and have all three of the cubes having to handle all of the speech at once. It's generally smarter to make one object in your scene that keeps track of all of the audio sources at once. This is kind of how things are built for this. So to do that, I'm gonna add an empty object. Gonna make sure it's in here and I'm gonna rename it to be called, let's call it speech handler. And what do you know when we go to our empty speech handler, we are going to add a component to it called speech input handler. Awesome. Here it is. We want to make sure that focus required is turned off because it should be listening at all times and everything else should be good except we do need to add a speech confirmation tool tip. Think of this as a way for it to tell us when the speech has been recognized. So all I'm gonna do is just type in speech. Oops, I think it's looking in the wrong areas. Let's go back to assets. I'm gonna type speech. Oh, it's actually under packages. Speech, here we go, packages, speech. Confirmation tool tip. What's this one called? What is this one called? It's not telling me it's too small. Speech confirmation tool tip. Oh, there we go. There it is, our prefab. So yeah, you wanna type in speech confirmation tool tip like that, right there. And I'm gonna go back to our speech handler and drag this one right here. And there's a way to get to this file, by the way, it's just a pain in the butt. You go to packages. You go to mixed reality toolkit foundations. You go to SDK. You go to features. You go to UX. You go to prefabs. And then you go to tool tip. You see how this is a pain in the butt? And here it is, speech confirmation tool tip. But we were able to find a bunch of searching here. Just wanted to give you a backup plan, just in case. Okay. And that should give us speech being handled. However, it says that no keywords have been assigned. Hmm, that's odd. Well, what that means, let's click this plus button here and let's choose term blue. And I'm gonna click the plus button again. And let's choose, let's choose turn green, plus button again, turn red. All right. So here we go. Here's all the speech commands. It's now listening for all of our words, but it doesn't know what to do. You can see it has a section called response. So when it hears turn blue, what does it do? When it hears turn green, what does it do? When it hears turn red, what does it do? Well, let's tell it what to do. And let's do this for all of the cubes we want to have. I would like to have three cubes. So I'm gonna just duplicate this cube. Bubble, I'm gonna duplicate again. Bubble. There we go. We've got our three cubes just chilling out here. And if we go back to our speech handler, which I'll put at the bottom for organization's sake. If we click the plus button here, you get this little thing where we can drag in an object and then we can use any of the functions attached to it or attached to things on it. So you'll notice when we drag in the normal cube, we can go into here. And our script that we made called color changing has a couple of commands in it. Well, all right, when we hear turn blue, what color should it turn? Probably it should turn blue. So now when it hears turn blue, it's gonna call turn blue on the cube. However, it's only gonna call it on one of our cubes. We have three cubes. So let's do this again, do this again. And now let's add in cube one, cube two. And we're gonna be able to speed this process up really quick after we do it the first time. We're gonna change this one, color changing to turn. What's this one? Still turning blue, we wanna make sure all of these ones are turning blue. Okay, I'm gonna actually do something weird. I'm gonna delete and delete because when we add a new one, it's gonna use the built-in parts here. So watch when I click plus, boom. It already has our color changing turn blue. All we gotta do is set this one to be for turn green. We gotta change this to be to turn green, change this one to turn green, change this one to turn green. And plus this one's gonna be for turn red, change this one to turn red, change this one to turn red, change this one to turn red. Okay, so now whenever it hears these keywords, it's going to call the respective command on each of these cubes to change their color. All right, let's save and let me show you what this would look like in the actual app. So let's go. Cool, so you can see we have all three of our cubes. We have our hands and we can grab the cubes. What if I wanted to change the color of these cubes? Well, turn red, sometimes it takes a second, turn green and turn blue. So that's how it works. We can change the colors of all of our cubes with our voice. So there's one other method of input I think is really helpful in augmented reality and that is eye tracking. So let's make it so that our cubes change color when we look at them. All right, let's do that. So back in unity, it's time for the final step. It's time to make our eye tracking work. So the first thing we need to do is check that we have the data provider. Let's go up into our mixed reality toolkit. Let's look into our input data providers and let's look for eye gaze. There's nothing here. We have to make one. Add data provider in here, type. We're going to choose. It's called OpenXR Gaze Data Provider. Here it is. And now we can change the settings a bit if you want. This is really up to you. Personally, honestly, I think I prefer my eye tracking unsmoothed. So I'm gonna clone. I'm gonna clone just so I can uncheck this box. I like my eye tracking nice and jittery, like my actual eyes. But if you want it to be smooth, you can just leave that on. And now all we gotta do is go to pointers. And obviously clone it because we want to change things. Now look down here and there should be a setting in here called is eye tracking enabled? Yes. Yes, eye tracking is enabled. So turn that on. And one thing that can be really helpful when you're testing with this, we're not gonna really use it today, but something that's helpful in my opinion is to go to the input simulation service. So it's, sorry, it's under input data providers. We go into here. And I like to clone this one. Again, this isn't super necessary, but I would recommend this if you want to work on your computer at all. So right here, eye gaze simulation. I'm gonna choose camera forward axis. You can also choose mouse, just wherever your mouse cursor is. In fact, let's go with mouse, either or. And here, this is what it would look like. If I click run right now, it's gonna show that you see how the little cursor moves around with my mouse pointer. This cursor represents where we're looking. So let's make it so that when we look at this cube, it turns yellow. And when we stop looking at it, we want it to turn back. So all we gotta do to do that is to go to all of our cubes. I'm gonna select all three. Oops. Just holding control and selecting all three cubes. And with all three cubes selected, I'm going to add a new component. It's called eye tracking target because these are all targets of eye tracking. Cool. All right. Now, all we wanna do is when we start looking at them, we want to change the color. How do you do that? Again, we have all of these selected. We're gonna edit all of them at once. On look at start, we want to have each cube change its color. Sorry. On look at start, we wanna make each cube change its color to yellow. And when we look away, we want it to turn back. So when we start looking at it, we're gonna make it turn yellow. And when we look away, we're gonna make it turn back. We have to do this on the cubes one at a time, though, unfortunately, because we want each cube to change its respective color. So cube one, drop it in, drop it in. Again, this is just gonna be kind of annoying, but when we start looking, we wanna turn yellow. When we stop looking, we wanna turn back. Same here. Drag your cubes in. When we start looking, we wanna turn yellow. And when we stop looking, we wanna turn back. Finally, our final cube. When we start looking, we wanna turn yellow. When we stop looking, we want to turn back. And that is it. Right now, we are good. It will change color when we look at it. And let's test it just to take a peek in here. Remember what I said? Our mouse simulates our eye gaze. And what do you know? Boom. Awesome. So that means it's time. Let's take a look at what the final product looks like. But actually, I'm not gonna take a look at it in the like in a pre-built way. Let's build the whole thing for real and stick it on the headset and run it. You all can follow along. You can build this one out and put it on the headset. So all we gotta do is go to file, build settings, everything set, build, gonna go to HoloLens video and it's gonna replace the old build. It's gonna build over it. And yeah, right after this is done, I will bring us into Visual Studio. So it finished building. So all we gotta do now is, once again, we're gonna open it up in Visual Studio. And we wanna make sure that our HoloLens is connected right now through USB, USB-C to our computer. And release, Arm64 device is all set up. So all we gotta do is go to build, deploy solution. And I'll see you all at the end of this wait. All right, so once it finishes, you'll see at the bottom that it will say that the build succeeded and that the deploy succeeded. If you don't see that, Google is your friend. So anyway, let me show you what it looks like. It's been pushed to the headset. I'm gonna run the actual app that we just built. Let's go. All right, we're under our apps. We called it HoloLens workshop video. Let's check it out. And we gotta give it permission. Okay, here we go. Let's take a look. Here's our cubes. Here's our big thick shelf. And grab a cube, grab our cubes. You can staff our cubes if you want. Which I want to. And yeah, you can see whatever I look at, turns yellow almost instantly. Yeah, it's pretty fun. And we can always just say, turn red. And everything turns red. And the changing still works because we have it turning back the colors. And yeah, that is the creation. We now have a little app we can play around and it showed everyone how to use eye tracking and tracking, distance grabbing, the whole shebang and voice controls. So yeah, that's what it is. And I'll see you all back with the face cam. Hi, so that was the HoloLens intro workshop. I hope you learned something and I hope it helps you with any projects you're working on. Once again, don't forget, if you want to check out a headset from the library, you can at the Torg Bridge. There's the studio technology landing desk and you can get HoloLenses, you can get quests, you can get pretty much anything there. So yeah, thanks for watching this workshop and I hope you have an awesome day. See ya.