 Hey everybody, welcome back. We've got Swicky talking about mixed reality with Xamarin. Take it away. Hello, hello. Hey everyone, I'm Swicky. I am actually from the San Francisco office. So if anyone is here, come by, say hi. I'm at the fourth floor. You can heckle me from the window on the side here. So let me share my screen. That should show you my slides. Okay. All right, I am going to start sharing my slides now. Let's get rid of this little puppy on the side here. Oh, here we go. Okay, hold on. I want to do that one more time. You just want to minimize the skype window as soon as you get there. All right, let's do this again. So, okay. Sketch you out of the way. All right. We're in business. Cool. Welcome everyone. This talk is about mixed reality with Xamarin and Azure Spatial Anchors. So I'm Swicky. I work in Microsoft. I'm in the Dev Dev Customer Advisory Team, specifically in the Mobile Customer Advisory Team. So what is Xamarin? So I know we had a bunch of talks David talked about and Maddie did it earlier today. But just a quick overview for everyone who's new who doesn't know what Xamarin is. So basically Xamarin is the C-sharp offering from Microsoft. It allows you to write C-sharp code and generate native iOS and Android apps. So you have all the power of .NET and C-sharp. So you're sharing code. You're doing all of that .NET awesomeness and it gives you great native iOS and Android apps. So what that kind of means is I totally stole this slide from Maddie earlier on. So thanks Maddie. But this is kind of what it looks like. So you're writing shared code, shared logic and all of it in C-sharp. And then the output that we provide to you is a native iOS and a native Android app. So again, just a little like sort of how it looks like. So you have all of this, you know, all the systems stuff available. You have all the native iOS UI kits available to you to write your apps. Similarly on Android, you have all of the awesomeness from Android provided to you. But you can just access all of it through C-sharp. All right. So let's get into the meat of this topic. What is augmented reality? What is mixed reality? And how does all of this work with Xamarin? So what is augmented reality? So imagine basically that your camera is a giant canvas. So anything it points to is your canvas. And you can throw 3D models at it to generate in any space that is visible to your camera. So essentially in very, very simple layman terms, in augmented reality, your every frame, so your view frame that your camera provides to you is your canvas. And you'll be virtually drawing objects on it within this special coordinate system, which is relative to your spatial location. Now, since we're talking about augmented reality, so AR with mobile applications, so we're going to be tapping into each native platforms, AR specialization. So iOS provides us with ARKit and SceneKit and Android provides us with ARCore and Sceneform. So you'll notice as I walk through the demo and I walk through the code, we basically will be piggybacking off all of these amazing native tools already provided to us from iOS and Android. And we're just sort of adding our own little magic and goodness to sort of make it all work together. So what is Azure Spatial Anchors? Then you'd be asking me, I mean, sure, we already have all these sort of native options from each platform. Why do you need Azure Spatial Anchors? Well, that's the cool part. So Azure Spatial Anchors sort of lets you share that spatial space that you're creating through the cloud. So basically my idea of this whole thing was I wanted to make my own virtual treasure hunt. I love puzzles. I love games. I love escape rooms. So I was like, hey, I want to make an escape room sort of virtual puzzle for my friends. So I'm just so sure why not. Let me use Azure Spatial Anchors for that. So the idea is you create an instance of your Azure Spatial Anchors. You drop what is called anchors. So there's basically those spatial points. And that's where your objects get drawn, those 3D objects that I was talking about. And then you can have, you can share that space with anyone else. You can have it access controlled. And your friends can sort of then see what you are seeing through their respective cameras. So the coolest part about this whole thing is Azure Spatial Anchors as a service is amazing. They have support for native iOS, native Android. You can write Unity apps using it. You can write a HoloLens app using it. And you can also write it using Xamarin. I love Xamarin. I want to use Azure Spatial Anchors. So this is a perfect solution for me. Also other cool stuff that we'll be doing in this demo is we'll be using Cosmos DV and an Azure App Service to sort of drive some little bit of data that we have to store to help that sort of sharing happen between devices. And it's a sort of a future feature I'm going to add at some point, but it can also sort of add login based control to the system if you want to. So it's a really powerful tool and it's available to everyone right now. So let's move on. So what I'm going to be demoing today, like I mentioned since I love Vuzzles, it's a game that I call Xamagon Hunt. So it's basically like a virtual treasure hunt. And if you want to see the code right now, you can just go onto my GitHub. I'll be sharing all the links again right at the end. So Xamagon Hunt, right? So the idea being there is everyone who's playing the game gets the app and you have to sort of walk around and search for my lost items and so find them for me as simple as it sounds. So what is what sort of drives this whole app, right? So there's three basic parts of it. So there's the client app. That's what you'd be giving to anyone playing the game. So they'd be using that to hunt for your virtual objects. There's the admin app that sort of drives the whole hey, this is where I'm going to drop or hide my virtual objects for everyone to find. And then there's the web backend service, of course, which includes the Azure Spatial Anchor service itself, as well as TV service that sort of drives the whole sharing and storage of the different anchor points. We'll be looking at all of it. So I'm going to go deeper into each of these buckets. So what does the client and admin app look like? So if you'll go look at the code base itself right now, it is a super, super first disclaimer. It's completely edited off the current Xamarin tutorial for special anchors and that's the beauty of this whole demo. I just wanted to show to everyone that if you want to make your own game, you can do it all. The samples are a great starting point. All you have to do is sort of edit it to your need and you're good to go. It's amazing. So the client and admin app right now, the way it's structured is of course you have the Spatial Anchors Nuget because you need to use that to talk to the Spatial Anchor service. We're going to be writing it with Xamarin and we are going to be using Azure Web Service to sort of host the API that is going to talk to the Xamarin app and the Spatial Anchors and the Cosmos DB to store some extra data that we need to drive the app logic. So the next thing you'll see that I have broken it up into Xamarin iOS and Xamarin Android. So due to the nature of this app and the fact that it's very, very tied to the native AR kits and AR core provided by each platform, I decided to go with using Xamarin iOS and Xamarin Android natively to write these apps. We could totally have gone with using Xamarin Forms and done native embedding. I just thought I just found it easier just to maintain them separately and just not go into the headache of embedding it right now. And each iOS and Android platform further pulls out as I mentioned before. So we'll be tapping into ARKit and SceneKit for iOS and we'll be tapping into AR core and SceneForm for Android. So that's sort of the high level structure of how the client and the admin app sort of looks like. Let's see what the web services look like. So in Azure, we're going to be setting up three different types of services. So the first off, you'll have to obviously initialize your spatial anchor service. The system is currently in preview, which is and therefore it is currently free to use. The second is a Cosmos DB. So we're going to be using Azure tables. So there's a two purpose to it. The first part is we need to store sort of like the unique key per anchor. So that every object that you're creating, we're going to call that an anchor. So each anchor that you create, there's a unique key associated with it. So that's going to get stored on this tables. And then the rest of the logic that drives my game is tied to the Azure table. And we're going to create an Azure App Service, which will have the API that lets me sort of talk to this. So we're going to create an Azure App Service which will have the API that lets me sort of talk to this table. So yeah, that's literally it. That's all you need to get started with a very, very simple and basic version of a simple treasure hunt. So what does it look like? So I kind of went down to different parts and figuring out how to do this demo. I have both parts ready. So I'm going to right now what I'm going to show is a prerecorded version of the admin app drawing an anchor and the client app finding an app. If we have time at the end of the presentation, I have my phone hooked up and in this tiny little phone mode that I'm sitting in, there is one anchor right at my feet. And if you want, I can totally show that to your live and we can find that anchor right now. So let's have a look at what the admin app looks like. So the admin app's point, let me get this playing. The admin app's point basically is to allow you a way to drop these anchors. So this is AR, so drop me anchors. A bunch of pirate jokes. What you're looking at is me dropping an anchor near my desk in the SF office. I put in a little description for the anchor being dropped because the way my game currently works, the way I have architected it is I provide you a location associated with an ID number. And when you go and look for that anchor, you have to put in that ID number that you're looking for. So anchors away, friends. We have our first anchors dropped. Let's move on and check out what the client app looks like. So in the client app, this is what you'd be sharing with your friends who want to play the game. So they can just go, they'll have to go in. So let me play this video so you can have a look. So they will basically be going in, getting a list of all the potential anchors. So we're going to look for the sixth one that's near my desk, which we just dropped in the last video. And once they're in the space, it's going to generate the anchor for them and they can find it. Super simple, super easy. Just logically, if you just think about it, it's very, very straightforward. There's nothing complicated going on over here. So let me, this is not what I want to share. Let me show you what the code looks like. So I'm going to switch over to Visual Studio. All right. So if you clone the repo right now, you will basically see this solution structure when you open it up. There's, like I said, the three main buckets. The web service, the client app and the admin app. If you also go and clone in the Azure Spatial Anchors Xamarin, you know, sample apps, they kind of look very similar. It'll look a lot similar to what I'm showing right now because as I said, I want to show you that it's very, very easy to get started to make simple games like this. And yeah, I basically just modified the sample code right now to make it work for this game. So the first things you will be coming in to do is in the web sharing service. You'll have to come in and put in your Cosmos DB connection string. And that's all the setup you have to do to get your web service going. So right click, you know, hit publish, publish to Azure. You know the process from here. So let's have a look at the two important files that I think are important in this web service. So the first one I like to show you is the anchor controller. So this is basically what sort of drives that API I was talking about. So we have, so there's get anchor async with the anchor number. So this is the way the logic is currently driven. So you provide it with an anchor number. Like in the video, I drop anchor number six and I search for anchor number six. So right now I don't have it in a very complex state where you know, just giving a map coordinate or just getting into a space finds the anchor for you kind of like how Pokemon Go works. But trust me, future versions of this app definitely have it. And then another important one is the post async. So this is basically what the admin app taps into and this is what lets you post your description and the anchor key to your Cosmos DB database. So that's the only, literally the only important file in this whole anchor in the sharing web service bit. Super easy, super simple. And like I said, it's exactly the same code from the sharing service in the Spatial Anchors tutorial right now. So haza, thank you Spatial Anchors team. All right, so the next bit I'm going to show is when we collapse this away, we don't need you right now. I'm going to open up this zamegonhunt.com. So basically what's in this common folder is you have to come in here and obviously set up your account key and your service URL for that API, etc. The tutorial walks you through this. It's super easy to get this all set up and going. The next bit is basically these three files. These are basically just wrappers around that URL, the API basically to sort of send and receive anchors and all of our Xamarin apps are going to interact with this one common project. So hey, look at that awesomeness of Xamarin. So it has three basic files, one's the actual sharing service, one sends the anchor and one receives the rest. It's just a simple basic HTTP wrappers around that cool looking web service that we have set up over here. Let's go into where the important stuff comes in, right? So I'm going to close this. Let's take a look at the actual Android and iOS apps. So these two folders are the two apps that drive the game. So the admin app is the one that I used to drop me anchors madeies and the hunt app is the one that you share with me playing the game. That's the one you use to find the anchors. So let's take a look at the drop logic. So I'm going to open up Xamarin drop and let's look at what we have here. So in this there is a base, so it's like a demo base controller that sort of drives the whole view controller that I have right now. In this the most important thing is, if you notice this over here, it's called the ACR in view. So this is what I was talking about. So this is the frame that in the demo, if you notice a little dots that were appearing, that's driven by this basic frame. So that's it. This is the camera provided by ARKit and that's the panning around the screen and it grabs all of the features for you. To set this up, it's like you just initialize a new AR scene view. Again, all built in nothing special and has to be set up for this. You have a couple of delegate functions that you need to set up. Let's take a look at what those basically are. So this is basically just the logic that drives the scene view to just tell it that, hey, grab each current scene and sort of just process each frame. And it's the one that sort of collects all the data points that will be needed by the Cloud Anchor Service to sort of update and understand where to draw your anchors, et cetera. The next bit that's important in this is let's scroll down a bit. Also, awesome hack. I always forget what are important parts of a code to show. So I just draw breakpoints, whatever I think is important. Cool. So the next thing is what is... So this is the line that sort of starts and configures the ARKit natively in iOS itself. So we had the view set up and this basically tells AR system that you can start tracking the system now, grab, this is the plane and just go for it. So this starts the session and gets all of the magic going. Again, you don't need to really understand too much about the system unless you're doing something really, really complicated. But for a very simple version as the app is right now, this is all you basically need to know. Let's scroll down a little bit more. So this little section over here, like my awesome helper comment, Desi, this is what we used to talk about in Azure Spatial Anchors. So talking to Azure Spatial Anchors is pretty much straightforward. Once you have the account key and the ID set up, you basically have a couple of delegate events to hook up things like what happens when an anchor is located, what happens when your anchor session is completed. And all you basically do is just start the session. So initialize it, give it the values that it needs and just start the session. That's basically it. Nothing else complicated, you need to provide to Spatial Anchors itself. And all of this comes from the handy little nugget package that is just the one small reference you need to add here. And boom, that's it, you're done. And then let's see. And the last bit that's happening over here is this is how we draw that box that you see on the screen. Again, there's not too much rocket science basically that's happening over here. All it's basically doing is grabbing a visual anchor location and then we create a SCN box which is an inbuilt geometric shape. So you have basic geometric shapes like boxes and circles provided for you. If you want to create something more complex, you have to sort of, you can embed your own personal 3D models that you want to sort of render on the screens. And yeah, so I just set up the box, give it a color and I told give it the visual anchor position to generate the box in and that's it. That's all you have to do. So that's all the place cube it's doing. And the other two things to look over here is so like I mentioned this is the admin app. So this is the primary function of this app is just to create anchors. So the way the anchor creation happens is right over here. So all you're doing is making sure the anchor is not null. And it's found location and all of the information that it needs of the scan space and the actual anchor location. And all we do is we send it over to our spatial anchor session, create anchor async and we just give it the cloud anchor parameter. That's it. Simple, super easy, super straightforward, very, very easy to do. And this is basically what creates the admin app. So this is all we're doing. So since I showed you how the iOS stuff works for dropping an anchor, let's quickly take a look at the client code on how to hunt for the anchor and this time I'm going to show you how Android works because that's also very, very interesting to see. So let me close this up. So now we are in this folder, the client folder. So the main purpose of this is we will be looking at how we're going to locate those anchor points. So in, okay, so what was I looking for? So this is, yeah, so this is the activity in Android that drives that camera view similar to iOS. So let's go down over here. So this is the AR fragment that is similar to the camera view, the ACRN view in iOS. This is the AR fragment that drives it on Android. Again, super straightforward, initialize it and set it up to that frame and it passes and just set it up to provide each frame to spatial anchor so it can collect its required information and that's it, you're done. You have the camera setup, super simple, super straightforward. That's the one layer that you work around with. In Android, let's look at what the setup looks like. We are now using this for just searching. So this app only locates anchors so the way it does it is if you remember from the video I entered in the anchor ID so we basically take in that anchor ID and on Android we have a special separate separate class setup that does the talking to spatial anchors. So let's pop on over here to see what's going on. Similar, you'll notice this is very similar to how it was set up in iOS so we're setting up the spatial anchor session giving it the account key and ID. So this is what it does to start locating it so depending on what information was provided it from the response it will come in here, create what we call a locate criteria which just sort of has all the parameters it's supposed to be searching for. So it starts looking for a check and it starts looking for it and once it is found it generates it basically creates so sorry this is how the criteria is set up it's basically coming from the AR fragment itself again you're not doing anything fancy here you're just giving it parameters it needs. Okay this is what I'm looking for I'm going to go down here I think is where I have it basically triggers the session complete and that on the Android side will just go in and draw that box on your screen on that AR fragment. So that's literally it that's all that's happening here so again super straight forward like I said it's I just went in and sort of just edited what was already there in the sample code and sort of just made it work for myself and I think that's the standard knowledge but not that much as spatial anchors knowledge so it's sort of just sort of found a nice medium in between so it's sort of a quick recap or I think it's sort of important to sort of remember if you are working in this space so anchor visual object you see this word turn up in the documents is basically that object like the box in this example it contains information of the position and the position of this mobile app each platform you tap into its native views and you render it frame by frame and that is also your canvas on which you place and remove the object. So your spatial anchors layer is can be shared between your two apps as you saw I have it in a common folder and that's what you use to send the object to your cloud TV and back and to create a shared so you can share it with all of your friends but that's no fun why would you want to play a treasure hunt game on your own that makes no sense. So let's make it a shared experience so to share the experience we're going to pull an Azure app service put in a Cosmos TV to share to have a safe space for all of those anchors that I'm dropping and then I can share it with all my friends or you can make it Adel Access Control and only allow specific logins to come in and use an app and there's a lot of great ways to implement this. You can have a treasure hand game that talks to IOT devices and that triggers location based notifications on your phone to let people know hey then you're a potential treasure hunt point. So you can get very very creative with this. This is a great sort of starting point a starting solution to build up on this. So yeah. So thanks for listening and these are all the links. I'll share this presentation as soon as we're done probably even put it in my GitHub repo in case you want to access these links. All these links are on the repo as well. Yeah. And thank you for listening again. I'm going to pause here for questions. We can talk about. So rest button pushed ask. I wonder what the benefit of using Azure's Azure Spatial Service versus building your own besides the time to market. Is that the only benefit. The major benefit that I see is they handle a lot of sort of rendering through each frame and collecting all of that spatial data. You don't want to sort of do all that. Unless you're an expert then please go ahead or anchors away. That's right. Yeah. But I just the service is super simple to use really easy to onboard and they handle all of that overhead for you. So you can just concentrate on the rest of the architecture of your game. Like how do you want to interact with it? What does your UI want to look like? Things like that. Cool. Awesome. So the next question and they're kind of they're one big thing is that when using the client aka the Hunt app do you need an Internet connection? Right. Okay. Yes you do. Unfortunately the service talks to that Azure service the spatial anchors is an Azure service so you do need to have it. You could do like I mentioned you could do like a native embedded version of it is make your app fatter and I don't think that's a nice experience if you do it with online you can use so many more sort of services and make the game more interactive. And that was the next question is like if you know can you can you change it to make it work offline. I'm going to go here. Exactly. You totally can. You can make a complete offline version of this if you want. I just wouldn't recommend it just because I'd like it to be a little more interactive and rich and you'll notice right now it's a very basic implementation of you know what the game can be. I just wanted to give everyone like a starting point but you can completely adopt it fork it and make your own versions of it make it fancy or make it cooler. I myself be iterating on it as I get time but yeah I mean I can totally see the vision of this working out hundred percent. Gotcha. So another question here that I just saw come up from Wesley Waffles is that can you for the anchors can you orient them orient them like is the north south east west or how does that work. Okay. So great question. So if you go into the spatial anchors page like if I can pull it up right now and not mess up the setup. All right. So special anchors they have a play in the hole. How does the location work. How does it understand how to render it. It currently is very localized to the space that you scan. So it's very relative to the carpet versus the wall versus the table and it it works off edges that it finds. Gotcha. So it kind of figures out it doesn't do like a GPS or a location base sort of information gathering basically it renders all of like these edges and you know points on in this space and it's really smart in the sense that I was able to drop an anchor on the table and on the floor and it treated both differently. Cool. So yeah. So it's the system is a little different on like the traditional ways of sort of like geolocation base. This is a little more spatial based kind of like how the connect sensor works. But that's that's the sort of a way to sort of correlated it's not exactly the same. But yeah, I'd suggest totally coming in and like even more guidelines and understanding or how to sort of conceptually you could make more complex and complicated apps. Excellent. Well, thank you so much for taking the time to talk about Mixed Reality and Samarin how easy it is to get started with an application very quickly. So thank you for that. Everybody we have our last speaker coming up where we have Rachel talking about making accessible websites for everybody. So what we're going to do typically she's actually in the studio a longer pause so we can move stuff around. But please stay tuned we will be right back.