 I have a presentation here to talk about contextual awareness solutions that our team has been building with various products that we have in SD MicroSportfolio. Some part of the team that builds these solutions around accelerometers, gyroscopes, microphones, environmental sensors like pressure sensor, all of those. We have existing things that we have all around us, devices that can interact with human beings, the devices such as mobile phones and trackers that we use. What we essentially are trying to do is to put solutions together which recognize the context. Your watch on your wrist is a device that gives you the context, the time. Your GNSS system in your car gives you the context, your location. And then after that, what is done is you take action, and which is where your magic is, which means you can build solutions around context that can be provided to you. So there are diverse applications in all kinds of domains, smart me, your fitness trackers, starting with that. To smart home, the one that very intelligently manages the energy consumption in the home, to smart office, to industrial application, healthcare application, fitness, gaming, virtual and augmented reality applications, and this goes on and on. Now with a lot of work that is being done in the area of human machine interaction with machine learning algorithms, with deep learning, there are so many more applications that suddenly become possible. I will talk a little bit more about all of these in this talk. So I'll start with some examples. Some examples for healthcare applications. So if you can just consider a simple device like an accelerometer, it can monitor periodicity in acceleration and thereby you can determine if the person is walking or running or jogging or stationary. So this essentially has been utilized by many companies that commercially sell fitness trackers. You can also imagine applications, context-aware applications that can actively support elderly or disabled people in performing their everyday activities. You can work with systems that can potentially detect potentially dangerous situations in a person's life. Like a man-down application, like if a person has fallen down to be able to detect that a person has fallen down and send help based on the location of that person is one such application. Industrial applications, there are quite a few. Some examples are like you can make activity-aware applications that can support workers in their tasks. If you can imagine putting together the electronics in a cockpit of a Boeing 787 aircraft, it has so much of electronics in it and the sequence that you need to follow in putting all of those components together, if the environment around you can monitor what the worker in that cockpit has been working on and show the worker an appropriate user manual because things are so complex now, show the user an appropriate user manual based on the step in the assembly process that they're at. This is another one of those applications that can benefit workers in industrial applications. Entertainment and game. So now with motion-memes, accelerometers and gyroscopes and magnetometers that you can wear on the body, different locations on the body, feedback can be provided to athletes or to performing artists on their performance. In case of the golf club swing, there are applications that are available that essentially can help the user improve their golf club swing. And those are all using motion-memes. There are many other applications like you can already see trends where context-aware applications can do mood recognition and bring up the appropriate playlist. So you can imagine that as one of those situations where contextual awareness really helps the application. To understand the context, there are a few things that we just also go through here. The real value is in recognition of high-level context. So what do I mean by that? If you recognize that the person is in a shopping mall and is shopping versus person having lunch with friends at the shopping mall, those two are two different things. And the context here is different. Versus a person that's busy at office doing their work at their desk. So this is high-level context. Now, high-level context is built on determining low-level context components. And those low-level context components, the more accurately you can determine those, you can have a very good estimate of high-level context with a greater degree of accuracy and reliability. So the low-level context you can imagine are like human activity, human environment, human location, human interaction. So human activity is like you can imagine person is walking or stationary or person is sitting at their desk. Human environment, am I in a conference room or am I in a shopping mall? That's human environment. Human location, again, the same thing. And human interaction is the person talking to another person or is the person typing on the keyboard. All of these can be detected. So let me just then, with that, present a generic context awareness architecture. So there's quite a lot of stuff on this. But what I want to describe is everything starts with data or signal, which is the outermost layer here. So your signal could be generated by an accelerometer or a compass or a microphone or any radio signals like your GNSS or from camera or your light sensor. Even clock provides you the data or the signal. So now you take this data or signal and then this goes through the information layer. So you would take the accelerometer data and try to extract information from it, information such as periodicity in the acceleration pattern. Or you can look at variance in acceleration, which determines what is the user doing. So then this can be movement pattern. From microphones, you can determine noise patterns. From radio signals such as GPS or Wi-Fi, you can determine your user location. Camera can give you brightness. Clock can give you your time, date and time. And also provide you calendar or activity list. So now from the information layer, you go through the information processing, which utilizes these pattern repositories. Pattern repositories are the ones that have been built where a lot of analysis has been done on the data. What does the acceleration pattern look like when a person is walking? Or what does the acceleration pattern look like when your mobile phone is sitting on the dashboard of a car? So all of that sits in a pattern repository. So this information layer essentially will do information processing and convert this information into knowledge. And what does that mean? What it means is if I take the movement pattern, I can convert that into a movement tag. So looking at the acceleration pattern, I can say the person is walking. So that's a movement tag. Similarly, from microphone, you can say noise tag. Am I in an environment which is that of a conference room? Same thing for location. Again, you can create a location tag. This is a location corresponding to Santa Clara Convention Center. All of this goes into the context detection service. And context detection service is, of course, driven by a context repository, where you can combine information from multiple sources. And ultimately, you generate the context, and which is service through context APIs. So taking this and applying that to human beings, what we want to do now is to understand human activity. So human activity, if you look at here on the right hand side, here is your time scale. It can go from seconds to minutes to hours. And it can be broken down into multiple such components. The very first one is motifs, or gestures, or movements. Those are of the order of seconds. So you can distinguish activities such as bending the arm, or taking a step. Those are the activities that fall in the first class. The second one is low-level activity. So you can put together pieces of information from the first set to create low-level activities, where you can determine the person is walking or person is cleaning the windows. And then you combine these and go to the higher-level activities, which can result in determining what is the person doing, like person is on his desk typing on the keyboard, or person is shopping at the mall. So those could be of the order of hours. So how exactly do you make this happen with some of these sensors? So I'll start with some examples of that. So for human motion detection, it is an architecture that can be utilized. You can start with a variety of sensors. And these sensors are like accelerometer, pressure sensor, gyroscope, or a microphone. And the information from accelerometer, or pressure sensor, or gyroscope, or a microphone is processed through some windowing and feature extraction scheme. So you would look at a window of a two seconds worth of data, or three seconds worth of data, or a half a second worth of data. You do feature extraction. What it means is that you would obtain your mean, or variance, or number of peaks in that signal, or time difference between two different peaks. All of that is feature extraction. So once you can look at the acceleration signal coming in, goes through windowing and feature extraction, through this process, once you have the features available to you, you can compute the probability of a set of activities that are possible to be detected through acceleration signal. Now, you can do the same with other sensors, but you can, in this case, also combine them. Accelerometer and pressure sensor can be combined. The information from these two sensors can be combined. They're complementary in nature. Same thing, accelerometer, pressure sensor, gyroscope can be combined. If you want to determine the orientation, you can combine accelerometer and gyroscope. If you want to determine the context of the person, is the person walking or person climbing stairs, that can be done by combining accelerometer and pressure sensor. Pressure sensor gives you variation in the altitude. And then you, of course, can combine microphone. It's fairly simple, because let's say a person is walking in a corridor. So the periodicity that the microphone can detect in person steps can be combined that with accelerometer. And you can improve the accuracy of determination of context through these signals. So this was the emotion. Now, what about human environment? Human beings, for example, in this conference room, how do you determine that the person is in the conference room? So same thing. You look at microphone data. You can process that microphone data through some windowing and feature extraction scheme based on the features that you extract. Those features could be like MFCCs or delta MFCCs, whatever they may be. And you can compute the probability for your environment features. And in our framework that we have put together, what I want to show you is what are those possibilities? So for motion activity vector, you can imagine a vector that looks like this here, where your activity could be stationary or walking, jogging. The person could be on escalator, elevator. Person could be on a bicycle or driving, or none of this. So the most important characteristic of this is that each of these elements in the vector, they're sort of mutual exclusive. You cannot be stationary and walking at the same time. Only one thing can happen at a given point of time. You cannot be bicycling and driving at the same time. You're either bicycling or driving. For voice activity, here are some examples. You could have a situation like silence or face-to-face conversation, or a person talking on the phone, or none of this. So none of this is a class which can be expanded. You can put more and more as you have greater number of sensors that are available to you. Same thing for your spatial environment. You can have spatial environment detection like, are you on a street? Are you in a stadium, or is the person in a retail environment like shopping mall, or cafeteria, or restaurant, or is the person at home? Is the person in a moving vehicle, all of this? And then you have none of these. So based on the richness of algorithms, none of this essentially can be as small as you want it to be. So for these three vectors that you can determine at any given point of time, what do you ultimately want to do with it? So what you want to do is to combine the time sequences of the probabilities for each of these vectors to ultimately determine and produce a soft decision. Because remember, in these elements, you can assign a probability at any given point of time. So the element that has the highest probability is your actual activity. So let's say the person is walking. Your vector would essentially contain probabilities for all of these elements, but the one that corresponds to walking will be the highest. Since this is probabilistic, it is different from hard context. This is soft context. So you would have vectors like motion activity vector, voice activity vector, or spatial environment vector. You can combine all of these to ultimately determine high level context. So some notions on what does it take to build a good solution. So number one is the solution should be accurate when you try to predict the context. And how can you achieve greater degree of accuracy when you put the solution together? You should conduct testing with very large number of users for diverse scenarios. So it goes without saying. But I just want to list them here. The solution should be low power. If you need to keep charging a device every few hours, the user experience is not all that good. So smart sensor usage, use of hard acceleration, and if possible, do all the computations on the sensor itself. Again, another attribute for a good solution is low footprint. The solution should not basically require a large computing resource, because many of these solutions are implemented on devices that are like mobile phones or variable devices that have limited battery life and limited computing power. Well, I shouldn't say limited. Nowadays, mobile phones have eight processors in them. But the thing is, battery life is still a finite amount. And you would want to extend that as much as possible. And flexible. Flexible means that you should be able to implement your solution on a variety of platforms. Contextual awareness implemented on a mobile phone, you should also be able to implement some piece of it on your available device, or on your tablet, or your laptop, whatever the device may be that you're carrying, which has sensors plus computing capability with it. One of the other important aspects of this is personalization. The way I ride my bicycle is different from how somebody else would be riding their bicycle. And to improve the accuracy of detection, if we put together algorithms that allow you for personalization or for tuning of parameters that improves the performance, then it's always a very good idea. Because the user experience is enhanced when you have capability of personalization in your framework, in your algorithms, in your architecture. So we basically have a variety of solutions that are available. And these are available on Open.memes. These are available for free, essentially. So some examples of this, I'm listing them here. So they can be used to create your own complete solution. So the very first one is sensor fusion for device orientation. You can combine accelerometer, gyroscope, magnetometer data to determine the orientation of the device, to determine linear acceleration, to compute your quaternions, to determine the calibration status as to how well are your sensors calibrated. Another one that we have is a user activity recognition library, which can work for both mobile or wrist worn cases. It will determine travel mode, like is the person riding a bicycle, is the person walking, is the person a pedestrian, or is the person in a vehicle. We also have features where it can count squats, or pushups, or bicep curls. We have features like that that are available on Open.memes. Gesture recognition, that's another library that's available. It can form a building block for your contextual awareness framework. And this gesture recognition library can detect glance, like you pick up your phone and look at it, or you shake the device, you swipe it, you rotate it, so your device can be used as a remote control, essentially. So if your mobile phone can detect this orientation, I can use that to turn the volume on or off. Post determination, this library can help you determine whether the person is standing, or sitting, or lying down. And carry position determination library. So this library can help you determine if the mobile phone is in your pocket, or in your handbag, or in the holster, or the trouser pocket, backpack. So these scenarios can be utilized in building better contextual aware solution. So some more stairs. You can count the number of steps the person has taken while climbing up the stairs. So this library will only count steps when you are climbing stairs. It would also detect if you are on an elevator or escalator. It would be able to determine the floor change. We have a library that does bicep curl detection and counting. We have a solution that we have put together, which gives you situational awareness, combining audio and motion memes. So this is for environment detection. We are eventually going to add things like keyword spotting to it, and a number of speakers that you might have in a conference room. And we have a demo here. In fact, many of these are being demoed here on the show floor. We have a demo for predestined and reckoning. So this is a solution which can enable indoor positioning through use of motion and environmental sensors. So to talk a little bit more about our OSX Motion FX, this is a library that's available from open.memes. It combines the data from three sensors, magnetometer, pressure sensor, and accelerometer. And it would give you quaternions. It can give you your magnetic heading. It will give you your pitch roll and your angles or your linear acceleration or your gravity component. It has built-in your magnetic calibration library. It has gyro bias compensation built-in. It has dynamic distortion feature built-in, which means the hand jitter is already compensated for. And the library is fairly efficient. So it can be implemented on mobile phones or even wrist-worn devices, where the computational requirement for this is not that large. So this is, again, available on open.memes. So open.memes essentially has many of these libraries where you can look at the documentation which describes the APIs. It has test results, how it has been tested, how to put this together, and some tips in putting this together. We have activity recognition library that I talked about. OSX Motion AR, it'll provide real-time information on whether the person is stationary or walking or fast walking, jogging, biking, driving. The feature about this particular library that we talk about most is that it uses only accelerometer data and at 16 hertz rate. So it's a very power-efficient implementation because it essentially consumes the smallest possible amount of current for accelerometer. And some test results for this are available through this confusion matrix that I want to show you. So we tested this library with 71 unique individuals. We have 48 hours of activity data in this. And the confusion matrix here shows this is the classification output, and this is the actual activity that the user was engaged in. And what you see here is that most of the activities such as stationary walking, fast walking, or jogging, it's better than 90% accuracy for biking and driving. It's better than 83% or 86% accuracy. We have tested this library for use cases like the device was in the cup holder in the car or in the shirt pocket or the device was in trouser pocket. In case of bicycle, we had the device in shirt pocket or in the trouser pocket, many such scenarios. It has been tested. And the aspect that I want to emphasize here is that this is primarily accelerometer only. So if you remember the architecture that we had put together, you can combine accelerometer and pressure sensor, gyroscope, and microphone, and you can further improve this. But what we are showing you here is that the level of accuracy achieved with just accelerometer, which is sampled at the lowest possible sample rate of 16 hertz. We have a gesture recognition library designed using machine learning algorithms. It can detect things like glance, pick up, wake up kind of gestures. Again, available on Open.mem's APIs for this are available, fairly easy to understand and use. Carry position library, OSX motion CP. It can detect whether the device is on the desk or in hand, like if you're texting or near the head, like as if you're talking on the phone, shirt pocket, trouser pocket, arm swing. It's again driven by only accelerometer and has been put together using machine learning algorithms. We have post determination, which can distinguish like standing or sitting or lying down. It uses two sensors, a pressure sensor at 10 hertz rate and accelerometer data at 1600 rates. So the reason why we bring in pressure sensor is to distinguish lying down kind of use case scenario and to improve the accuracy of detection when you want to distinguish between sitting and standing. The performance latency for this thing is less than two seconds, so fairly fast response. So some of the scenarios in which we have collected data for this include when the person is standing with their hands in their trouser pocket and standing, or head resting on the hand, like sitting down, hand resting on the side of the couch while watching television. So essentially what I'm trying to say here is very diverse use case scenarios and a very large number of users that have contributed to collecting the data for this. So in this case our probability of detection for sitting is about 97% and probability of detection for standing is about 88, 89%. This essentially is one of those libraries that, again, just like any other library, you can try it out on our nuclear board or our sensor tile. The APIs and the information that is available to you to integrate it is fairly self-explanatory. Then vertical context determination library. So this is where you can distinguish is the person on an elevator or escalator, is the person taking stairs, climbing up the stairs, going down the stairs, is the person walking on the same floor. Because if you notice, these solutions are built with pressure sensors. And pressure sensors have inherent phenomena that you need to take into account. One example is if you're standing near an air conditioning duct and if the air conditioning kicks in, it would show a spike in pressure. So the algorithm has to be smart enough to distinguish that the person is on the same floor and that the floor change did not occur when the air conditioning kicks on. Or if you walk in, and if you open the door, or if you walk into building, that is air conditioned. So when you enter the door, there's a change in pressure. So that change in pressure should not be you. The algorithm should be smart enough to know that you're on the same floor, that your floor or your altitude did not change. We have another interesting library for fitness kind of use case scenario, which is bicep curl counting. It uses a pressure sensor and an accelerometer and samples them at, again, 25 Hertz rate. And it will detect bicep curls. This is for, of course, the case where the device is worn on your wrist. And it will detect bicep curls. And it will count the number of repetitions. So a library that, again, is based on machine learning algorithms and can be tried out using a nuclear board or sensor tile. So if you have to build your own application, you can start to do that in less than five minutes, essentially. So these libraries are available. And Android and iOS apps are available for GUI. So here is an example here. So essentially, if you take your sensor tile and run this app on your Android device, what you notice here is essentially your temperature measurements, your pressure measurements, your humidity measurements. This one doesn't have a light sensor, so it's not present. So essentially, with the available apps, you can put your solution or your proof of concept, if you want to call it, together very, very quickly. You can put this together in about five minutes. So the hardware components that you need to put this together. So there are two choices. You can choose either the nuclear board or you can choose a sensor tile. These are all available from st.com. So Gerbers are available for both of these. BOM is available. Schematics are available. So if the solution works out for you, you can take components from the schematics that are available for these devices and use them in your own design. You can then download the firmware from open.memps. So you essentially, for the hardware abstraction layer and for the layer that sits on the top, you have the source code available. There are application notes available. There are user guides that's available to you. And something that can help you get started. Then you can download any of these libraries, like Activity or Carry or Gesture, many of these that I have talked about. And there is a simple click-through license that is available to you, which allows you to test it, build your proof of concept with it. If you essentially like this thing and want to go to production with this library, that's also available to you as long as you use ST's sensor solution. This license is given to you for free. So production Lua is provided to you once you determine the use cases in the library for your use case works the way you want it to work. Then you essentially install your app. Fundamental things that you can do is data logging or viewing the output, like for OSX MotionFX, which is your library for sensor fusion. In real time, you can look at your pitch, roll, and your angles. You can obtain your, you can look at your quaternion values. We have a cube demo, which will show you the movement of the cube in a three-dimensional space for your testing. You can test it for rapid acceleration. Or you can test it for a known amount of rotation that you apply to the device. So essentially, you have the ability to put your complete solution together using these building blocks. Then the other piece that we have here is, of course, your STM32 nuclear board. This is the hardware piece that I talked about. This is one option. So the STM32 nuclear board essentially has many of these features that are helpful in building your actually proof of concept. So it has a flexible board power supply. We can power it through USB or external source. It has two push buttons and two LEDs. It provides you means to actually interact with your algorithms to do testing with this. And what you see here is Arduino extension connectors. What it allows you to do is to stack up different shields to bring in different pieces of hardware. So you can create your own proof of concept or your application using a variety of sensors or microphones or if you want to bring in Bluetooth low energy to do any of the BLE kind of applications. Then essentially, the heart of this thing is your STM32 microcontroller. And you can pick the flavor of STM32 that you're interested in, that STM32 L4 is a very popular flavor, which is a very low power consumption microcontroller. And essentially, using this, you have access to all your MCU IOs. So if you want to write your own logic and do your first testing, this nuclear board provides you the hardware environment to do this. This is an example of a shield. So this is your sensor shield. It has a three-axis accelerometer and a three-axis gyroscope on it. So LSM60S3 and DS0 are available. Two flavors of this sensor shield are available. Then a magnetometer, LIS3MDL, a pressure sensor, LPS25HP, and a humidity and a temperature sensor. So this essentially can be plugged on to top of the nuclear board. Once you put these two pieces together, then you are ready to try out any of your libraries like carry position, sensor fusion, gesture recognition, any of those that we have available on OpenDOT MEMS pertaining to motion or environmental MEMS. So now the other option that we have is sensor tile. Sensor tile has multiple components on it as well. So it has your motion MEMS. It has accelerometer and gyro combination. It has environmental sensors, such as the pressure sensor on it. So the LSM60SM, this is our latest six-axis sensor solution. LSM303AGR, this is our accelerometer and magnetometer combo chip. You have a pressure sensor, LPS22HP, on this sensor tile. Sensor tile, as you can see, the form factor is suitable for variable applications. It's 13.5 millimeters by 13.5 millimeters in dimensions. It has a MEMS microphone on it. It's a 3 by 4 digital microphone. And you have your SDM32L4 microcontroller on it, which provides you the computational platform. And for connectivity, you have blue energy. So a complete solution, which is again available to you. It has an SD card slot for extra memory, which is where if you want to log data while walking, running, jogging, whatever, or audio data for any of your use case scenarios, you can log all of that. And fairly easily programmable. So all of the information that is needed and all of the components that you need, like open.audio, open.mems, open.rf, they will utilize this as a platform and building block for contextual awareness. So a little bit more about our free software licensing. So the idea is, as long as SD components are used, the license to any of these, open.mems, open.audio, or open.rf, is available to you. You can do proof of concept. And then if you want to go to production with it, you can get a production license for this as well. Open.mems, this is one snapshot from the web page on st.com, where this is an example where it shows your motion AR and motion CP kind of libraries. So you can see all the detail from this. You can order hardware and download software from this. So here is a list of the various libraries that are available to you, like motion GR for gesture recognition, FX for sensor fusion, carry position, pedometer, activity recognition, and so on and so forth. So the idea, essentially, as I have already mentioned, is to combine components from all these three. You can combine pieces that you find on open.mems, open.rf, or open.audio, all within the open.framework structure that we have put together to create your own proof of concepts or test environments for your own ideas for contextual awareness. So this is a little bit more. We have also our ST Blue Microsystem framework, which puts many of these components together and also adds connectivity to it. So ST Blue.ms is the one that will combine connectivity piece plus sensor fusion with it. So this is a screenshot from your mobile phone, which essentially is communicating with your sensor tile. And you can run your STM32 cube, you can run your cube demo on this device and actually see the orientation change as you move your cube, as you move your sensor tile. That's it. That was my last slide.