 ok everyone let us get started ah welcome back remember that last time we went over um perception of motion we talked about stroboscopic apparent motion and then we got started on tracking and in particular I am looking at head tracking and um we got into the case of integrating gyroscope measurements which are providing readings in terms of ah angular velocity so radians per second and in the two dimensional case I gave a simple example imagining a merry-go-round or a top spinning and if you get measurements of that here is the here is the example of the measurements just scalar measurements um essentially it is rotation with respect to the vertical z axis. So, you have a omega k hat at each step you have delta t which is 1 millisecond for our examples we have been giving and so we just have this very simple update we calculate some change in theta estimate and just start adding we add in each step here and just keep repeating this equation over and over again and then I showed you that the three dimensional case is very similar except that we switch everything to quaternions and so the replacement for this omega k delta t is constructing a quaternion that will um give you the change that occurs during delta t it is an entire um small rotation about some particular axis v k and the total amount of rotation is theta k so these together become delta q sub k hat um I put quat here in front of it to make it look like a a constructor from a class let's say suppose you have a quaternion class in c plus plus so just to emphasize that there is a quaternion that is constructed here and remember these two parameters come just directly from the gyro reading if you just normalize the omega x omega y and omega z components you would get the axis of rotation for here and then the magnitude of that original omega x omega y omega z vector will give you the total amount of rotational change estimated for that 1 millisecond right you remember that right. So if we do this over time what we mentioned at the end of class is that we end up with what is called drift error so the problem over time is called drift error so it is a gradual divergence between um the true orientation and our estimate of the orientation and these will gradually get worse and worse especially because um most of the errors that we have in this process are are systematic they are due to calibration um difficulties in some way so it may be that the the gyro along the x axis always gives you a reading that is an um overestimate by a certain amount and that will just continue to um to increase the drift error linearly. So if we think about drift errors um in 2D I could say that at each step the drift error let us put d sub k for drift is the true orientation theta k minus theta hat k in this case it is a signed drift error and in 3D we could say that the drift error in this case I want to subtract 2 orientations but if I am using quaternions I have to use the algebra of quaternions and we have to also remember non-commutativity. So I want to take my true quaternion minus the estimated quaternion and in defining this drift quantity. So let us suppose the true quaternion true orientation of the head if that is what you are estimating is q sub k then I want to essentially subtract off my estimate q hat minus 1 um I guess I could go in either direction with this as well um I will get 2 different results but they are both measurements of the error right. So depending on which way I want to sign I could have also written um q hat minus 1 sub k quaternion product q sub k right there is it is non-commutative it will give me an error in the essentially the other direction um is this ok. So, the concept here I mean normally for most problems you have dealt with in engineering you may talk about a scalar error right there is some real value to error it is non-zero um and therefore needs to be reduced or eliminated in some kind of way you may have considered a vector of errors before just keep in mind that the drift error in this three dimensional case here is an entire 3D rotation. So it is a quaternion and so if you apply the inverse of that you will be correcting the error and so this one it is a kind of an unusual quantity to have for an error. So, the errors that we get the drift errors tend to accumulate over time. So, drift error if left unchecked may grow out a linear rate may grow faster than that may grow slower than that it depends on a number of things, but very commonly if you go take plots of the drift error of a gyro after a doing integrations um you will very often see linear behavior especially for a fixed angular velocity. If you go faster slower the rate may change and these curves may um go up or down and then there is also noise factors in some as well if the gyro is very well calibrated then it may have a different behavior that is mainly dominated by um say white or Gaussian noise. So, what to do next is try to use other sensors to try to reduce this drift error back down as close as possible to 0. So, correcting for drift errors. So, two key points for correcting for drift errors one um use another sensor or sensors to provide a world reference. So, in other words some kind of reference information with respect to the world frame as the body is rotating taking its measurements from its body frame and then we want to somehow gradually apply corrections and this is a difficult problem it has to be fast enough whatever corrections we apply they need to be fast enough to fully compensate for drift, but the same time they need to be slow enough to avoid simulator sickness or other kinds of disorientation or discomfort. So, suppose for example, you are running along for 30 seconds and then you get a sensor reading from some other sensor that is not a gyro that tells you you are off by 30 degrees. Do you just all of a sudden apply that right or do you gradually apply it within the course of a second if you do that you you can perceive that the world is all of a sudden tilting the virtual world will be tilting. So, you have to provide it in a very gradual way it may be the case that your gyro is so bad and this drift rate grows so fast that there is no way to compensate for it without causing some kind of sickness. So, it could be the case that it is so sloppy that that you cannot compensate quickly enough and and still be within thresholds it turns out we are very fortunate in today's world of MEMS based gyroscopes that appear on our top of the line smartphones these days that the gyro quality especially if the gyroscope has been very well calibrated especially for temperature dependence and other factors. Then the growth rate ends up being slow enough so that it is possible to compensate for this without as far as we can tell without perceptual side effects causing simulator sickness. So, we are very lucky to be in this kind of regime today where we can in fact accomplish this. So, let me give you some of the technical details of how these corrections work. So, we would like to divide the air up into two components so separate rotational drift air into two components components one is called tilt air and this can be viewed as a combination of pitch plus roll right tilt air. So, has the world become tilted? So, it may be the case that my virtual world as I am doing the head tracking appears to have a tilt in it right. So, this can be any kind of pitch or roll imagine just a few degrees of it. So, tilt same kind of thing you might imagine in the real world corresponds to cases in which you are not aligned with gravity right where the surface that you are walking on it appears to be tilted a ball will start rolling in any direction in any one of a number of directions if there is a tilt to it. This is separate from another air which is the yaw air is the remaining component and this has to do with losing track of which direction you are facing right. So, the yaw is the remaining component we had troubles with this early on at at Oculus for example, because when people were playing a cockpit game. So, imagine you are sitting inside of a car and you have a dashboard or you sitting in an aircraft or spacecraft and there is a cockpit after 10 or 20 minutes of play the cockpit starts drifting to the side and you can sit in your chair and if you have a rotating chair you could rotate with it without even realizing and then before too long you are top tangled up in the cables because of this yaw drift air. So, it ended up being a very serious problem. So, the reason why they are divided into tilt air and yaw air is because the kinds of sensors we use to correct for these tend to fit very naturally into these two categories. So, I am going to first talk to you about how tilt air is typically corrected and then and then we will go in a little bit into yaw air, but it works very similarly all right. Now, to correct tilt air what we would like to have ideally we need or would like to have an up sensor. In other words a sensor that measures which way is up in the global frame that would be nice would not it because that is what the definition of tilt is. It is some mismatch with respect to up right some deviation with respect to up and then in the case of yaw air we would like to have some measurement of which way was forward initially and we would like to keep maintaining that. So, we need something that behaves like a compass. Now it turns out that in the real world we can never get either one of these very precisely we do not get perfect devices that will give us these. So, what we are going to use in the case of measuring tilt air instead of our up sensor we are going to use a linear accelerometer. And in the case of instead of having a compass in case of this we are going to instead use a magnetometer which measures magnetic fields which might include the earth's magnetic field and it may include many other fields superimposed together. So, let us suppose we have this perfect up sensor and it is accomplished by a 3 axis 3 axis accelerometer. So, it is measuring x, y, z components as meters per second squared and then I want to take a look at this in a particular coordinate frame. So, let me let me draw out a coordinate frame here have x, y, z coming out as usual and then I am going to draw on this frame and then I will say what the frame represents. Let me see here I have this vector here I will call it a with a hat on it representing some kind of estimate which is a measured acceleration. And this what I would like this to represent is my current estimate of my orientation, but the inverse of it applied to the accelerometer readings this should be the current accelerometer reading. So, in other words on the device on the sensor itself which is moving with your head I have an accelerometer it is measuring meters per second squared acceleration in the x, y and z directions. Let us assume just for a moment and I will return back to this assumption let us assume that your head is actually not undergoing some linear accelerations at present or the sensor is not undergoing linear accelerations. If that is true what would an accelerometer measure? Just just gravity right just acceleration due to gravity. So, let us just assume it is going to measure that component alone. If I take that measurement that measurement is going to be done in the in the body frame because that sensor is attached to the body, but if I believe that I have constructed a nice estimate q hat sub k of the of the transformation that will move the body into its frame from the world frame then I should be able to take the measurement and transform it back into the world frame using the inverse. So, that is what I do. So, I take the measurement I bring it back into the world frame and if that is if my if my sensor is correctly reporting up and if my estimate of this orientation of the body is correct then this yellow vector should be aligned perfectly with the y axis. Instead what is typically happens it will be off by some amount. So, the amount that it is off I will write as phi and this will be some estimate of the drift error. I suppose I should put a hat on that if we keep using hats for estimates right. So, if that is the case what I would like to figure out is what needs to be done to bring those two into alignment and so you can look at it either way you could try to bring the yellow arrow up to the white wire or rotate the coordinate frame y to line up with the yellow arrow. I need to take a look at what should be done here and so let us take a look at this a vector and let us let us write it out as a equals remember this is already transformed I have already applied this q inverse to it. I have three components a x hat a y hat an a z hat right. So, there is three components to it and now what I would like to do is perform a rotation that does not cause any yaw to change I just want to change pitches and rolls. In order to do that what I want to do is find an axis in the x z plane right. So, if I can find an axis and I always do rotations about axes that go through the origin. So, just imagine that I have I need some axis that goes through the origin it also lies entirely in the x z plane. So, when I grab on to that axis I can perform a twist that will bring these two together right. I will grab on the axis you can imagine is attached to maybe the the white let us say and it just performs a twist that lines it up with the yellow. So, that is what I want to do. So, easy way to do that is I just take this a vector and I project it down into the x z plane. So, let us try that. So, I project down into the x z plane that gives me a vector I will call it v tilt which would be the tilt axis oh sorry that is just a projection not yet v tilt sorry I need to do one more operation. This axis here is a x hat 0 a z hat correct. So, I just get a 0 component for the y. Now, I want to pick a perpendicular vector to that and since I am only really dealing with two dimensions now I want to pick a perpendicular vector. So, I just you may remember you can just simply swap the two coordinates and negate one of them right which one you negate will tell you which direction the vector points. So, I want to obtain the following axis. This one which should be perpendicular and I will call that one v tilt and that will be a z hat 0 a hat x minus a hat x I got to negate one of them in order to get a orthogonal vector. So, this will be the tilt axis what I want to do is grab onto this and perform a correction by performing a rotation. Now, as I said I do not want to perform the entire corrective rotation at once. So, let me give you a very simple way to do the correction and it is equivalent to many alternatives simple way to correct. So, use what is called a complementary filter. So, in this filter we will have what is called a gain or gain coefficient alpha greater than 0, but alpha very close to 0. So, for example, maybe alpha equals 0.001 um the particular rate of alpha the particular value of alpha is going to depend on the sampling rate which is delta t is 1 millisecond for our continuing example we have been talking about. So, now in order to correct for drift in each time delta t in each step we have now a corrected orientation which will be some quaternion we generate v tilt and at this point I could apply the entire theta here and try to correct for it. What is going to happen though is that it is very likely that my theta is a noisy estimate and it is too much correction to apply all at once for perceptual reasons that I talked about. So, if you try to apply this and I did some experiments like this in the code myself I tried applying corrections all at once you end up with a very unstable view the orientation is just changing wildly all the time. So, this is where we use this gain coefficient. So, it is alpha times phi. So, that will perform a very tiny rotation in each step that is making progress in the right direction and then it just continues like this you end up with a very stable representation of the orientation right you end up with a very stable view, but this alpha needs to be large enough so that it is applying the corrections fast enough to compensate. One of the things we did was perform experiments with a robot moving the sensor through space at different angular velocities to determine upper bounds on the maximum amount of drift error that could occur. So, that we could set the coefficient large enough to overcome that and luckily as I said it was still within thresholds. So, that it did not seem perceptible or sickening to the user. So, we got very fortunate with that. So, we apply this correction to the to the previous estimate q hat questions about that let us see. So, what I want to mention now is a problem with the accelerometer yes. That is a good question. So, I every time I go through this I have to go back and ask myself is it minus or plus based on the way I have set up all the coordinates right. So, one of the possible ways to change it from plus to minus depends on which way I have made this axis right. One of the ways it could be plus or minus depends on whether I am rotating A or rotating this and am I applying the inverse or the forward. I checked it this morning and I believe it is positive, but maybe we should go back over it again during the break and see if we find an error. So, unless you have a unless you are convinced it is minus. I think it is correct in plus, but again it is quite possible somewhere along the way I may have may have inverted a step. If you write the code for this, you very quickly see that you have made a mistake and you can go back and check each one of the components. Of course, I never suggesting coding just changing signs until it works because you will end up with quite a mess after doing this two or three times, but I believe it is positive in the way I have set it up, but I admit I might have made a mistake because there are several steps in a row here each one of which could flip the sign. I think in this case I am applying an additional rotation to move this frame towards the yellow. So, that is why I think it is positive. That is very good though. I was anticipating that question. If someone usually asked and I usually ask it myself and like have I gotten the sign right it is very very important to see that to check that. All right. Other questions? So, let us see. One big problem with this as I mentioned is that the accelerometer is not a perfect up sensor. Let me put that here I guess. It is a problem. The accelerometer measures the following vector sum. There is the up part which is 9.81 meters per second squared, but there is also the true linear acceleration. Let me draw it off in a different direction here. There is also the true linear acceleration of the body. So, linear of the body and so it is actually the vector sum of the two that is getting measured by the accelerometer right. This is what is measured. So, what do you do in that case? Well, you could invent some simple heuristics I have seen in some published papers for example, someone might say well, if the magnitude of this yellow vector the part that you are trying the measurement that you get if it is close to 9.8 within some small epsilon of 9.8 then assume that this part must be 0. Is that a reasonable assumption? Well, it is not bad except I should be able to move downward and that compensates for some of gravity right because if I were free falling I would completely cancel off this up vector. So, I should be able to reduce the up vector by moving down in some way, but I can also linearly accelerate while doing that and I should be able to get it so that I convince the drift correction system that makes this assumption that the world is tilted by a large amount right. So, you end up with this kind of situation of making some heuristics to try to separate out these two vectors so that you can get a reliable up reading. This particular assumption by itself will not work all the time it is reasonable in many cases, but you have additional information because you also have access to the gyro which is giving you the angular velocity. So, if you all of a sudden get an estimate of up that is extremely far you know it is very very different from what you previously believed up was in the global frame and it happened in a short amount of time right you can very easily also compare that with the gyro history because you are integrating the gyro forward and the drift rate is not too bad in a short amount of time. So, you could give very low confidence or rule out very unusual gyro very unusual accelerometer readings that seem to indicate that up is say 30 degrees off from what you once believed. Now, if the person continues moving their head very wildly and violently for a very long period of time there may be no good way to compensate and you may be able to kind of fool the system into believing that there is significant amount of tilt before it is able to go ahead and correct that. So, that is one of the strange outcomes of that. So, and it is not really the fault of any engineer in particular it is a problem of the fact that when we measure accelerations in some moving frame we always get this vector sum and there is no way around that and we are just trying to separate it out. We do not have the ideal up sensor and it no one will be able to make one right purely based on inertial measurements.