 Ok, welcome back continuing onward. In the previous lecture we were talking about perception of motion and we looked at various frame rates all the way down to as low as 2 hertz where you have the beginning of perceived motion this is stroboscopic apparent motion. We get up to 16, 24 frames per second motion picture industry standard also old CRT based television signals were on 25 or 30 hertz frames per second and eventually I talked about CRT monitors and how as monitors got larger there became noticeable flicker as people sit close to them because of the flicker perceived in the peripheral vision and there became minimum levels beyond which people tend not to directly perceive flicker but then it may still cause some discomfort headaches and fatigue until you get up beyond a certain level. Very often people quote 85 hertz as being sufficient for just about everyone. So, I want to talk a little bit more about problems with displays and our perceptions of motion and stationarity which is the absence of motion. We talked about how the brain perceives motion earlier before this topic in the last lecture. So, one of the problems we have with displays is display scan out which is that the the image that is produced on a monitor this goes all the way back to CRT monitors but it still happens on most nearly all of the screens that we use today is that the image is being scanned out in some kind of order line by line it does not have to be in the same order that it was back in the days when CRT monitors were prevalent but it is some kind of progression that goes line by line goes pixel by pixel and then as it is scanning out the pixels start switching. So, one of the effects of that is that if I am looking at some object that is supposed to be stationary in VR and I turn then because of the time it takes for the scan to go down. So, if you are at 60 hertz for example, then it may take up 15 milliseconds to complete the scan 16.67 milliseconds the total time from frame to frame at 60 hertz right. So, it is 1 divided by 60 1 second divided by 60. So, there is a blank time as well. So, but it is about 15 milliseconds going down and so, if you do that and you turn your head back and forth then this causes a waggle effect. In other words some object that perhaps it is supposed to look vertical, but when you turn your head back and forth when you have the scan out problem may look like this right. So, as you turn your head back and forth this object may seem to waggle side to side in some way undergoes some kind of distortion. So, that is one problem that we face and another problem that we face is even if you are to update all these pixels going in this progressive pattern the pixels themselves could take up to 20 milliseconds to switch. 20 milliseconds is a long time right by that time you are ready for the next frame. How long is it take to switch it depends on what particular kind of display you have if you have an LCD display it could take up to 20 milliseconds if you have an OLED display it may take only about 80 microseconds it could be much faster. So, one of the ways to help alleviate these kinds of problems is to use OLEDs instead of LCDs because you get faster pixel switching, but that may then lead to another problem which I will explain in just a bit. The waggle effect has to do with the scan out time if I have if it takes a long time to update the pixels and other thing that will happen is blurring it will look like this object is smearing in some way because the pixels can't change fastness from turning my head back and forth trying to use the vestibular ocular reflex in addition to the waggle effect there will be a blurring or smearing. This was very obvious in the Oculus Rift DK1 for example, if any of you had a chance to use an LCD display and you see very obvious blurring you see other virtual reality headsets with LCD displays you may notice a kind of blurring like that. So, you can switch to another kind of display to try to overcome the blurring problem. We still have the display scan out problem I'm sure people in industry are working on this as we speak to try to make complete synchronous scan out and updating of pixels so that you get one frame all together under the assumption that it indeed is necessary otherwise you have to show that it's not through enough human subject studies. So, perception of stationarity and smooth motion in VR in a VR headset VR head mounted display in particular I want to talk about. If I fixate on a point in the world and that point is not moving and I turn my head back and forth and I use my vestibular ocular reflex that point should be image in the same place on my retina right if I just pick one eye should be image at the same place on the retina. So, when I do that experiment of let's say I turn my head in one sweep then the position on retina of that point over time should look constant I guess what should happen in the real world. So, position on retina some fixed point should look constant. Now, what happens if I have a display in front of my eye and I do the same thing and I want to have that object that feature appear to be in the same place in the world. Well, I hope that my head tracking is working correctly and let's assume that's working perfectly and let's think about other kinds of problems. If it's not working perfectly if there's some latency being introduced or some other kinds of errors there's going to be additional artifacts on top of what I'm telling you today. So, let's think about what happens when looking at a head mounted display before we get to that in the process of getting to that let me just think about some different cases here. Just think about displays and I want to look at four different cases four different cases here. Here I want to have a world fixed screen in this row I'm going to look at two cases of a world fixed screen and down here I want to have a head mounted display in other words a head fixed screen and then on this side I want to consider the case of pursuit. In other words that's the kind of eye movement that's going to be happening and in this case I want to consider vestibulo ocular reflex. So, let's think about pixels on a display. So, I'm up here I'm looking at a screen. So, I have some some pixels I have an object that starts moving. Of course I haven't drawn that right because I should be drawing a blocky version of this correct. So, I'm just trying to illustrate things here. So, I did not draw a blocky version try to make it simple, but it should be blocky and in the case of pursuit. So, suppose I'm sitting at the movie theater that's a fixed screen and an object starts moving through there right. So, the pixels will have to be changing and this is the place where people are arguing that maybe modern movie should be 48 frames a second instead of 24. So, that these motions become smoother and more natural like they are in the real world right. So, we've been accustomed to watching movies at 24 frames a second. So, if there's some very fast motion that comes across the screen it might not look right. And cinematographers, videographers, people who make produce movies know this and they be sure that they take care to avoid these kinds of artifacts and problems whenever they're constructing movies. Now, if I'm over here and you're sitting at the movies and again I guess maybe sitting at the movies isn't the right comparison because in that case I might imagine there's some kind of analog process going on through the projector and it may you know it's hard to see pixels. It depends I mean if I project a digital screen I certainly see pixels on the screen if everything is clearly in focus, but because of the focus part maybe it might look like it does some kind of a conversion to analog if you like. So, I can also just say looking at a fixed screen like an LCD monitor like we have in these rooms right. So, if that's the case imagine again I have my fixed monitor let's say suppose the monitor has some object on it and it stays in one place I decide to look at the monitor and I go back and forth looking at the monitor move my head back and forth using my vestibular ocular reflex. Do the pixels need to change while I'm doing that or the pixels are not changing. So, this is very similar to the way a stationary object would appear in the real world except for the fact that I have quantization due to pixels, but it doesn't have a problem with regard to changes over time. So, anything has to do with how the pixel gets scanned out how the pixels get scanned out from frame to frame the frames per second doesn't seem to matter too much here. So, if I'm just sitting looking at a picture on the monitor I fixate on some feature on the monitor I move my head back and forth looking at say a particular icon on the desktop it looks stationary in a very natural way. Now I have a head mounted display and I hold my head fixed let's say. So, I am holding my head fixed I am looking through my head mounted display I see pixels and the object starts to move pixels are changing right. So, again the same kinds of issues of frames per second are going to matter as they would at the movie theater. So, this is a very similar kind of situation if I decide not to move my head. If I move my head then I have additional problems of there is going to be even more motion of this object or less depending on how they combine, but if I am moving my head and the objects moving they are certainly going to be pixels to change right, but just the object moving by itself is enough if my head is moving in addition then it is some kind of combination of those two motions that are going to cause pixels to be changing as this object is represented. Now the fourth case is the most unusual and interesting and it is very important to pay attention to which is that we said if we sit and look at a screen and we are looking at some stationary object it does not matter how the pixels are switching or the frame rate, but if I want to use my VOR I have some object that is supposed to be fixed in the real world sorry supposed to be fixed in the virtual world right in the in this world frame. If I turn my head back and forth while wearing a head mount to display does this object is this object supposed to stay on the same pixels or not should not right. So, the image I sort of have or the feeling I have is like this it is almost as if when I put on a head mount to display imagine there were grid lines drawn on it, but it is like a piece of glass and I see through the glass to the outside world that is hard you cannot keep them both in focus the grid lines on this piece of glass and the outside world, but suppose you can forget the focus problem. If I move my head back and forth then I would see a grid pattern sliding back and forth with respect to all of you right. And so that is exactly what is going on here these are the pixels that have to be switching you know as I see you sliding back and forth across my grid or the grid sliding back and forth across you it is does not matter which way you look at it. This corresponds to the part that has to change. So, if these pixels are not changing fast enough then how does that affect my perception of stationarity. So, let us look at a case where we have a very fast switching display. I have the same picture here I want to look at a particular object just call it a point if you like do not worry about you know maybe it is so small that it is like one pixel wide, but it can be a few pixels wide. We fix it on one particular point and let us suppose that maybe we have a display that updates at 60 hertz I am going to forget about the waggle problem. Let us just suppose you know I update all the pixels at the same time they switch very very fast, but they only update at 60 frames a second right. So, I go back and forth like this what is going to happen to the image on the retina. So, what you should instead get is a picture like this where if you are at 60 hertz just as an example let us say 60 hertz then this amount of time here this would be 16.67 milliseconds. Does that make sense why that is happening right because if the image is frozen and my head is turning it is not supposed to stay the same image for 16.67 milliseconds. So, your brain will perceive that object as moving a little bit. So, that will cause an object that is supposed to be stationary seem like it is going back and forth like this. It will gradually move across and then jump and then gradually move across and then jump gradually move across and then jump that is what is happening on your retina when it is supposed to be stationary terrible problem right. So, one name for this this kind of oscillation is judder. Judder is kind of a fairly generic term used for all sorts of video flaws and things, but this is sometimes considered one example of judder. Now remember I said that you could flash the screen on for as little as one millisecond and that will be enough to trigger your photoreceptors. So, that is what I could do here and that should help with this judder problem. So, I could right this is the always on case and this is going to be the other case which I called what what did I call the other case pulse. So, pulse case. So, in the pulse case it will look something like this. I will get a picture like this where during all of these intermediate times between these small diagonal line segments the screen is black. Let us say completely off. So, I pulse it on for one millisecond. So, this I make this little thickness here be one millisecond right it was not rendered too well here, but. So, it is on for one millisecond and in the remaining fifteen and something milliseconds the screen is completely off. So, I could do that and then when I move my head back and forth the amount of judder the amount that this object appears to be sliding is significantly less. If I try to make it too much less than a millisecond it might not fire long enough to fire my road to trigger my photoreceptors. Now I have another problem if it is off most of the time and it is just pulsing these bright lights I perceive flicker. So, you solved one problem and I have another problem. So, because of flicker we have to go back to the same scenario that we had back with giant CRT monitors in the 1990s and we have to get the frame rate up higher. Now we could try to use that trick of multi blade projectors and just show more frames here, but if we do that the object is still not going to be in the right place. So, you can add more frames, but it will start to look like it is juddering or stuttering you know it is not moving exactly in the correct way right the object will not be in the right place of the right time. So, if you increase the frame rate from 60 hertz up to say 90 hertz and you do this single simple pulsing then you will have solved the flicker problem and this judder problem and the perception of stationarity will be solved. So, that is the motivation that is why people are pushing for higher frame rates right now in the business of head mounted displays. So, this is sometimes called low persistence mode because the frame is not persisting. So, this will be high persistence mode up here on the top. The frame persists the light is on for the entire time in the low persistence mode it is there for just enough time to trigger your photoreceptors and then offer the rest of the time, but the whole frame rate is fast enough so that not only do you not perceive flicker, but hopefully it is just comfortable in the longer term. So, there is no subconscious implications as well where you may have fatigue or headaches. So, once you are above 90 hertz it seems to be fine, but just be aware that even 90 hertz is not enough to solve the problem of if I have an object moving very fast and I am not tracking it it will still generate multiple images across your retina because you cannot track the object and keep it in the same place on your retina anyway and you will perceive at least a zipper of some kind in that case. Generally speaking the higher the contrast the more you tend to notice these problems as well that is why in the real world getting a bright LED in doing and moving it generating a pulse signal with a pulse generator and moving it in the dark is the best way to do the experiment. So, it is a high contrast whether it is the real world or the virtual world will bring out these effects the most. Questions about that? The LCD case if you have very slow switching pixels I would just generate a big blur zone here. So, I would not even get sharp images it is just as I turn my head the object gets blurry and then get sharp again. It is probably not going to make anyone sick, but it is not very it is hard to say I have to do the experiments to see, but it is just not correct right. So, instead of having perceived stationary to be right with a very slow switching display like an LCD display the objects become blurry when they are supposed to be stationary. So, they blur and then unblur maybe that is better than judder, but this pulsing method is better than both if you can support the fast frame rate and reduce flicker. Questions on that? Ok I am going to switch topics now significantly and I want to talk about tracking right. So, even in these examples when I am moving my head back and forth doing the using the vestibular ocular reflex in my perception of stationarity head tracking is coming into this right because you need to figure out which way your head is pointed ideally you would like to know which way your eyes are pointed as well. So, tracking becomes essential to get the viewpoint correct so that you can draw the images correctly in the right place at the right time and minimize perceptual artifacts and problems. These are a bunch of perceptual artifacts and problems that are simply due to the displays based on resolution in time and space, but in addition to that we have other problems which is the ability to accurately track over time and space in the physical world and use that information appropriately to generate the right information or render correctly to the display with as many with as few spatiotemporal anomalies as possible right. Especially the ones that were most critical to our perception that's how the that's how it goes.