 So, okay, I'll be talking to you about HDR models for cinema This is very much a work in progress that we're doing with with Juseb Blatt from the here the interactive technologies group and David Kane and Ricardo Marquez so What is this about so? Let me start by talking about what's the dynamic range and what is dynamic range of a camera? so in general the dynamic range of a scene is the You can think of it as the ratio of contrast as the difference between the brightest and the darkest points in a scene and you compute this ratio and that ratio is the dynamic range and Cameras are able to capture a given dynamic range Which is usually much smaller than that of You know common natural scenes For instance, this is a picture taking with a you know regular camera with a short exposure time We can only see what's outside the window as we keep increasing the now We increase the exposure time and we take another picture and now we see more Okay, as we increase the exposure time we get to see more and more inside but what's outside gets overexposed and After some point we just don't see anything outside. Okay, so why is this happening? Well, the response of camera sensors is linear within a range Which you can see there so from dark to saturation. There's a linear response of sensors and if a scene so this is the histogram of some scene it has a lot of well, it's the histogram of the Intensity of the light coming from that scene given the exposure time that you're using. Okay, so if the exposure time is short Most of the points will have a very dark value. Okay, so as the first picture I showed as we increase the exposure time Now we get to see more things, but the things that are let me see this work Yeah, so things that are brighter become saturated. Okay, and as you increase more the exposure time the effect I mean the area of the saturated pixels is even larger So this is problematic in particular well in many instances in particular in the cinema it's problematic because It forces you to introduce artificial lights, which are cumbersome, it's a very time-consuming Expensive process So you what one thing that does not work is just to use short exposure times and then boost the gain So if you do that noise becomes apparent like you see here. Okay, so it's not that's not a good practice so Yeah, so what what people do in in cinema shoots is they add lights Not only in low light situations like here, but also in you know daylight situations You are artificial lights so that the dynamic range is reduced and that sorry That was what I was trying to show here So if you add lights to the scene what you get is that The brightest points are not much affected, but the darkest points get brighter So you reduce that ratio remember that the dynamic range was the ratio between the brightest and the darkest points So if you make the darkest points more illuminated then that histogram fits within the available dynamic range of the scene, which is what You do or when you use a flash it's exactly the same thing so People do this cinema crews do this all the time and That takes a lot of resources away from doing photography for artistic reasons So there's artificial lights are there sometimes or very often so that you can see things properly and Often also because you want to express some artistic intent But the voting time to this makes less time available for the other or makes the whole process more expensive Like this you would like to devote all your you know available resources to do photography for artistic expression So in the what are the challenges that we we are addressing in the in this project So the first one is tone mapping so The dynamic range of cameras is there's like an arms race It's ever increasing so dynamic range of cameras increases and dynamic range of displays increases and so on so This is some Yeah, like pamphlet from some renting house from 2009 and dynamic range at that time was around You know between two and three orders of magnitude now This is 2014. It was around 14 orders of magnitude. I'm sorry 14 stops around 34 orders of magnitude and Presently it's getting closer to to five. Anyway, that's Enough in many situations. It's not enough in any general situation. There are common Natural scenes with a dynamic range of six seven hard higher orders of mind so but even if you Build a camera, which is capable of shooting or recording Footage with a very high dynamic range you that footage you need to display it, okay, and the if you can now nowadays get a digital cinema camera like that one with a Dynamic range of four orders of magnitude a digital cinema protector can get this and a home TV can get Like two orders of magnitude. Okay, and if you go to a store People by selling you the TV will tell you that's a total lie. We have a contrast of well a few years back They will tell you that have millions You know five million to one contrast That's That was a line that they're not telling that anymore But nowadays you can you can find you can buy TVs that will tell you well The difference intensity between the brightest and the darkest point is like five thousand to one and that is true But the way you compute that is not in real operation of your TV when you're you know Watching images on your TV In a room for instance, there's a lot of Light bouncing back from you to the screen that bounces back again and so on and that reduces the effective dynamic range So it's about that. Okay, and definitely it's it's more than dynamic range of the camera. So there's a Something that has to be done and it I don't worry. It's it's done So all the movies that you see have gone through this process by which the Dynamic range of the footage is reduced so that it's it fits the dynamic range of this place and of you know projectors in cinemas that process has several names in Here we're calling that tone mapping which is like the usual term in in both in cuba graphics and an image processing So we have to do tone mapping We have to reduce the dynamic range of the footage so that it fits that of this place And we have to do that preserving the natural appearance of the image so So what is this so your you know any camera from your cheapest mobile phone camera Does the following it does a non-linear correction. So it reduces this dynamic range in this matter so this process called gamma correction and it's basically Consists in passing the footage or your your picture your video through a non-linearity. We had which has this shape Okay, and this is this replicates how we perceive Intensity so the way we perceive luminosity is not linear with respect to the intensity of the light it has this non-linear response which can be in Often in many cases well approximated by this by a power law, okay of an exponent of I know almost one half Okay in cinema though the This is not the way this is done. You don't want your camera to take your artistic decisions for you so what people do in in cinema shoots is They keep the The the information image information in linear form They just pass it through a non-linearity for encoding purposes so that they don't lose in the quantization when the image is digitized Okay, and that non-linearity have has a logarithmic shape, but it's only for for encoding So if you look at so that is what that is just the the representation of what the camera is Recording so it's not ready to be seen So if you take away that logarithmic encoding and just put the usual gamma correction curve You would see something like that. Okay, this is what your mobile phone or Any regular camera would do when recording a JPEG or a MPEG video This is the kind of output. So in cinema what you do is you keep this use and You undo the logarithmic encoding in post-production and there's a skilled colorist That goes that's a grading process But which he or she adjusts the contrast and also the color so that you get to see This for instance, so take a look at the background. Okay. It was overexposed here now. It's no longer exposed and Okay, so this will be manual term mapping and manual term mapping is what's going on in post-production It's done manually. It's not done by algorithms So this term mapping is important because if you don't do it right if you just say, okay I will show this everything linearly. Well, you don't see anything like so this image This is an image in which we have done a linear term mapping So zero corresponds to black and one corresponds to white and you don't see anything Why because the histogram is very much skewed to other low values So there's a peak here in the low intensity and there are some few pixels which are very bright so if you do proper term mapping something like Well, similar to his or my qualification you get that Anyway, another challenge we want to address Merging sources of different dynamic range. So in cinema, so well, not in cinema. So when you work with With photorealistic synthetic images so this synthetic 3d scenes are built with very much very accurate models of how Light interacts with objects in the scene. So they're really really they can be really really accurate and At the end the final stage is you have a Image which is a representation of the light coming from the scene and you have to pass that through a term mapping algorithm So that you can see that on your screen and that's what you know any you know software for architecture or some Or software for for video games does it would it does it takes this physical reality. I'm sorry this physical Approximation of physical Reality and tone maps the intensity of the light so But the original scene is high dynamic range and In sorry in cinema what you usually have is you have for special effects for visual effects You have to mix Footage which has different dynamic range. So you can have standard dynamic range or even low dynamic range for your actors And you have to combine that with high dynamic range CGI generated objects and you have to do that properly and that's done. So some examples there That has to be done manually through some very much Well, so through professional software tools for post-production like some of them you can see here So another challenge we are avid it very much interested in is Image quality assessment. So let's say Well, our goal is to do a ton mapping method Which is automatic and it matches in terms of visual appearance what a skilled colorist does in post-production That's our ultimate goal But in order to address if whether or not we have reached that goal, we have to validate our results And so we need to do some image quality assessment for the tone mapping and it turns out that there are very very few Objective I'm sorry quantitative metrics for assessing the quality of tone mapping. Actually, I'm only aware of to I mentioned one so here This is the metric of aiding at all from 2008 So what they do here is they use models of visual perception that have been computed So this models of visual perception are derived from experimental psychophysical data in which people are presented with artificial stimuli like great things and Gabbers and With different levels of detail and luminosity and these people are asked Do you see this change? Do you not see that this change? So they This allows Allow the researchers to compute Thresholds of visibility of change So in this manner with this what this metric does is it takes us input a high dynamic range image and its tone mapped version and Comparise the visibility of details. So if they are if there are details, which are visible in the high dynamic range scene But not visible in the tone map tone map seen this Paint that with represent that with green if the opposite is the case if there are things you cannot see in the high dynamic range scene but you can see in the tone map version this represent that with blue and Red here represents in version of contrast if something that was you know a boundary going up It turns out. I mean a difference of intensity that goes up now in the in the top up Image it goes down that's represented by red. So in this manner you can This metric allows you to see to gauge which are the changes of visibility in the Tom mapped image like for instance here So this is on the left a Tom mapped image on the right the error There's a lot of error in terms of loss of visibility groups represented by green Anyway, this is a very useful metric It it correlates Well with visual perception. I'm sorry with with the opinion of Subject it have but it has not been thoroughly validated and we have found that it does It could be improved. So it correlates. Well, it it could correlate much better. So there are moments or I'm sorry There are times in which you apply this metric and estimate that the quality of a tombot image is Very good, but people do not really find that image that good and the opposite can also be the case. So there's Really much Well one further point this metric only considers contrast It does not consider color and color is very very important in how you got the appearance of an image the quality so another challenge therefore is to develop Quality and image quality metric to assess tom mapping results So for instance, you have here different Tom mapped results and here the errors according to this metric So our previous work We have fun. I'll go very quickly through this So one thing that happens is in the human eye light bounces inside it And this has the effect of reducing the effective dynamic range. So even if we are seeing Looking at the scene, which has a very high dynamic range the dynamic range of the retinal image The image that forms in a retina is very much reduced and that's good for us Actually, because it gives us hope that we can do a proper Tom mapping algorithm using the Limited dynamic range of the footage of cameras. So this is an example of a retinal image Well, I don't have time to go deeply into that but the retinal image is definitely not the final Ward, I mean in our visual system does a lot of processing in the on the retinal image and it It's able to recover Many of the apparently lost details that we think, you know are unrecoverable here, but they can be recovered But the our point is that you start with something which is limited in dynamic So this is just to highlight that in effect the dynamic range of Retinal images is reduced So we did this work which has Yeah appeared this year In which we so we were presenting We were asking people to judge the quality of of images as we varied a different number of variables and we find we found a significant correlation between the image quality that people reported and the level of Uniformity of the lightness histogram of the scene And that allowed us to you know derive several conclusions regarding the you know the the gamma curve that both the encoding and the display should have and also to touch upon the Effect of the background and the surround of the image on the perception of the image so We derived a another thing that we did is we we developed a tone mapping algorithm that is based on natural image statistics and basic principles of human visual perception This is also a work from this year returning imaging so it one one principle that it's it repeats itself in several stages of In of the human visual system is that of efficient representation. So The Availability of resources is limited. So there's a lot of processing that is done in order to optimize the And the way these resources are used and the and optimize the results Are produced with these limited resources. So one thing that is done is that? Given that the response of photoreceptors is limited Photoreceptors have a response which does Which is nonlinear and does sort a sort zero zero minutes great Okay, so thank you very much anyway for the receptors, though do a sort of history my equalization on the on the input and this is an average of the histogram of natural images and In log log coordinates. So what we did here in this work is we took so this is computed from photographs Okay, and natural photographs and this is the average histogram. So we said let's create a Nonlinear curve which would flatten the histogram of a typical image which would have this typical histogram so this is basically what we did and so this method of ours has Some parameters which adjust on an image-dependent matter Automatically, okay, and after that we do contrast enhancement, which is also and really just a very Ubiquitous principle in in in human vision, which is that of divisive memorization okay, and so this is tone mapping algorithm based on on vision models and natural image statistics and we get results which So this is a JPEG image What you would get and this is what we get applying our algorithm to the raw image Okay, again JPEG and this is what we get applying our algorithm to the raw image, which is very close to what a skilled colorist would achieve So that's that's good, but the problem is very much not Closed why is that? Well, okay, so there's There's very few high dynamic range image databases. This is one of the most popular one by Mark Fairchild and it has like 108 images and Properly taken and calibrated and so on so it's it's great, but it's not enough for doing statistics like we are basing our model on so Some examples of those images. So this is the You know the average histogram, but it turns out that if you take this database some his some image has this Histograms here are in blue this histogram, which resembles that shape But you can have by model distributions or this weirdly shaped ones. So We want to improve our metal. I'm sorry. We want to improve our model Which is based on statistics. We need a large database of high dynamic range images Taking high dynamic range images properly is very very very complicated. It's time consuming. You have to do it very carefully It's it's not straightforward at all. So what we're doing now is we take we're taking The graphics approach we are producing synthetic scenes and in which is very much simple to change the light okay, and What we found is the following this is the distribution of Median luminance versus dynamic range in this Database that I mentioned it has this very interesting negative correlation and We have found this is very preliminary results that with synthetic scenes. You can get the same behavior Okay, so that's a good start. So our goal would be to produce High dynamic range images synthetically millions of them Okay, from which to derive statistics, which are more accurate Which can then be used by us to improve our model, which is based on natural image statistics So first we have to ensure that the natural image statistics that we can derive with synthetic scenes match This statistics of real natural scenes. So this is our example of You can see there How the median? How that behavior there of the median luminance versus dynamic range Changes as we change the light Okay All right, so thank you everybody Yes By collecting photographs available on the web would that be possible or is not possible will you have to analyze them? In order to decide whether they are suitable for your research or Well No, it would not be possible in the sense that we have to Ensure that things are done properly. So we need We need this scenes that are So the usual manner in which high dynamic range images are created is the following you take differently exposed pictures that the ones I showed in the beginning and then you fuse them, okay and This sounds simple enough. It's really If you want to do it properly, you have to be extremely careful So misalignments are very problematic the way you fuse them I mean if you change weights your results can change the weights that you give to different images The results are changing appreciably So if you give so the dark images are noisy and the overexposed images Well, they're exposed so you have to read out those values plus you have to calibrate You have to go to the scene and measure the actual with a photometer the actual amount of light so I'm not aware of people doing that on a systematic basis. So maybe some someone has done that and posted that on the web but What usually is called HDR images on the web is actually not HDR images is the tone mapping result Which is also usually done just for artistic purposes So you get something that does not look natural. It looks weird with halos and so on and That's what many people in Flickr for instance called HDR