 Okay, so we moved to our next talk. Please welcome Aaron, who will speak about JPEG 2000. Good morning, everybody. My name is Aaron. And today I'm going to talk about JPEG 2000. Just a show of hands. Has anyone ever intentionally stored an image as JPEG 2000? Okay, that's impressive. So it was actually designed to replace JPEG. It never, about 20 years ago, but never really made it. But I, it's been one of my obsessions for the last five years. And I think it's actually the code of the future. And that's what we'll go into today. So to understand JPEG 2000, we have to go back to its predecessor JPEG. And I'm going to take you back into the 90s to the rise of JPEG. So this was standardized in 1992. And it really came of age with the internet, with the dawn of the internet. And by the end of the 90s, about a quarter of the internet traffic was JPEGs. Now, we all know that was mostly porn. But still, that was an important way of rising up in the internet. And now we all have a JPEG encoder, a decoder in our pockets in our phones. So it was extremely successful standard, but there were some issues. So here you have on the right a picture of a flower. And you can see this is about a one bit per sample image. So you can see the block artifacts that you get in these, that you've all seen over here, because JPEG uses an 8x8 pixel block to do its compression. And so when you get very low bit rates, you start to see the boundaries around the pixel blocks, the 8x8 pixel blocks. So that's a big problem. Another issue is that it's only lossy. So if you need lossless compression, you can't use JPEG. There is another, there's a lossless standard for JPEG, but nobody really uses it, and it's a completely different codec framework. So which isn't ideal. We're also limited to 8 bits. So if you wanted to do medical or remote sensing, you can't use JPEG for that. More limitations are, you can only use three components. There's limits on the dimensions of the image to 16 bit. And one of the big issues is rate control. So the only way to actually target a specific rate for an image is to continually recompress until you get to that target image. And each time you recompress, you're losing data. So it's really not ideal. So those were the limits that gave rise to JPEG 2000, which was supposed to address these limitations, but it also added a whole bunch of new features, which I'm going to go into in a moment. So only six years after JPEG was standardized, this JPEG 2000, they began to standardize JPEG 2000. It was based on the work by David Taubman, his thesis work. Those are the monikers for the ITU and ISO standards. And it is royalty-free. Taubman actually donated his IP so that he would make sure that it was a royalty-free codec. And when you introduce a new codec, you definitely want to have better image compression than the original. So that is, in fact, the case. So JPEG 2000 has up to 200% better performance than JPEG, meaning you get smaller images for the same bit rate or same signal-to-noise ratio. But besides that, you look at the artifacts between these two images. It's a one-bit per sample image of the LENA image, but you see on the right is JPEG 2000 because of the transform for JPEG 2000, we do a wavelet transform versus a 8x8 transform for JPEG. So when you lose data, you just blur the image. You don't get those pixelated artifacts in JPEG. So that was a big advantage for the new standard. And also it supports lossy and lossless in one framework, which is nice. The only difference between lossy and lossless is the transform is a little different for a lossy. And there is quantization where you lose data, but otherwise the framework is exactly the same. And then you can get visually lossless encoding where there is loss of data, but the human eye can't detect the difference. So that's visually lossless. One of the big innovations, though, is the progression that it introduces. So progression is used by a codec so that the decoder can do a partial decode of the image and still get something useful out of it. And here this is what people are mostly familiar with is progression by quality. So you can see that image of the butterfly being downloaded and already by the first, second, or third frame you can start to see something useful even though you've only downloaded about 8% of the image. So JPEG 2000 allows you to create quality layers and package up those layers. And if you progress by quality, you can progressively download and get a useful image when you haven't downloaded the complete result. And this is how we do it. So that's an image of a castle in 8 bits. So these are the bit planes for all the bits in each of those samples. So that's the most significant bit over there. And so we create these bit planes and each plane is compressed. There's three different passes that we run through the plane to compress it. So it's very computationally expensive but we create many different layers or passes for that image that we can then package up into different quality layers. And you get a very fine grained set of quality layers that you can break the image up into. And that's a graph of the distortion as you build up the bit planes. So for an 8-bit image, you could have 23 different passes and each pass can be packaged up in different quality layers using this EBCOT, which is the formula for the algorithm for determining these quality layers. And so that was a big innovation. But there's three other different ways of progressing in JPEG 2000. This is progression by resolution. So that's a wavelet transform of this image. We do low and high pass filters on the X and Y coordinates. And so we actually, on the top left, you get a low res thumbnail of the original image and then high pass bands. They're called sub bands. So there's four different sub bands. And so built in, baked into the codec, you have lower resolution replicas of the original image without any redundancy. And we would do this wavelet transform maybe four or five times. So inside the actual image, there are multiple lower resolution replicas that you could then, as you're decoding, you could bring those images out without actually storing extra data as a thumbnail. It's actually built in. So you can progress by resolution if that's how you encode it. There's also progression by spatial region. You can see these are the original sub bands that we had from the wavelet, these big blue boxes. And then inside we have the precincts. And inside the precincts we have the code blocks. And so you can actually encode by precinct and store the image progressing that way. And that's useful for, let's say you have a scanner. Let's say you're limited by memory. You might want to encode one precinct at a time and then move on. And it's also useful for random access into the image. And the final one is progression by component. So if it's an RGB image, you can break it down into your luminance and your chroma channels and you can progress by those channels. And if you're a grayscale printer, maybe you can just throw away the chroma and keep the luminance. So that's another progression that you can get. There's a bunch of other features that you have in JPEG 2000. One is error resilience. So you built into the code stream when you're encoding it. You have to terminate each code stream segment and you can terminate those segments in a way that when the decoder is decoding those segments it can tell whether there's been an error in the segment. And also we have these precincts that contain errors in the code. So if there's an error in one precinct it can't migrate into the next precinct. So we contain errors that way. We also have recompression resilience. So typically an image might be compressed and decompressed multiple times. And so the codec doesn't lose a lot of information for the subsequent compression and recompression like decode and decode. So that's another feature to have that we have. And then you get random access into the code stream from if you do progression by code blocks. So if you have an enormous image from a remote sensing let's say you want to just decode a small piece of the image you can actually put markers into the code so that you can locate that particular region of the image and decode that which is useful for huge images. We also do low latency because it's an intra-codec so each frame is encoded individually and so you can get very low latency unlike for example H.264 where you have groups of pictures that are encoded together. Each frame is encoded separately and you can get low latency that way. Also a region of interest where you just increase the quality in a certain part of the image. Let's say you're doing video conferencing you want to focus on the face so that's built in as well. So with all these cool features who adopted J2K? First of all medical imaging it's considered the gold standard because you can do lossless which is important for legal issues and you can do high bit rates like 10, 12 and 16 bits. So medical imaging it's popular in remote sensing because you can support 16,000 components if you like and very large dimensions so it's popular there. And also a few people realize that whenever you watch a movie you're actually watching JPEG 2000 decoding because it's the standard for digital cinema and the reason is because of that resolution by progression by resolution because the 4K image that you encode has a 2K image built in so the 2K projector can actually just decode the low level sub-band and throw away the rest so it's backwards compatible and that's used it's a standard today. And so what went wrong with all these wonderful features? Yes there is an XKCD comment for this. Well first of all patents there was a company called Lizard Tech which claimed all wavelet based codecs that you had to license from them and that was eventually invalidated but that was cost some fud for the emerging standard. Microsoft was the monopoly in those days and they had their competing standard called XR, JPEG XR so they didn't support JPEG 2000. There was browser support was poor even this is 2015 you can see only 20% of the browser support is standard and hardware support. The hardware vendors were waiting for the software people to be able to display the images the software vendors were waiting for the hardware to produce those images so they never really supported them in hardware and finally performance. Because of the complexity of the codec it's very very slow because of all those passes through all those bit planes it's a very very complex codec and so it was very slow and JPEG was already there so that was good enough so those were all the reasons why it never really made it. Now there's been a change in the codec to address the performance called high throughput JPEG 2000 and they keep everything the same and they change the block coder which is the piece of the very center of the codec that actually creates the bit stream and that's what creates all those bit planes and all those passes so they've simplified the block coder and this gives you a big speed increase ten times possibly and it's was standardized this year, sorry last year 2019 and so it's also royalty free and the difference is that it's actually friendly for GPUs and vector calculations which was different than the original encoder which was designed when actually multiplication was expensive so they tried to avoid multiplication by creating branches and doing a very serial approach and so that doesn't really work on modern hardware so this one is designed with SIMD in mind there are fewer passes and you just sacrifice the quality scalability that you had from the original JPEG but everything else is the same and you can also losslessly transcode between the two formats so you can encode it one way for speed and then later convert it to the original for distribution and there is a decoder in it with inscription that will hopefully address the browser issues that they had with the original standard these are the open source toolkits JPEG is the granddaddy of J2K toolkits there is grok which is one that I maintain and there is something called OpenJPH which is only for high throughput JPEG 2000 any questions? I have a lot of time for questions we have questions from the online forum I have a lot of questions about JPEG XS actually I had slides for JPEG XS and I took them out because I thought I would be going over time so I can talk about that a little bit so JPEG XS is actually a competing standard for the same space it's based on a wavelet transform it's also low complexity, low latency the big difference is that it's actually a patented standard by Intipix and Fraunhofer the people who brought you the MP3 patents they are hoping to license that so I'm hoping that the open solution is going to win but they're in the same space and actually from what I've heard the J2K is actually faster than the JPEG XS and it's actually the image quality is higher for the same bit rates so it looks promising but that is a competing standard in the same space how do you think Any questions from the floor? What about the new JPEG XS called XL Google algorithm that you will start next year How fits it in your scheme? That's called XR? XL The question is how will the new JPEG XL standard compare with JPEG 2000? Actually I'm not familiar with the XL standard so I can't really comment on that There's a new progression algorithm from some people from Google and it's great I'm not familiar with JPEG but it's complicated but it has additional features from JPEG 2000 Really? It's hard to say but the nice thing about standards is there's so many to choose from we're actually in a renaissance of JPEG standards which is backersely compatible with JPEG but adds HDR and things like that so I think it's a good thing but I have to learn more about this codec to comment You mentioned that it's a kind of a scalable project you had a point and get a lower resolution we're interested in that for for instance lower resolution texture and we want a lower resolution to be just a little bit from the approach So the question is is it possible to decode an image starting with the lower resolution first and progressing upwards? Yes so you can start Yes you can but the progression is the way if you encode it and store it by resolution then you can actually choose I only want the lowest two or three resolutions and you throw away everything else and you can only send that low resolution piece so it's designed to do that You can The question is if you can throw away the rest from the encoder you would not become I think it wouldn't really be compliant because the header would say I have ten resolutions and you've only stored three but most of the decoders are used to handling abuse of the standard so they will handle that and complain about it but you could definitely do that and there's actually a there's something called jpip which is a progressive format for doing this for sending it over the wire there's actually a standard part of the jpeg2000 standard for doing progressive sending it progressively over the network yes I wanted to speak for a long time the question is jpeg2000 available as mxf so mxf is the exchange format so mxf supports jpeg2000 and so you could package it this is very new so I think you'll be getting I think everyone supports it they're going to be adding it to mxf I know that and they're going to be adding it to the digital cinema standard and I think everyone's going to be adopting this because it's just fixes some problems and doesn't really have much of a downside for those workflows because you don't need the progression by quality so much in that workflow this is the first end of last year that they standardized the high throughput but I think it will come so the question is how much slower is jpeg2000 versus jpeg it depends a lot on the implementation so for example open jpeg was in terms of open source toolkits so David Taubman has his own codec that's closed which has always been very fast and had all the features and the open source toolkits were slow and filled with bugs for the longest time but recently they've gotten quite fast so they now support multi-threading they've tried to optimize them so I don't have actual numbers but on a modern machine even with jpeg or with my toolkit I think you're looking at maybe twice as slow for the J2K but with the new high throughput I think we could actually get faster than jpeg because it's designed for GPU and for vector operations so I think we can actually get faster than jpeg which is going to be really cool I think sorry can you repeat what's the expected encode time for it depends on the size of the image and a 4k image okay so on modern hardware so now you could encode that encoding some of the codecs they don't focus on encoding they focus on decoding and encoding is slow it actually opens jpeg it's single threaded and a 4k image is like an 8-bit image or a 12-bit 8-bit I guess maybe you could do it 10-15 frames per second on a modern machine and with the multi-core with the new thread rippers and all that you can really get good performance it would really depend a lot on the image how many cores and all that but you could get maybe 10-15 frames per second encoding on a modern CPU and the decoder could be even maybe 20 frames per second decoding a 4k which is and the high throughput is maybe 3 or 4 times faster than that when you do the whole encoding the block coder is much faster but the whole codec is going to be significantly faster so the question is encoding and decoding different in terms of speed there are actually some tricks you can do in encoding that you can't do in decoding because you know the whole you actually know all the pixels that you're going to be encoding before you're encoding them so you can actually do some tricks to speed things up but I think decoding is going to be faster than the encoding it's the way it's designed because most you usually decode many times, you only encode one time so it should be, it depends on a lot of other factors how they implemented it and that sort of thing Thank you Alan