 of RGB. Raise your hand. Good amount. How many of you know CMYK? Good amount. How many of you know have a digital camera? Okay. That's excellent. How many of you have played with Photoshop? Okay. How many of you have used the Python pillow library? Okay. Now, we're going to get into something interesting. How many of you have tried to do a photo print? Okay. Yep. How many of you have tried to shift the color space of an image? Two, three. Excellent. How many of you have run monitor calibration? Really? Why do you do that? You only need black and white. So, still, why did you run your monitor calibration? Any particular reason? No reason? Okay. Right, right. How many of you have a printer at home? Okay. Okay. How many of you have tried to print a photo of your own? Okay. See a consistent pattern here. Some people have gone a bit too far. Okay. So, I guess this talk, we have one minute. I guess this talk is less about one particular library, which I'm sure, like all of you can go back and read a doc. I want this talk to be a little bit about the concept of how to reason about these kind of stuff. Because when we started, I'll let you know why I'm getting into this mess. But I found it very hard to think of this whenever there's a problem, when you think of it, there are multiple input points that change when they change, they actually affect each other. So, for example, how many of you have a driver helicopter before? Hey. Yeah. How many of you have to bug a system where you don't know the relationship between multiple inputs? Yeah. How many of you have been bitten by a bug in a multi-threaded application? Yeah. So, that's the problem. Like, this whole image processing is a very sophisticated, complicated, arcane, mathematic tense topic. And the classic textbook on this topic is 600-page thick printed in black and white. A book about color theory has nothing but mathematics and black and white ink. So, when I got into this field, I realized there's nobody talk about this whole theory in a very easy to understand way. So, today I'm going to give you the talk, which is from Adam to Pixel and back. Hopefully, by the end of the talk, you'll have a good understanding of when you take a picture, okay, take a picture of anything, you see the picture showing up, the very sophisticated process that happened in between. It's a fascinating process. And hopefully, that one day when you sit down with someone else and they point at something on the monitor and say, that color doesn't match, right? You know, you know roughly what to think about it. And also, I would like to present this talk in a way that, you know, I spend significant amount of time writing JavaScript code now. I love Python, but more and more of my time actually going to writing JavaScript. So, there is a lightning talk yesterday, talk about like, you know, the some people say like the lifespan of Python has gone to the peak, you know, the development actually stagnant. So, I want to show you guys the possibility you can do with Python that sometime actually extends further from the core language actually, you know, you can see examples of using Python to generate JavaScript code at the very end. Okay, so let's go in. For those of you who don't know me, in my previous life, I written a search engine. I have done some interesting work. About three years ago, my wife and I, we decided, you know, we really want to slow down and have a baby or something. So, I was looking for, excuse me, an industry that's very old industry that can be improved dramatically by using software technology or robotics. So, the one I found was a photo printing industry. Photo printing industry, the last revolution, it has been with us since like late 1800. The last revolution in this industry is about dot com bubble where you can finally print your own photos from online. But the process of printing the photo and making the product has not been changed. In the U.S. alone, there are about 2,000 companies do photo printing this year, which means that there's a lot of labor intensive, there's not much automation going on. So, I saw the industry and I feel, okay, that's the industry I can do better by using software to automate a lot of the stuff and by using robotic technology to automate a lot of the labor physical stuff. So, that's how I get in. So, the topic today has very little Python in it and I would like to call it a hate and love story with Python or the story of abandonment and redemption. Basically, the deep status quo of the new process in Python is like terribly broken. It's not even usable. But let's get on to that. So, we have four topics I want to cover. First, when we take a picture, how do we capture light? The second one is once we capture the light, how do we store them? You know, store them in a JPEG file, but what happened behind that? And then the third one is how do we display color? You know, when you get a JPEG, you open up on your computer or you open up, you know, project onto a screen. What happened behind there? The last one is really the redemption part. It's some really fun thing and everybody I talk to is really surprised you can do that. So, let's do the capturing light. A camera, a good camera. So, when you take a picture, this is a camera with the lens off. When you take a picture, the light will, how can we see light? The light come in because the sun generated these like, you know, spectrum of frequencies and it project onto the earth. And then different object, we, you know, reflect the light, right? So, the atom, each atom when hit by the light, it has a different frequency and then, you know, as your physics, if you remember that, it reflect different frequency of the light. So, that light going to the camera through the lens onto the sensor. So, let's go into the sensor. The sensor is really nothing really fancy. It has a few mechanical components. This one, low pass, basically filter out some really, the light our eyes cannot see, but the sensor can see. So, you want to filter those out. But what's really interesting is this sensor, CMOS sensor. So, the light hit the sensor. What happened there? This is a sensor if you zoom in. Now, each manufacturer has very different way to make these sensors. This is a CMOS sensor. So, what's in the sensor is that you see an array of RGB sensors. So, each of them allow one, it's a, it's a, like how many of you know the signal processing or the frequency filtering? Yeah. So, basically if you shine a light onto a red stuff, only the red light comes out, right? This one is done in a way that you actually break incoming light into three different type of, into RGB bands, basically. So, when you lay millions and millions, so every centimeter has about a million of these. Just think about it. It's crazy. And then each one is a cube, is a very square. And then every square has three bands to it, the RGB, which is a, this one is not 100% accurate because it's actually, you know, it's like three, three lines. And then what happened there is once the light getting here and then hit the, the bottom, it generates electricity. Very tiny electricity. I can't see this. So, you actually generate electricity on here. So, you can see, okay, this one has a little bit of electricity. So, we can see it's a, you know, it has a little bit of R in it, things like that. That's the process pipeline. And then once you, you get the electricity, you have the sensor, you can measure them, you can convert them into digital signals and then you give them no value. And that's how you get RGB value. So, when I ask you guys, when we started, I said like, you know, how many of you know RGB and how many of you know CMYK? Right? So, CMYK is the technology when, when we print stuff. So, the t-shirt or things on here, all printed with CMYK. But the photos we've taken are in RGB. So, now here, there's a story I need to tell you guys. So, Otago Uni, a friend of mine, worked for Otago Uni, the library. And they just spent about 160 grand, scanned a lot of the archival photos into digital, digital image. And they came to me and say, some of the color looks really weird. Okay, let's sit down and have a look. So, it turns out that they was using a very accurate library called Pillow. Let me show you guys what I mean by that. So, it's very simple. We have an image. Let me open it up. Okay. Very simple. We have a color mode here, CMYK. So, when the designer designed, this is an example. It's not one of the copyrighted content. It's a CMYK. So, let's say, okay, let's convert it into RGB. I won't get into details of the Pillow, but, okay, so, that's the standard Pillow library. You open it up. Very simple. And they even provide a very helpful library called Convert. You can convert it into any mode you like. We do Convert, and then we save it. You know, simple as that, we can possibly go wrong, right? And let's name it RGB. It's not 100% accurate, isn't it? And I have, like, even better examples. Sometimes the background turned completely black and white. Just not right. So, you know, this talk is not about, like, bashing Pillow. I think it hasn't been upgraded for quite a while. It's more about, like, help you to think, okay? When something's not right, like, where the potential problem is. So, here's clearly the color mode is not correct. So, whenever you are dealing with a, sorry, where's my presentation? Whenever you're dealing with an image not accurate problem, it's really only three parameters you'll be looking at. And this is one of them, like mode. Okay, what is the mode? Let's get to the second one. Oh, another very important thing is when you convert from CMYK to RGB, or convert anything between one mode to another, you're going to lose data. So, that's another very important concept, which I'll show you later in another topic, but just keep that in mind. Okay, so now that we have all these, like, you know, digital data representing an image, how do we store them? So, let me introduce you the concept of color space. So, color space is when you think of the color space, RGB, each color, if you use 8K to represent it, right? So, on theoretically, you have a color space of 256 power 3. But in fact, not all the colors are human eyes visible, right? Our eyes can only get to certain, like, you know, spectrum. So, when you plot out the massive color space our human eyes can see on a two-dimensional space, it looks like this, like, you know, like this weird shape. And you can see all the color, you know, our human eyes can see. Now, this called pro-photo RGB is the maximum theoretical limit of how much the current CMOS technology can capture. Okay, so it means that as long as you are taking a photo using a CMOS sensor, there are some color that are going to be lost the moment you take the photo. So, if there is an artist came to you and say, hey, I think you know computer well, can you help me to scan these photos into the computer? Now, there's some smart person might say, oh, easy, I'll just set it up, I can take a photo of your painting and that way you don't need to buy really expensive scanner. That most of the time that works, but if you have colors very rich in this area and this area, you know, those information will be lost. So, now remember this pro-photo RGB is only for CMOS-based sensor technology. Now, if you use a scanner, it will have quite different color space. If you use a different, not projector, chemistry, for example, you develop your photo, color photo using chemistry, it has a different, you know, area as well. So, CMYK, the color that can represent it when you print using the CMYK technology is much, much narrower. That's the shape you're getting. Very narrow. That's why, like, sometime when you scan a very old photo using CMYK technology instead of a painting, it's actually very safe to take a photo of it because, you know, your space is well covered. SRGB, our friend. So, one of the fundamental assumptions of PILO, which is no longer accurate, is everything is done in SRGB. So, PILO assumes all the images it's handling is in SRGB, which is not 100% accurate. As you can see that SRGB has still slightly smaller. It's some overlap with CMYK, but not a lot. So, all the monitors or image processing software, by default, will use SRGB. So, be very conscious about it when you open up a file and do anything about it because unless you deliberately set the color space to a much wider color space, it'll default to SRGB and then you will lose tons of color. So, these colors are wedding photographers, a lot of the color around here. If you are doing landscape photography with sunrise or sunset, a lot of color here are gone. So, you know, depends on the subject. So, when you get a JPEG file and you want to store them, store all the information we get from the sensor, each JPEG file actually has three parts to it. That's the defining the JPEG specification. Metadata, very simple, longitude, latitude, DPI, that kind of stuff. Not interesting. Color space definition, that is the key. So, it's not unusual for many image files, because I'm printing a lot of photos for my customers. A lot of the photos, especially Android phones, they're bad. Most of the photos you take with the Android phone, they don't even define the color space. It's like, well, guess it's an SRGB. And that's very lethal. I'll show you why that's the case. Now, with the latest Apple hardware, now you are getting this retina and then the Apple camera now captures much wider color space than SRGB, right? And I hope Android one day will support that as well. So, you need all three parts to work. Like missing any parts, you'll have to default to a common denominator. So, give you an example of what happens when you try to squeeze the colors in one color space into another. So, for example, you take a photo with your proper camera and you take it to a warehouse stationery. They have this super fancy printer and then they print using Windows printer or something. It's not even close. It's like Windows printer. Side story. The best operating system support of color is OS X. Like as early as 1995, they have a huge range of color space supported. That whole thing does not exist on Windows. Neal. I haven't even checked into Linux. I don't hope much. It's like you got to put a lot of work into it to make the API for that. And to use a Windows image viewer, it's just like, what can I say? Okay, let me give you a couple of examples. We have two dots in this photo. So, Adobe RGB is what you call the industry standard for photographers, which is a short for Adobe RGB in 1998. So, that's how advanced the whole system is. Everybody is still using 1998 technology. Anyway, that proves our point. So, when you squeeze that to a, you know, SRGB monitor, you can see the top circle actually pushed down. So, that lose color. You can see the color actually. On the top, you have a slightly darker greenish, and then the yellow is kind of balanced. And when you push down to the laptop, it's almost already a little bit. Yellow to yellow. And then that inkjet printer was assuming a, I don't know which printer that is, but you can see because from the laptop display to the inkjet printer, that color space actually, they have a huge overlap. There's no color loss. The bottom there, because, you know, it's just a color shift, it's actually, you don't lose that much color. Okay, so I want to show you same image, how different they can look when you do nothing but change their color space. And to make the demo even more fun, I'll use some interesting, I mean, it's professional work. I've got agreement to use them, but you'll see like if this is your kid, you don't want them to look any, this is like done by a photographer, a professional photographer. She got the skin color, you know, a little bit. This, the color gamut, the color space of that thing is very narrow. So I turned the lights off so you guys can see more color, but it's very narrow. I hope you can still see most of the skin color is, you know, white with a little touch of pink, not a lot. Here is slightly more pinkish than on the top. Now let's open them up in Photoshop. Photoshop is not working. The rest of the demo is gone. Okay. Okay, let me full screen that. You know, that's what you see is the intent, what the photographer have in mind. Now, in Photoshop, there's a color setting. So let's change a few things. It's a, can you guys see the change at all? Apple IGP. Very, very white. Color match. Now, this one, you never want your kid's photo to look like this. You know, so what happened is once you take a narrow color space photo and try to explode it out into much wider space, it will push everything out. And that just destroys the whole intent. While we are here, I want to show you. So basically, there are really only two engines that I know of that does these kind of things, does a good reasonable job. One is from Adobe, which is a newer version, or Apple. So both of them actually did excellent job. Okay. Let's go back to our, so let's recap here. We talk about the color mode. So when you take a photo, it depends on which kind of sensor technology you have, you know, the mode determining a large chunk of what you're doing. The second thing we just showed is the color space, i.e. when you store them, like the physical file you're storing the imaging, also have a huge impact. Now, sometimes you, because we don't have to worry about it, sometimes you take the photo in sRGB, they store in sRGB, they display on the sRGB. So the whole thing is actually quite simple. But as soon as you diverge, you will run into problems. So let's see the third thing, which is once we have the image all stored in the file, when we display them what other complexity is here. Ah, I forgot to show another thing that's messed up with Pillow. So Pillow has this like really nice, in a new version, they have very nice feature called ICC list. Basically ICC list is an attempt to read into the file and then pull out the color space and all that kind of good jazz. So that's what I got when I run the thing. Empty array. Oh, it's better than a now. Okay, how do we display colors? So I'll give you two examples of how we show the color, how we, when we print the photo, how do you get the colors out? So this is one LED. So when you look at a monitor, how that works is you have, remember when we, the sensor, how it works is you have RGB, right? And then the RGB light is coming and then depends on the strength of the signal. You know like RGB. Here is almost the reverse version of it. It basically shines a white light through a matrix of RGB bars. And then it has a polarizer. Basically polarizer, polarized lens is that you can decide where the color go and how, you know, the stronger they go, the stronger they go straightforward into your eyes. And by tweaking the polarizer, you kind of like, you can control the strength of the light. So for a display technology that's pretty much like RGB in, RGB out. That's how it works. You know, the lighting, then you use the polarizer to divert the color and then reduce the strength. What's becoming more interesting is this one. So professional photo printing, so CMYK has only four color. Your typical home-based photo printer has about six color. If you walk into a having normal warehouse, they might have a nine color printer. And in the printer, we've got about 12 color. Okay. So what does that mean is that we have multiple color to represent one color in, so CMYKC is cyan. You know, it's a greenish color. So you can see here is that we have a photo cyan and a normal cyan. So what happens there is all these printer manufacturers has come up with very sophisticated algorithms to say, okay, for this point in the color space, you know, I will break that color down into an array of 12 colors, such that I can give it as much, you know, ink as possible to represent the color more faithfully. Now, this technology is completely different from how we see things in monitor. There's no RGB whatsoever there. You basically try to use four color to emulate the sRGB. As a result, you can see, you can realize that there's really no simple way to say, okay, I look at a file and know roughly what the RGB numbers are. I can, like, you know, just get a print faithfully. That's not going to happen because there's a huge translation layer. You know, depends on which kind of ink set you have. Now, if we take one step further, if you print onto different paper, you have thick paper, you have matte paper, which is like more subdue, you have high gloss paper. That's the gazillions of paper. A color, same color printed onto different paper will look dramatically different. So that actually brings out the question. How can I making sure that the color I'm seeing is the color I'm getting in the physical file? You know, if the ink are different, if the paper are different, how can I do that? So there's a third bit coming in, which is called a color profile, which I'm going to give you a demo. So color profile basically describes, given any color space, here is how I drive the hardware to do that kind of job. So let me show you. So I have four different paper stock here listed. Don't worry about the exact name, the different product we're offering, the paper are different. So that is the same printer. Same printer, same image. By swapping out the conversion definition between color space and what you end up getting, I can actually simulate what the final result might look like on the printout on the screen. So I'm going to show you this. You can see the color actually dim a little bit. What can I do this one without? Maybe I'll move it over. Okay, so metal print is a metallic paper. It's very high shing. So when you get it, it's very faithful. Now, this one. This one is a nine color printer on fabric. So it's not a paper anymore. Because it's a fabric, it reflects less light. It's less shiny. So when you do this, you can see the color actually less saturated. It's more matte. Now, same material, different printer. So this one you're seeing is nine color. And I'm going to show you what it might look like on a 12 color printer. More color you have, the more details you can come out. Now if you're a good photographer, pay attention to the cheek when I switch them again. You know, slight. But when you print them big, it's actually really make a huge difference. So recap. We have a color mode that determine how the light is coming. We have a color space defining how the color coming and what kind of device we're using. We also have a translation layer definition that tells us, okay, how can you translate this color space into the target color space? That's all fine. But, you know, in Python, how can we get this one work, you know? So pillow is totally not working. So at one point, what I did was I used a bunch of guys in Silicon Valley. They wrote this web service called image IX, a bunch of YouTube guys. So they basically bought about two or three thousands of Mac Minis. And then they used Object C, which is a programming language, to code against the QOS 2D, which is a data translation mechanism that the operating system offers. And they make that capacity available via a web service. So for a while I used that. But one day I read, I was like, you know, I can't remember what I was doing. But one day I browsing the net and I found out that actually Photoshop has a TCP server built in. And even better, you can automate Photoshop using JavaScript. So that opens up a huge range of possibilities . So theoretically you can create a TCP socket to a Photoshop port and then let it run whatever you like. So that's how I found this library. It's called Pi PS. It's a very thin wrapper around the Photoshop TCP port. There's no documentation, no doc stream at all. Basically Go figure! For those of you who want to have a play, I want to show you where that... So in Photoshop, preferences under plugins, you can say enable remote connections. If you toggle that on, you get access to the TCP socket. And now I'm going to show you the same photo we've done. Okay, this is the file we were looking at. And there is actually a Photoshop command, let you switch mode. Okay, if we switch mode, switch mode. It doesn't change your color at all, which is quite amazing. So let's do a bit live demo to give us a bit of hope. So we basically import that and then we create a connection and then we connect using the most secure password. Four digits, six digits. And let me show you this line. So basically this line is a yes, three or four standard, so we can pretty much run most of the JavaScript code this way by do see send sync, which is giving the current Photoshop application get to the current active document, change mode, and a parameter. Very simple. So let me do that. Was it successful? Yeah. Successful and then when we see... No, it's not. Okay. Oh, sorry, that was the wrong command. It should be app, not document. Is this right? Yeah, RGB. So now this has brought us to a very interesting topic I want to talk about, which is Python because Photoshop does not support scripting via Python. So I probably have to write JavaScript, but the way I ended up doing it can be useful is that I generate all my data like, you know, data-driven programming, I generate all the data in JSON format, which is language neutral, and I use Ginger template in Python to render to generate those JavaScript. And once I have a huge JavaScript ready to go, I'll send it through to the API. So this way, I can still use Python to still use the Python like unit test and all that kind of like nice thing I can control without using any mock or... And still get the result I want. So I think that's a fascinating thing. And another next year, maybe I will give a talk on robotic. And if you think of robotic, all the CNC machine and you control them by issuing what they call G-code. And G-code is invented by... In the 60s, it has a few characters at the beginning basically represent get your knife up, move to a point, put it down, do a line. And then the rest part are just parameters. It's pretty much assembly language. But because I'm having all my data structure in Python, I generate G-code using Python. You know, just do whatever I want and translate that into G-code. And that's a common theme I'm seeing here is that as more and more features move to the front end that require a lot of JavaScript on the front end and more and more stuff we do are hardware related. The Python has become this middle layer that managing data and manipulating data and changing the shape and do all the processing. And then the output or the presentation, they are controlled by other languages and that's the whole architecture I end up with which I'm very happy with. Especially when you're thinking of like how can you test a JavaScript stuff. I can test them now, but it's never pleasant. And not to mention all these nested stuff. But Python just read them through and the test framework is gorgeous. So that's about it. I don't have any slide except for say thank you. Questions? It's very much Alex. I don't know that we have time for questions. I'm sure you'll be happy to answer questions around this. So there are a couple of notices. Up next is the hacking session which will not actually be any sort of structured thing.