 Okay, welcome. 4k video lessons learned. Cisco Systems Norway is actually the first I've been presenting for seven, eight years at conferences like this and this is the first time I actually tell a bit about what we are doing because it's actually of relevance to this talk. Usually I talk purely the technical stuff but to understand the challenges that we had it is useful to first see what exactly we were building. So Cisco Systems Norway used to be called Tundberg until we were acquired by Cisco several years ago. We make video conferencing equipment and one of those, oh one too far, one of those is the Cisco Spark codec kits plus I don't make up these names that's marketing so I just tell it how they call it. So we make a whole range of video conferencing equipment from desktop systems to full immersive rooms and one of those is this box and it is primarily meant for integrators. So that means as a company you can call up on an integrator and they buy this box and usually a Cisco camera as well and then you can add your own microphones, your own sources, your own cameras if you want, your own displays. This box has three HDMI inputs, two HDMI outputs so you can hook up two screens, you can hook up a camera up to two laptops for presentations and the box itself takes care of the video conferencing part, encoding and decoding etc. and sending it over to remote sites over the network. But the critical point here is that we have to deal with a variety of displays and a variety of sources and we have no control over what the customer uses. If you sell an integrated system you can control the display, you can control the cabling, you can control just about everything except the presentation source which is usually a laptop provided by the customer. That's not the case in a system like this and that has some makes this a lot more complicated than it otherwise would be. So first, let me see, first some bit of an explanation about the HDMI connector. So this talk is primarily about HDMI. We don't use DisplayPort because pretty much every laptop has HDMI support through adapters or whatever and there are several lines on the HDMI connector. One of them is three differential pairs for data and a differential pair for the clock and that is where all the data, the video data goes through. This is high speed obviously. Then you have the DDC channel that is basically an I2C bus that is used to read out an EDID. The EDID is a piece of eProm that the display has and that tells the source, your laptop, what capabilities the display has. It is also used to negotiate HDCP which is the HDMI encryption and it is used for HDMI 2.0 for what is called the status and control data channel which is new for HDMI 2.0 and that allows a source to interrogate the display about things like error detection and how well it's handling the data but there are any issues. We never had that feedback before so that's new for HDMI 2.0. There's a five volt line. It's supplied by the source. If the sync the display detects a voltage between 4.7 and 5.3 then that means there is a source present. That's according to the specification. There is a hot plug detect line that is extremely important that goes back from the sync to the source or display to your laptop saying hey there is an EDID that I have that you can read now. It's setup. You can read that. Again there is a voltage range that you would have to look at to detect whether there is something connected or not. It was designed in such a way that the five volt line that your laptop provides can power an eProm and be looped back as hot plug detects so you can even read the eProm when the display is completely off. Not all displays support this. A lot of them actually only do this when the power is on but they can design the hardware in such a way that this happens. The important thing is that it can also happen on adapters. So adapters can do the same trick. They may even have an eProm inside. It's an EDID. There are more lines. CEC or your return channel. I maintain CEC so if you want to know more about that look up on YouTube. I recently had a presentation at Kernel Recipes about CEC. There is a lot of information there but that won't be discussed in this talk. Next thing what on earth does 4K actually mean? So it really only means the resolution. 3840 by 2160 pixels. If you're pedantic then there is a variation. 4,160 by 20 I think by 2160 but that's rarely used. This is the 4K resolution that almost everybody uses. It comes in a whole range of frame rates. In practice only 30 and 60 hertz are of relevance. All the ones that are lower are done by increasing the blanking. So the actual bends with and the frequencies used over HDMI remain the same. Next thing is you can send out. I'm still in the definition phase of what this all means. So you can send out video in different ways. You can encode the pixels in different ways. One of them is the common red, green, and blue. Eight bits. You can go to deeper bits so 10, 12 up to 16 bits. Our equipment does not support it. It is generally achieved by just increasing the clock frequency so you can accommodate the extra data. I won't discuss that any further. I'm sure that will have its own issues but since we haven't used it in our equipment I can't really tell about it. Not meaningful. The other one is YUV where Y stands for brightness and UV encodes the color. So if you would just look at a picture just with the Y component it would look like a black and white picture. This is ideal for video. What is nice about it? It separates the detail of the picture. That's in the brightness, the gray colors from the color itself. And the eye is much more sensitive to brightness differences than it is to color. So it allows you to reduce the amount of color information. And for the eye it still looks good. If you have a video it still looks fine when you do, when you reduce the amount of color information. So you have YUV 444. There is still each pixel has its own luma and color information, chroma information. YUV 422. There you halve the amount of color. So for every two pixels, every pair of pixels has just one chroma information. Totally interesting since under HCMI it will just add padding. So you do not actually save any bandwidth with this. YUV 420. That becomes more interesting. There every group of four pixels shares one chroma pixel. So that means that you have halve the bandwidth of RGB. And that's actually how it's implemented in HCMI. However, not every display or source will support this. It's a relatively, well you wouldn't say that, but it was really made, it's a bit of a hack. It was created to allow old HCMI installations without the high-speed versions of HCMI to still do 4K. Then you have a number of different pixel clock rates that you can use. I'm just rounding it up, saying 148.5 takes too long. So it's just 150 for me. So you have to 150 MHz. That is the old 1.2 HCMI. F to 300 MHz. HCMI 1.3 and 1.4. That doubles the rate. And HCMI 2 goes up to 600 MHz. So what is nice about YUV 420 is you can actually do 4K at the old bandwidth of 150 MHz. 420 at 60 Hz and RGB at 4K at 30 Hz. You can do a 300 MHz. If you want to do RGB 4K at 60 Hz, you will need 600 MHz. HCMI 2.0. The problem in our products is that we use it both for video, via conferencing. So there we could use YUV 420. That would be enough. By the way, all Blu-ray and Ultra HD Blu-ray, they all use YUV 420. So you don't gain anything if you're playing back a video by going any higher. But when you connect a laptop and you show a presentation, that is all graphics. And using 420 with subsampling means that the cores are a little bit bleeding. It becomes a little bit fuzzy. And that's not a good experience. So for presentations, we want the best possible quality. So we want to use RGB. But the frame rate doesn't matter that much. I don't have to show this presentation at 60 frames per second because not a lot changes. If you use a system as a desktop at the same time, so we have some systems where you can do that. So you can connect your laptop to it and use it for work. Then again, 60 frames per second becomes important because 30 is too slow for the mouse. You will notice that it's lagging behind a little bit. So if you want to connect a TV to your PC and use it as your main screen, which I do at home, it's by the way fantastic. 48 inch screen, 60 frames per second. I can highly recommend it as your work environment for developers. But you need 4K P60 RGB for that. 4K only refers to the resolution. It says absolutely nothing about which bands with you using, what's sub-sampling. And the annoying thing when you go out and you look, okay, I want a 4K TV, they don't tell you. You need to look very carefully in specifications. And quite often they still do not tell you. Then you have to go to the forums where people actually tried connecting the TV to a PC. And they tell you, yeah, unfortunately it uses 420, so you get sub-sampling and you can't use it as a PC for your PC. This is a common threat throughout my presentation. That's the terminology that sales people use is extremely confusing and it's not detailed enough. It's very difficult to figure out what the hell it is that you're buying. So what does this all mean? So if you want to support 4K P60 RGB, that's 600 megahertz. That is at the edge of what is possible. That requires really good cabling, really good and very clean signals. You really need to check that with an oscilloscope, check the eye diagrams. Are they clean? Do we have enough margin there? Clock signals, voltages. And when you measure it as a source, it might be still within spec, but you add a long cable and at the end it won't. So you need to have enough margin there. You really need to pay attention to that when you design the hardware. I'm not a hardware guy, so anything regarding hardware and FPGA that I do wrong, all my fault, but they checked it so it should be okay. The other thing to realize that when you measure on a device like this is that actually the clock, TMDS clock runs at a higher frequency when you're doing 300 megahertz then it does at 600 megahertz because at 600 the divider changes. So with 300 megahertz you do 10 bits, transfer 10 data bits per clock cycle. For 600 megahertz you transmit 40 bits per clock cycle. So actually the clock frequency is half that at 600 megahertz compared to 300 megahertz. So you have to test that 300 megahertz as well because it's different. I'm sort of working my way up from, so we had hardware, we're going through the protocol layers and so up further through the stack. So ACMI protocol, one thing that we had major issues with was hooking up our codec to certain screens and it started flickering. We couldn't understand what the hell was going on there. It turns out that the ACMI specification allows you to send null packets, sort of a no operation. You can use that during blanking just to send something. It's allowed by the specification, but clearly some manufacturers never read the specification because they didn't like it. So we had to change the IP to send nothing. And that works. We don't really know the cost because we can't look inside the vendor's code. We don't know what they are doing, but it was really frustrating for us. It took a long time before we figured out that it was related to that. Another nice catch was with reading EDIDs. So we are, as I've shown in the beginning, we have both source and sync. So you can hook up a laptop like a MacBook to our system and then we are the sync for that laptop. So we have our own EDID. So the laptop tries to read the EDID to get the capabilities of our codec. What can we handle on that input? Normally, pretty much everybody in the world that has to read an EDID starts at zero, reads 128 bytes, and it checks if there is another block. Yes, and it reads another 128 bytes. Everyone except Apple, they do random access. So we didn't support that originally in our IP FPGA where the EDID was stored. We were assuming everybody was just reading from the beginning to the end. So when Apple was reading bytes 32, it actually got byte zero. It caused a lot of confusion. We had some really strange errors due to that. So if you design something like that, just remember, Apple always does things differently. So from what I heard, from the people who are researching this, the question is how can you do that because each block is checksummed. So my understanding is that they first do some random access to read out a few bytes, to be honest, I don't know which ones, and then they read the whole block. But just those initial byte reads already give you a wrong offset, so everything goes wrong. This is really Apple. It's weird. Status and control data channel. That's quite interesting new feature for HDMI 2 because it allows you to check your sending data to your display and you can get some feedback. Does it actually understand what I'm sending? Does it get bit errors? Can it lock through all two or three channels? It also can be used to enable scrambling. You need that for high bit rates. It's nothing to do with encrypting. It's just scrambling to reduce interference. It is highly recommended when you write drivers that you can dump this information because it's very useful when you debug. Put it in debug.fs or in the kernel log or whatever. You want to know this so it makes it easy to see how the display is handling your data. Before I start on this topic, I need some water because this is the hard one. Quantization range. Oh my God. So what is it? You have two ways to send. So this is specific for RGB encoding, not for YUV. This is RGB. You can send your RGB data in two ways. Full range where you use for red, green, and blue components, you use the full range, 0 to 255 in the bytes. Limited range goes from 16 to 235. Why do we have limited range? That goes all the way back to old analog TV where you didn't have nice digitized bytes, but you had this voltage and you got overshoots because it was not digital, so you got and it was clamped or they did several things to make the picture look good. So you had the full range was this, but actually white and black levels were at these positions. When they digitized that, they allowed to actually have that additional information so you could see those overshoots and handle them accordingly when you digitize. Another reason for limited range are certain transport mechanisms like SDI where sync information is transmitted, is embedded in the data stream as special values, 0, 255, and 1, and 1. Now there are several values that are reserved, so that is another reason why you use limited range. HDMI spec is a problem with this. It's difficult to understand how to use it, but the main thing if you interpret it wrong, you get really wrong colors. The main one that causes problems that customers will immediately see is if you are sending full range and it is interpreted as limited range, because for example Excel sheets, they have this white and light gray backgrounds alternating in the rows. That light gray is value 16, so if you, sorry, that's of course value 240, and white is 255. So if it's interpreted as limited range, you suddenly lose that light gray information, it all looks white, and they no longer see the alternating backgrounds, and they complain quite rightly. The other way around is harder to see, so what happens is that everything gets a bit washed out, so instead of getting white, you get light gray on your display. It's surprising how long it will take before you realize that I myself, and I know this stuff once ran with a display that had to set up the wrong way, and I didn't realize for months, and then you change it and you think, yeah, this looks much better. The problem here comes with that interpreting the quantization range in the HMI protocol depends on what timing you are using, and this is particularly a problem when you start mixing laptops or IT systems with consumer electronics like Blu-ray players and TVs, because they do things differently. So you have timings that are, let's say VGA based. They come from the PC world. They are full range traditionally. You have timings like 4K and 1080p, 720p, they come from the TV world, and they are traditionally limited range. So how to interpret it depends on your timing, type of timing, whether it's IT or CE, consumer electronics. This is already very confusing. Then SYNX display can support what's called selectable quantization range. If they turn that on, that bit on in the EDID, that means that the source, your laptop, can actually signal what they're sending. They can say, I'm sending full range or limited range, and the display will interpret that correctly and do the right thing. But that requires that sources will actually read that bit and do the right thing, and that's not a given, and it requires that displays enable it. We enable this bit on all of our systems, because it's a major pain if you only allow the default rules. So if you ever make a display, if you ever make a sync, always enable this bit in the EDID. Please, please, please, please do that. If you make a source, always check if the EDID supports it and signal it explicitly. Please, please, please do that. That saves so much really annoying things for us. To check what is in an EDID, just a little plug, there is an EDID decode utility. Recently it was upgraded with a long patch series that I did that syncs it up to the very latest standards, and you can give it an EDID, and it gives you in human readable format what it all supports, including this. Highly recommended. It also does sanity checks, is your EDID sane, so it also helps you when you create an EDID. The same with logging HDMI info frames. An info frame is metadata that you send during blanking, and it gives information like what is the quantization range that I'm using. There, too, there is support in the kernel to lock that. Please use it in your driver. Very, very useful for debugging. Now, this is the theory. Now practice. So the reality is for our codex, people have presentations on their laptop. There are three types of laptop that they use. Mac, Windows, Linux. While engineers use Linux, everybody else uses the other two. They're all from Intel, almost without exception, because Intel is the chipset most commonly used on laptops, so that's what you get. They all behave differently. An Apple Intel driver will always use full range for RGB. They completely disregard all of the rules that the HDMI specification sets. They don't look at the selectable quantization bits in the EDID, they just send full range. Windows driver, they always use the default rules for RGB, so they correctly change between full and limited range, depending on the timings. They don't support selectable quantization range, to my knowledge, haven't seen it work. The Linux driver, fantastic, does everything right. This is from the same company, right? But I know it's different teams, but really, Windows, Apple developers, you can learn something from the Linux guys here. And just add support for detecting the selectable quantization bits in the EDID. That's enough. Just please help us out here. So you have a little problem here, because an Apple laptop will send full range, a Windows laptop will send limited range for, say, 1080p, and a Linux laptop is explicit. The solution here, and it's really the only way you can make sure that you can always decode the information correctly. From our perspective, we can't tell whether it's Apple Linux or Windows. We just get video. That means that you have to enable YUV 4.4 in the EDID, because Apple, and only Apple, if it detects YUV support in the EDID, it will start sending YUV, and that is always limited range. All the others will still send RGB, and they do the right thing for that. So if you want to support all three variants, this is the only way you can make it work. This is a big issue, because if you do it wrong, it is really visible. You look at what people are doing, and I'm waiting to have to be a bit quicker, I see. This is an important one. You need to pay a lot of attention to this. A few display supports selectable quantization range. I don't know why not, because it's really not hard to do. If you go into the menu of the display, they tend to call it a weird name, like black level. They have several values. They don't document what they do, so the best thing is just to try out what looks best. 4K introduces yet another problem. You have Vic codes that basically tell you what timing it is. It's sent in the info frame. If Vic code is zero, it's an IT format, uses full range, otherwise it is a CE format, and it's limited range. Then when they introduced 4K, the HDMI committee added their own HDMI Vic codes for several 4K values. That's separate from the original Vic code. The idea was that you set those, and the older Vic code stays zero. But if it's zero, then 4K, P30, for example, would be full range. But they also say the 4K, those HDMI Vic codes behave exactly the same like all other CE timings, which would mean limited range, and nobody actually defines what you should do. This is really two committees, standardization committees, working at cross purposes and not communicating, so we get a really mess. And based on testing, what we can say is that 4K, P30 is sent as full range, almost every source, or basically all sources that we test do that. 4K, P60, that's the weird bit, they don't have an HDMI Vic code. They have their own nonzero Vic code, and that's limited range. This is very much specific to 4K and is really confusing, and it is basically not documented correctly, and it is complete chaos, quantization range. Very quickly here for drivers, I see receivers in particular, they seem to be poorly tested in my experience when it comes to interrupts. They tend to get into situations, we've seen it with different vendors and at different times, where the interrupts lock up. Particularly if you do tests like quickly connecting, disconnecting an HDMI cable, you get really, it's a really good stress test, and a lot of interrupt handling, their fields, so it appears to be in the hardware that they get in an inconsistent state that you can't get out of. So quite often we have to replace interrupt handling by polling, just to make it work. I've never seen this with, say, platform HDMI drivers, HDMI hardware, that works fine. It seems specific to iSquad C. In the interest of time, I'm skipping the next one, because the next session is also very important, because this is the part that we don't control. We don't control what sort of adapters and cables the customer uses, and there are some really nasty surprises there. So display ports to HDMI adapters, like this one, it's also on the picture, mini display port to HDMI. It's a passive adapter. It turns out there are two types, type one that goes up to 165 MHz, and a type two that goes up to 300. For type two, there is a little chip inside with some registry information, a little e-prom that tells you that it's a type two. Type one doesn't have that chip. So a driver should, a sync driver should detect, or sorry, a source driver should detect whether it's type one or type two, and then it notes its rating. Again, Linux Intel driver, fantastic job, guys. You check that. If you have type one, if you run X-Render, you only get resolutions up to that frequency. Unfortunately, the Intel Apple and Windows drivers do not check for this, so they will just say, yeah, you can do 4K, P30. But if it's a type one, you have no idea if it can actually do that. You might get lucky and it works. It may not work at all. It may work intermittently. You have no idea. And the only way I found to test this, to test an adapter, is to connect it to a Linux laptop, run X-Render, and then I see what it can do. This is a problem when you don't know what your customers are using. They start complaining to you, yeah, I don't see my presentation. Yeah, you have the wrong adapter. And for this one, this is a Lenovo. They actually mentioned it, type two, in extremely small print. Most don't at all. And you have no idea what they can actually can do until you test it with a Linux laptop. Another very nasty one, USB-C to HDMI adapters. They tend to lose a lot of voltage on the 5-fold line. And that can have real problems where you just don't see, if you hook up a display through that adapter, you don't see the display at all. If the 5-fold is looped back as a hot plug detect pin, then it would fall below the minimum voltage. Or you can't power the e-prom on the display or other reasons. You need to check the 5-fold. And a lot of adapters fail that test. They're out of spec. And then you don't know what happens. Some display can handle it, other displays can't. You have no idea what is done there. So that is a very difficult problem. Again, you literally have to list to your customers what are good adapters. Another really big problem with those is that they tend to try to negotiate HTCP encryption, which is a killer for video conferencing because we cannot, if we receive encrypted video, we cannot send that to a remote site. We can only send it out encrypted again to a local display. They have to feed the purpose of video conferencing if you can't show your presentation. So if you have an adapter that just starts trying to encrypt, that's no good for us. In some cases, it can be done with a firmware update. Unfortunately, that firmware update is hard to get and only runs on the windows. So that's a bit of a pain for people with Macbooks. This is really, again, a very big problem and it is basically out of our control. And another issue with HDMI cables, because of the high speeds, if you want to do 600 megahertz, they actually had to set up a certification program so that you can certify your cable, but it actually complies to the specification. If you buy cables and you want to use this, always buy certified cables. The problem is that, again, with labeling, you can label it on the cable, but most of the time they don't do that and it's only labeled on the packaging. Which is fine when you buy it, but then you've used it, you've thrown away the packaging and you throw the cable in the box and two months later you pick it up, but you don't know what it can do. There are no rules that there's nothing in the specification that cable or adapter manufacturers have to label their cables or adapters according to specific standards. So you don't know what you have. Finally, another issue, five-fold. Again, with very long cables, you get a voltage drop. I had a great cable. For several people it worked fine. It could do 600 megahertz, no problem. I hook it up on my laptop. I don't see the display at all and that's because of the voltage drop. It's just completely unknown whether or not that will work. It depends on your whatever environment you have. So manufacturers pay a lot of attention to the high-speed signal and then they forgot about the stupid five-fold line. In general, our experience with HDMI accessories is that it sucks. They're very difficult to get good specifications about what you buy. They just say, yeah, it's a 4k. It's meaningless. It could be 150 megahertz. It could be 300. It could be 600. You don't give me any useful information. They're poorly tested. There is no, as I said, there's absolutely no standard of labeling cables. Like in Ethernet, you can see a cut four, cut five, cut six, cut whatever, what the highest number is. It's standardized how they should be labeled. There's nothing like that for HDMI. And if you think HDMI is bad, wait for USB Type-C. We're looking into that. It's an utter, complete nightmare. You buy a USB Type-C adapter. You don't know what it can do. There's nothing on it. It could be USB 2. It could be 3. It could be Thunderbolt. You have no idea. I think that it's really, really bad from the committees that make these standards, that they don't have anything for that. Because you have no idea what you're buying. Some resources here you can find the EDID, decode utility. I think it's very useful, whether you're making EDIDs or whether you just want to read an EDID and see what it can do. My email. And I have, well, a few minutes for very few minutes. Say one or two questions. Oh, wait. Mike. Can most of these problems with HDMI be solved by switching to a much more modern protocol like a DisplayPort? No. Not for us. So DisplayPort uses, the only place where DisplayPort is used is in PC monitors and PCs. You don't see it outside of that. So all the installed base for HDMI is huge, far larger than that for DisplayPort. We just cannot offer it on our systems, because it's just not common enough. And we typically, the displays that customers use tend to be TVs, large displays, not the small ones that you use for a PC. And they're all HDMI. So we, it might change at some point that something else comes up. As I said, HDMI USB Type-C, that is definitely interesting development. But a lot of the cabling problems and labeling is way worse there. That is really going to be a big problem with that standard. It's not, the technology is fantastic. The fact that you can insert it in either orientation is, it's such a small thing, but it is so nice. But it's all let down by very poor specification. You don't know what you buy. Very fuzzy and very unclear what is happening there. All the questions? Wait for the... You spoke about long cables could being a problem potentially. Do you have a range of length that are safe towards not safe at all? Depends on the bandwidth. So 300 megahertz, you're probably okay for up to four or five meters. After that, the quality really becomes important and you need to do very good quality testing. You can get longer, but if you just buy it off the shelf, it might work for you, but not for someone else. That's the big problem. So if you simply do not know, and it is... So for 600 megahertz, at least you have a certification program. So you can check for that. For 300 megahertz, you don't. So you have no idea what you're buying. Okay, thank you. Okay, one last question. Everybody wants to go to the next session. Okay, thank you very much.