 Hello! Welcome to my talk about using MIP-DSI as main display interface. I'm Marcel Ciswiller, the Software Team Lead Embedded Linux BSB at Toradex. I joined Toradex in 2011. I basically spearheaded the Embedded Linux adoption there. I introduced an upstream-verse policy, and with that I got into top-10 U-boot as well as kernel armsock contributor at times. And our latest platform is the Industrial Embedded Linux platform, Toryzen. And it's fully based on mainline technology. It's using mainline U-boot with Distribute, KMS, DRM graphics with Etnavive and Nuvo. And it uses ODIRA update with OS3, and also supports containerized workloads with Docker. What will we cover today? I will introduce MIPI-Display Serial Interface, MIPI-DSI. Then I will show you the display adapter system design that we've chosen for Verdin. That is the latest system-on-monule family from Toradex. And I will then also give a quick introduction into the Linux DSI subsystem. Then I'm covering the DRM stack DSI bridge chip integration. We also look at the DSI bridge chip ecosystem. And I will talk a little bit about bridge chips that are supported in mainline Linux. And then the next part is basically about the auto-detection. I will show you how we implemented that using display adapters. And they have an earprom and based on that contents we can auto-detect which display basically a customer is using. I also show you how we integrated that in U-boot. So U-boot is reading the earprom contents and then selects an applicable device relay basically. I also talk a little bit about how you can integrate that into a U-boot fit image. So for a final product that's a really nice concept to have it all in one image. And you can use port specific and display adapter specific device 3 overlays. And I will sum that up with a live demo where I actually can show you a port that I have here which has this whole DSI auto-detection implemented. Okay, let's dive into the MIPI display serial interface topic. It's basically a specification by the mobile industry processor interface alliance, the so-called MIPI alliance. And it's basically about the high-speed differential signaling point-to-point serial bus. And it's used as an interface between a host processor or a SOC and a display module. Whether that is a discrete kind of display or like in our case when you go through a bridge chip. It's a high performance low power and low electromagnetic interference EMI solution. It compared to kind of legacy parallel displays, of course it has a very reduced pin count. And another goal was to have compatibility across different vendors. And usually you have like one high-speed differential clock line and then one or more data lanes that actually carry the actual pixel data, sorry. Then for transferring that, it's one differentiates between a low power and a high-speed mode. While the high-speed mode is usually unidirectional basically just, you know, finding out the display data, the low power mode usually also allows some bidirectional communication between the display and the host processor. From a specification point of view, the initial version came out in May 2006. The current version is version 131 that got released in December 2015. There is also a successor specification I just quickly want to mention, whose initial version was in January 2016. There is now the version 11 out, published May 2018. And that one has support for both DeFi as well as the newer DeFi. And it supports ultra high-definition, so not only 4K, but also 8K. So it's kind of for future applications. Okay. Then how does that look from the layers? So in the physical layer, it uses Meepid DeFi. There are various versions which basically increase the speed of the lanes. So it started with one gigabit per second lane and can nowadays with DeFi 2.0 go up to four and a half gigabit per second lanes. And then on the higher layer, you have the Meepid Display Command Set, so-called Meepid ECS. And that also incorporates display stream compression, DSC, that is basically a standard video electronic standard association the VESA guys have done. And nowadays on higher end SOCs, it's a de facto standard. And unfortunately, as it comes mainly from the mobile industry, it's quite hard to get long-term available discrete Meepid ESI display panels. So for a long time, years ago, there were hardly even any panels available. That changed a little bit with a lot of kind of make-up boards pushing to that. So nowadays, you can, for example, for the Raspberry Pi, you can get such a display. But keep in mind that a lot of these displays are actually at the end just basically a DSI bridge chip again and then use a regular parallel or LVDS display on the actual display panel side. On such bridge chips, they can convert from pretty much or to pretty much any kind of display interface like RGB, LVDS, embedded or regular display port or HDMI. Now I want to show you our display adapter system design. So this basically shows kind of one part of system on module design. So you see there the DSI mezzanine connector. That's basically the connector that will go to such a display adapter. And that basically is a generic system concept. And the idea is that those DSI display adapter boards, they can integrate various bridge chips. So it also has an e-proma on it to store identification or parametrization data. And then this kind of board-to-board mezzanine connector. On there we have a meep DSI, of course, with one clock lane and up to four data lanes. Then we have a bunch of GPIOs, for example, to enable backlight or if there is also a touch control integrated for the touch interrupt. We also have two independent I2C buses. The idea behind that is depending on what actual display you might connect, it might use that for like DDC edit. And if you now have a bridge chip, you basically could even have kind of bridge chips kind of in a chained configuration because you don't know what your actual display might do if this is a display port or an HDMI display. So we try to have an independent I2C bus for that. Then also further signals might be PWM for like backlight brightness stuff. We also have an optional I2S for integrated audio, which can be found basically in HDMI or in a display port. And we also use generic kind of system control signals that we have available on our module standard, which is basically power-enable mochi that is kind of our abbreviation for module out carrier in or sleep mochi or reset mochi. So those are kind of generic system signals that are also available. This is basically one of those modules in that family. We call it the Word in iMix 8M Mini. It has basically an nxbi.mix 8M Mini system on chip. This one has a single display controller. It's basically of the LCD kind of family. If you're familiar with the link stack there, then as a display output, it really only supports mid-bDSI output with up to four data lines and as kind of the DSI controller IP. NXB uses the so-called Northwest Logic IP. From a DeFi point of view, it supports version 1.2, but only a maximum data transfer of 1.5 gigabit per second, while in theory DeFi 1.2 would also allow for higher ones. Then from a resolution point of view, you can do like full HD, 60 Hz, or even kind of a wide format, this 1800x1200. Then now let's go and look at some of these display adapters. Those are the ones I can show you at the demo at the end. This is, for example, the Word in DSI to HDMI adapter. It integrates a Lonsium semiconductor LT8-912B mid-bDSI to HDMI bridge. This one is HDMI version 1.4 capable. It can do full HD at up to 60 Hz in 8-bit RGB kind of fashion. It also has another chip on it, the so-called SDHMI 2C1. That is kind of a single GPSD protection and signaling conditioning chip. So basically that's that little chip just next to the connector that will basically do all this kind of fancy stuff with the signaling that you can easily meet the standard HDMI type A kind of specification. And then of course also the year prom that we talked about so that we can actually identify it. And of course there is a lot of power stuff involved and some kind of interrupts. For example that chip has an interrupt that basically it can interrupt you when it gets a new screen or something like that that you could kind of renegotiate things like that, hot block things like that. Another adapter that is more common in embedded world is the Word in DSI to LVDS adapter. That one uses a Texas Instrument SN65 DSI 84. It's basically a mid-BDSI to dual-link LVDS bridge. So it supports configurable single or dual-lane LVDS up to like kind of this wide full HD resolution or in a single channel this kind of extended HD in 60 frames also and up to 24 bit per pixel. This one also integrates a touch, capacitive touch controller. Basically it allows to directly connect. It also uses from a connector point of view that kind of connector that we also sell this Toradex capacitive touch displayed, 10 inch one. Then let's have a look on the Linux side. How does that look? Basically this DSI subsystem basically it has one piece the DRM mid-BDSI core that basically common logic and some helpers to deal with the mid-BDSI preference. And then basically you have a display engine driver and I also show there the actual calls. So they basically register into the DRM core and then basically have the DSI part, the DRM mid-BDSI core and there in our case the actual DSI host controller is this Northwest logic one. So the NVL DSI and that one host register. And then we can have one or multiple bridges kind of in a chain configuration that goes through the DRM bridge core. There for example in the DSI to LVDS bridge this TA SN65 would then basically call the DRM bridge add so it gets added there and then on a panel side you basically for LVDS there is for example the generic kind of simple LVDS panel thing, the panel LVDS and that one would be added with kind of panel add. That's kind of how it works on the interface side. How does the actual kind of data now get out to the display? So you usually have a display engine driver. So that might be the LCD if and of course the whole kind of rendering. You might have a GPU support for that. For example on the I.M mixes with Fetnavif and then that kind of renders into planes and through the CRTC it then gets fanned out through a DSI encoder in our case. So that will be this kind of meet the DSI Northwest logic thing and then there is a concept of DSI connector and that's how it flows to data then. How does a DSI bridge chip now integrate into that? So from a DRM point of view it's basically a DRM bridge and these are these DRM bridge functions. So there is an attach, enable, disable and the DRM bridge add. So that is into the DRM bridge side of things. And then of course you have the DRM connector side and there are the DRM connector functions. You have to fill modes, detection, destroy and then there are also some helper functions basically to manage get modes, mode valid. So those are used to negotiate the mode and make sure that it's driven by the valid mode. Then there is the connector init, helper add, attach encoder so that's where the meet the DSI encoder kind of gets hooked in and then of course it calls the DRM panel attach which then will basically call into the next one which is then whatever panel is also kind of attached there. From an I2C device point of view it's an I2C device on a bus and it's basically detected using regular ID table or off match table. And of course a common kind of design element there is to use RegMap so basically to obstruct the register of such a bridge chip one can use regular I2C RegMap so with the DRM RegMap init I2C and I2C set client data to actually kind of publish that and on the meet the DSI device side there is this meet the DSI device register fall which registers it and then once it's ready you can attach it with meet the DSI attach. Let's also talk about some kind of bridge chip integration pitfalls that we have met along the way basically one problem kind of as usual is that a lot of those data sheets seem to be super secret so it's kind of tricky especially with some of these kind of Chinese vendors they don't seem to want to give those out easily and then of course the vendors themselves they usually only have some kind of engine downstream driver if you're lucky or even just some kind of bare skeleton driver only where you can kind of guess at the register set and also kind of in what combination and order stuff needs to be done and then even if you have some kind of code of course it's all very hard coded so it's not very user friendly in a sense that if you're trying to support the generic kind of system module concept like we are trying to do which we basically would want that our customers could more or less hook up any display then to that display adapter that makes it rather hard so for example the current LVDS adapter implementation that is kind of hard coded for single channel because there is a version of that chip that only supports single channel that kind of was the only driver that was halfway kind of available so we're still looking into actually working more on that then one problem we saw is that the whole link up sequence stuff is not well documented if you have some kind of a skeleton driver you're not sure in what order and by what things that will be called and that may require a lot of trial and error especially with like the other problem we also saw that of course usually in embedded devices the stuff is rather limited you have some limited kind of dividers frequencies available either on the controller or on the bridge side or even on both and you kind of sometimes have to do some trickery to actually find some combination that halfway works so in our case we had to kind of get a little bit outside of the spec of our actual display and then it suddenly worked fine because all the other frequencies there we just had problems that this north west logic thing in the NXB chip just didn't want to drive the TI chip basically somehow so you might have to do some tuning on this frequency stuff okay then how about the bridge chips that we currently use so I can talk a little bit about more stories in that area so on the DSI side we use this Longium semiconductor that's some kind of a Chinese vendor basically rather nice chip it's just not really well documented or anything luckily we found some downstream driver from the ROG chip Linux guys on GitHub so we actually adopted that one but of course unfortunately with such downstream stuff you know how it is so it was some ancient kind of version of the DRM API so we have to kind of forward port it to later one we also had to fix some kind of confusing naming of some structure and the instant stuff and we had to revert the whole kind of driver to be a proper I2C device because before it was just kind of a hacked thing and then also we had to you know have a look at the full register set from some pseudo code that got actually provided by Longium kind of bugging them over months we got some pseudo code from them so we were able to you know integrate more of the register set then we also improved the whole rack map integration there and you know another thing that is kind of a feature is that those bridge chips often have more than one I2C address so it's basically can be handled in Linux with the so called I2C sub addresses so we also properly reserve those now and then another thing we added regular I2C based DCC ED handling so basically when you with HMI adapter when you there have an actual monitor connected you might want to actually find out from the monitor what modes you can do and we integrated regular handling there then also the hot block detection was missing so it wasn't you know supported that you would during runtime kind of connected different display so we added support for that and then of course we saw some problems when we did that because in our case the GPIO is actually not kind of a SOC native GPIO but goes through a GPIO expander because we kind of were you know short on GPIOs and we found out that then of course because it goes through an I2C bus to actually get to that expander that crashed because we used the wrong kind of you know variant of the GPIO get value stuff so you need to be careful there and another thing that was also done wrong on the DSI side was that the bus format was not properly set still not very pretty but it works and further cleanup and upstreaming there is pending how about the other one that was the HMMA one the LVDS one Texas Instruments basically we started off with a downstream driver taken from the Kompulap Yogdometa layer on GitHub luckily they had adopted it already for kind of an item say the mini design unfortunately it hardcoded for a single channel LVDS use case which luckily our regular panel is actually a single channel but of course we would also want to support the customers to basically attach any kind of a panel also to our channel full HD ones so that is pending and also further cleanup and upstreaming spending how about this DSI bridge chip ecosystem so if you're getting into this kind of area it's of course kind of you face a lot of design system design questions if you want to integrate something like that and unfortunately we met that a lot of those vendors they're still very reluctant to kind of mainlining their drivers so neither Lonsium nor kind of Texas Instruments has really put any work in so it's basically us from the community that have to step in there and that's also kind of the reason there are only very few mainline supporter bridge chips I will actually kind of talk about those on a later slide and therefore also there are only very few kind of examples where we can copy stuff from and find kind of solutions to some things that you might face and of course if you do actual hardware and want to sell that like we at Toradex do it's also quite tricky to actually procure some of this actual silicon that there might be mainline driver but you might not really be able to actually buy such silicon chips and then another thing we saw is of course kind of the conformance of course MEPDSI when the specification got developed it was the goal to actually be kind of vendor independent but you know how that goes with other stuff like USB or so if some of those guys are not really adhering to the spec then it can get tricky and so this conformance with various kind of bridge chip silicon can be problematic how about kind of the chips supported in mainline you can find those on the driver's GPU DRM bridge and I kind of filtered out the ones for DSI of course because that was the only thing we were interested here so one thing to note is that you have to differentiate between kind of SOC internal IP that you might also find there so certain SOC chips basically might already have a bridge integrated for example I believe the regular 8M has kind of a already in the chip does from DSI some other kind of interfaces as far as I remember so you might also find drivers for that and then of course also this grid external bridge chips that are and of course you find also directly connected panels so one thing for example you find there is a northwest logic that was recently added for the iMix 8 series that is basically one of these SOC internal IPs then on the discrete bridge chip side there are some analog devices parts supported the 33 and the 35 which are basically VPDSI receivers with HDMI transmitters so basically similar to the consume one that we are using then another one is from Parade this is a VPDSI to EDP converter we also used some customers use that in some designs then from Texas Instruments there is actually mainline driver but not for the LVDS one that we are using which is the 85 but the 86 is basically a DSI to EDP bridge and looking closer at this driver the driver is actually written kind of in an agnostic way so the idea would be that we could really extend this driver to because kind of the back end DSI side would be the same but of course the front end in this case is EDP and in our case it would then be the LVDS another one that is quite common used is the Toshiba ones they have both version 64 at the end for LVDS and there is a version 68 or 78 that is basically for other display interfaces then on the discrete panel side like you already mentioned there is basically one of the readily available ones is the 7 inch Raspberry Pi touch panel and that one actually inside it uses one of these Toshiba ones actually a 62 at the end which you won't find the discrete kind of tier and bridge driver but in the Raspberry Pi kind of panel driver it's kind of hacked in there and this is a DSI to parallel RGB bridge basically so I guess they just had the goal to kind of mainline that panel but I think it would have been kind of smarter to do that in a more generic fashion to also have a regular kind of bridge driver and then just use a regular panel simple thing on top okay now let's dive into the auto-detection topic so basically the straightforward idea was to well why not just store the device tree in the earprom now why not just using some kind of FTD features in UBOO to just change some disabled versus ok thing in the device tree but unfortunately if you look closer at device trees in this area you have this kind of graph device graph linking stuff with the end point and the remote end point so it's not so easy to kind of just you know set something to ok and have that work so the second straightforward idea was to just you know store the overlay in the earprom unfortunately at least the earprom that we have chosen because they were you know earprom they also cost something so in the two case eyes it's kind of hard well you could maybe compress it or something but it's still very small so as a compromise we now just store kind of identification so called configuration block and then just use that to actually apply a device tree overlay so how is UBOO doing that we already have this kind of config block handling code for NAND and EMMC integrated we generalize that to earprom and then we have basically a table of this product IDs that will then map to overlay filenames and of course with hdmi one can also do hot block the deck of the actual screen and it will do ddc edit or one can do custom display specific parametrization and so it's also possible to actually cascade device tree overlay so I can have basically the auto-detected display adapter device tree overlay and then on top of that an actual kind of hard coded overlay that has the panel or monitor specification in it in the LVDS case that's usually anyway the way it's done so it usually requires further parametrization for the actual panel that is connected to it whether it's single dual channel, it's color format and of course the whole kind of timing resolution stuff so if you look at those overlays that's basically you have these fragments on the left side you see the hdmi one on the right side the LVDS one I don't want to go in too much detail but basically you can you have like a fragment for the iSQC bus which you set the speed and ok so this is actually kind of the ddc one the top one and then you have the fragment the actual iSQC bus where you have the bridge chip on you see that one at the address 48 and you hook up the other bus to do its ddc bus you have like op block detect and all this kind of stuff and then of course you see this device graph linking stuff with the ports endpoints remote endpoints and the last fragment actually exactly that that's from the dsi controller side it's port that links back to the actual bridge and in the LVDS case it's a little bit more complicated because we also have stuff like the PWM backlight we also have the touch controller things like that so there are actually some more fragments there ok now how can we combine that in a fit image basically you can store in a fit image that's the whole point you can store various images so you can have for example you know different device trees for the board we actually for example have there is a Wi-Fi and a non-Wi-Fi variant of that module available so you can have those kind of base device trees and then you can have device 3 overlays the so-called TTBOs like the one for the LT8912 so the HDMI one or the one for the you know for the LVDS Texas instrument one and then you can further combine that into configurations and there it actually allows kind of there are two ways allowed you can have kind of fixed configurations in this case here I show like a configuration 0 and configuration 1 and then you could of course also integrate overlays there but that would you know the more overlays you have that would give you much more configuration and that's why it also is supported that you can basically load the base configuration and then some further overlays you do that with the boot.m command basically you can have like the load address where you have that and then you can say like on config add and then that configuration and then you can just continue with like pound and then such device 3 overlays so you can have like basically a configuration which just has the separate device 3 overlays like that to plant. Then of course also some pitfalls that if you play with device 3 overlays that you might face one thing is of course some simple overlays they work just fine and then you find out you suddenly get this kind of error at least it says nicely that you don't have the symbols in there so you need to actually compile those with symbols inside so if you have more complex device 3 overlays then usually it's not possible to apply them without those symbols. Then another thing that I met is that if you're referencing nodes via hex addresses in the device 3 overlays all of this is actually case sensitive so kind of the base device 3 if you want to reference back to a node from the base device 3 you have to make sure that this has the same case usually I guess everybody uses lowercase letters in hex numbers but at least in some kind of downstream Manning's piece stuff that was not the case so if you look forever why it doesn't work but then I found out it really is a casing of those hex numbers another thing that is quite useful when troubleshooting what it's actually going on and what actually gets applied is if you actually boot into Linux you can use the device 3 compiler on the target board itself I just old package installed it into my image and then you can actually use the input format FS which understands this proc device 3 kind of notation and can actually dump it back into kind of a readable format in basically device 3 source format then you can actually easily see what really got applied if you have multiple overlays and then you know at the end you can't figure out what happened with all that stuff that's very useful okay I guess then we are ready to get to the demo for as a first part in the demo I will just show the console of that board basically you see here on the left that is the Aliyah carrier board with the HDMI display adapter and that kind of module on it so I will prepare this combination here on my table I will show the actual board at the end but for now I will quickly share my screen and I have here the console ready somehow and I turn the board on so it boots up and you can see basically of course it all goes a little bit fast so let me see that was the HDMI one let's see LVDS1 power it back on and it will boot into the into the other variant the thing that we are showing here is the Toradex easy installer and I can now check actually show you take the camera so that would be that board here you see the LVDS adapter I will show this kind of 10 inch panel I can boot it up again so it basically outer detects that display adapter and then initializes the 10 inch display and it then basically loads this easy installer configuration and I can turn it off I can put in the HDMI one and I can turn that one on and it detects that one and basically boots that one up and shows that on the HDMI screen here okay that's basically it put that back and then we can do the question and answer session still have like 5 minutes for that let me quickly look at that whether the low power signal is singland I believe it's both with differential as far as I know and it's just using only one line okay that would be that one more questions whether an end panel like LVDS supports partial update and how will the bridge propagate that I guess that will basically be a matter of the actual bridge chip that you use I guess if that one if the bridge chip supports such partial update I guess it would work it would basically I guess usually that is called kind of smart displays so you would probably need some kind of frame buffer also on the bridge chip side so I'm not sure I have not seen any such chips but I haven't really looked for it so I would assume that should be possible but rather special use case I guess okay this one let me see what else we have oh yeah whether we consider doing it in the Linux kernel of course we kind of you know I attended a lot of those overlay talks in the past and I remember that some of those actually got rather almost a little bit violent isn't it we didn't really want to get into that too much I think for an end product in our case for our customers it's usually you know it doesn't need to be ultra kind of runtime ish that you would connect that during runtime even or something like that so I think they're fine when we load that from your boot site whether the bridge chip can limit any of the DCS commands that I'm not really familiar with that why DRM is required with DSI well I guess DRM is kind of you know the graphics stack in Linux nowadays so I'm not sure how else you would want to hook up a DSI panel I think it makes total sense to have that properly integrated in the DRM subsystem so yeah we used to overlay stuff in Uboot yeah why the vendors are reluctant with mainlining or even giving out documents but that's a very good question well I think they just don't see the value too much I think a lot of those vendors they are very project driven so they just kind of sell those chips into projects and they kind of think well it anyway needs some engineering company to integrate that and fine tune it and whatever not I don't believe into kind of a generic Linux approach too much yet but I guess that is changing isn't it it's evolving I mean a couple years ago who would have thought that you could run open source graphics acceleration on our embedded ports like we now have with many different drivers okay so I believe we just have to push them further and keep reminding them and also of course try to mainline more such drivers and then show them that that's a good idea because they will then sell more chips and things like that variable refresh rate that I'm also not so sure I think that's also something that would it's kind of from the smart display mobile space I believe DSI definitely supports that whether you could also do that with a bridge chip I guess that depends on that bridge chip okay I guess I don't know when it will be really finished here if you have further questions you can there is also slack channel so you can bring those to the to that slack channel I will also hang out there some more and I'm happy to answer some more questions you also find my email address I believe I have some references here and yes I will publish the PDF slides I will just upload them okay thank you very much for your time and have a nice rest of the conference this virtual experience so I'm already past midnight in Europe here have a nice time see you guys thank you