 Right, so my name is Daniel Stone, I work for Collab Relimited, we're all fantastic, and yes, that's part of our logo. So if you came expecting whizzy graphical demos and stuff, this is not it, and you might probably want to go to a more visually interesting talk. This is about as shiny as my talk gets, so sorry about that, I picked the title in about three minutes. So basically the whole thing behind my talk is every frame must be perfect. I've been working on consumer and mobile devices for the last four or so years of my life, and it's been absolutely wonderful. So the problem we have is that the desktop is allowed to look fairly rubbish. You're allowed to have incorrect frames, so when you're resizing your windows are allowed to look horrible, it can look jittery, you're allowed to flush up random bits of colour, you can bring a window up and it can go black and then be repainted. And this is all right, partially because desktops are really fast, so you don't entirely notice it, and partially because people just have really low expectations of computers and I can't fault them. Sadly, when you buy a consumer device, like the wonderful Nokia N900, which you should all go out and purchase several of, they've really done wonderful, they've got java on them so you can wander around the conference like, hey, where are you? I'm in a Haku, cool, so am I. So if you get just a tiny little glitch or a visual artefact on that, it's really, really jarring because you don't expect that your phone does that, and the same goes for anywhere else where people tend to put X, like set top boxes or anything people pay money for that doesn't have a keeping basically. And so, yeah, that causes, when you're working on them, product management come up and say, oh my god, this must be fixed next week. Never mind that it's going to take about four weeks because, you know, it's a tiny flash of black, how hard can it be to fix it. So that's the kind of thing that gets you critical, oh my god, we're shipping next week and we can't ship with these slides. So maintaining X for a consumer device is really, really fun and I can highly recommend it. Right, so the rough laundry list of problems, this has come from having worked for Nokia for two and a half years before that on all the internet tablets and collaboration subsequently working on the N900 and a few others. Most of the things I've come up with, come up again, sorry, and also when I started at Nokia I did make a fairly conserved effort to pump down all the other phone and consumer device x-trees I could find and see just exactly how other people would push it. I tried to find all the other PAP sets again for this talk but sadly they're all hidden, all the SDKs tend to be hidden behind the beware of the leopard sign and downloading, you know, one gig of someone's SDK in a raw file didn't seem like a great idea of closings. This is mostly the stuff from Nokia that I've had direct experience with and what I remember from looking at rubbish packs that's about four years ago. So the first problem is that X isn't Wayland. Wayland's guiding design principle is that every frame has to be perfect. Everything that gets presented to the screen is exactly what you want to see. There's no tearing, there's no sort of intermediate frames, being that, you know, your resize and window gravity kicks in and sort of shoves your content somewhere else and then you redraw your contents and all that kind of thing. So if X was Wayland it would be great but it's not so here we are. Window reconfiguration is the hard one. When you're resizing and moving windows, you're almost guaranteed to get got all four results between the start and finish. Random reconfiguration is absolutely horrible and that's been a large chunk of my life I'm never going to see back again. Video doing correctly is very hard. Our server implementation, it was written by people who generally don't care about this kind of thing. Composite does fix most of these problems except that sometimes you don't want it. On devices we're often extremely limited by memory bandwidth so if you need to do a double copy generally your UI just gets hideous and slight. So you need to enable and disable Composite on the fly and it turns out that enabling and disabling Composite results in your screen looking rubbish and the meantime flashes of black and other news. The initial and final presentation of the server when you start and shut down is fairly ugly. Especially when you shut down, you kind of get all the windows undecorated in some random order and then you get your nice little... Then you get your nice little shut down screen that we paint onto the frame buffer and everything tears. So basically all of this is fixable to some extent. There aren't any particular fundamental problems that we can't solve. So as I said before everyone hacks it to be shippable. So if you look at the XOR quiz there's a bunch of people saying I've got this tiny problem, how do I fix it? Generally there's not much good advice for coming and perhaps that's what I've seen, they hack around it in the worst way as possible. Thankfully for them and their sense of shame it's generally really hard to find these types of apps. But it's also bad because it kind of perpetuates the problem. And yeah, the ones you can't find are probably even worse. So ideally we'd be able to fix this stuff up so people who want to use X in a mobile or consumer device can just install and ship it and they don't need to spend four months looking at code going I don't know what this does but it flushes black so I'm going to comment it out and hope for the best. Right, so Randa comes first by virtue of being the worst defender here. It's built on the assumption that we'd never want to change screen size which seems odd for an extension which exists to let you reconfigure your screen. It's built on the assumption that all you ever want to do is say plug in your projector every now and again or change size once a week or whatever. But it turns out we need to do this fairly frequently. There were, it seems to be less popular these days but suddenly a few years ago a lot of people tried to make games and they were ridiculously slow so they just resized the screen to roughly half. And in the year 900 we switched between portrait and landscape quite a lot so when you start the phone application it does a rotation transition and this used to take was it like three seconds in the worst case? Yeah, so it could actually make you miss your call because by the time you've rotated into the phone UI they've given up. And yeah, the current implementation is just an absolute slight change. Something changed. By God, let's redo the entire thing. In order to rotate I think you previously had to do, had to re-layout all your visible windows six times and re-layouting in GTK still takes approximately forever. So that was quite deeply unpleasant and it's sort of exacerbated by there's no single random entry point that you can say here's my new screen configuration please make it happen. If you want to rotate you have to disable the CRTC then change your screen size and then re-enable the CRTC with rotation. So if you're doing sledgehammer reconfiguration at each step then it's pretty awful. And it also means that if you want to rotate and you're doing a nice whizzy transition between portrait and landscape you have to have a natural black point in the middle where you can disable your CRTC quickly rearrange everything and then turn it back on. We used to there was a nice sort of gradual sweep between portrait and landscape but luckily we changed the transition so it kind of flips around and then it goes black very nicely and everything happens under the curtain and then it comes back up again. And it's also built under the assumption that no one actually ever wants to rotate. Rotation is generally punitively slow and it works by having a shadow buffer and then every now and again when we're kind of done with clients and we get around to it then we do a rotated flip into the transport space and this is partially because there are very few devices left with hardware rotation engines. So it was designed for the desktop which has no hardware rotation and even on the desktop it's unusably slow. And composite switching between the two modes is fairly punitive. If you want to switch between composite and non-composited you get flashes of black and then background and then other and mask. Same don't do that then is a really attractive argument I spent about four months saying don't do that then but sadly in the end really for performance reasons you'll be able to switch composite on and off pretty much with so you have to do it which means we have to fix it. It's largely hackable around from the client side spent about a week or so and the transition between composite and non-composited is now seamless on the N900 but it's really tedious how to get to there. Right, window reconfiguration is again fairly awful. Luckily you don't tend to do it that much in the mobile world which is really nice but basically it's specified to look hideous in between start and finish. You've got your backgrounds and big gravity which basically guarantee incredible frames when you resize something and stretch it it's going to move around the contents for you it might be painting the background for you when you bring a window back up and that doesn't really give you anything useful. All it means is that you get a flash of gray and apps it's partially the applications for example on the X-Term application on a certain unnamed mobile device I'm not going to name it because it's otherwise perfect and I don't want to slander it's a good name but if you set a black background for the X-Term it still leaves the window background as white so when you bring it back up it's going to go white, black, X-Term and that looks awful. So when you're reconfiguring windows especially if you're trying to move and resize windows at the same time as we do when we transition between full screen and not as we've got a title bar on the top then you basically need to be using composite during this stage so if you're in only composited mode what you really need to do is get into composited mode and try not to cause artifacts there reconfigure windows under the curtain wait until they're all done paint the contents and then composite them over again and yeah, all apps and toolkits are ridiculously stupid here they do all the wrong things which is partially our fault for never telling them what the right things are which is partially X's fault for often not having the right answer but they really do make our life difficult here and if someone wants to fix GTK please do are you going to document what the right thing is there? yes at some such I've got the entire limo patch set which I'm going through now and cleaning up and trying to get out and also to document sort of the pitfalls that we've encountered and how to work around them from the point of but that sort of relies on someone doing it so hopefully yes can you name some of the worst problems and proper solutions for stuff that GTK is doing wrong or any other to get your intimacy familiar with some one or two the worst ones the main one I find absolutely infuriating is using a background and telling X that you have a given background colour and ten points for having the background colour you tell X not be your real background colour so as I said with the X term thing if you set the background to black when you bring it back up it's going to flash white as X paints the background the apps that told us to paint when it'll flash black then you'll have the X term that's the main one that really irks me but a lot of them are I'll have to sit down and dig up my notes but I'd love to coherently yell at the GTK people it'd be very good window managers are also fairly done in fairness this is mostly our fault because it's impossible for a really good window manager you just can't really do it but a lot of the window managers are sweet I've got an event I'll chuck that on the screen right now and then if it's sort of got about five events in a row it'll do that five times in a row so you'll get five intermediate frames instead of batching them up and going I'll wait until we can present something useful so this does seem to be getting a little better but it's still fairly done when no managers just present exactly what you tell them to in order which is not what they should do So is there an exception with enlightenment? I've never used I've not used enlightenment since 1990 It's good with ease and it's a frame based window manager so it's not going to render whenever you want to update the window manager on the screen at least once if you have something to render Enlightenment may actually be the way forward you can do that you can take that and get through them any day now but you still have a bug in the qualifications sorry? you still have a bug in the qualifications because they're excellent at them you don't have to call it where it was nice don't close applications so even when you've fixed up all these little nipples in the X server it's probably still going to look awful unless you've got a friendly toolkit and browser rendering engine and app and other app order around you where you can say you're doing this please fix it and when you tell them that usually they say I'm not really sure exactly what's going on I think I'm telling that it's the right thing and basically everyone should come right video is pretty harsh as well so we have overlays which have usually the most hideous color you can think of a very tasteful lime green incidentally when I was about 5 I used to always change my mother's background to lime green and the rest of the colors came to fluorescent colors and she didn't smother me somehow which is amazing because every time I play a video and I see color keys ridiculously jarring so thankfully we have tech video going through the 3D engineer for the mobile lane but 3D engines especially in mobile devices will tend to use the power in the world overlays are extremely efficient with power and throughput tech suit video not so much but it does mean that you don't have to have the color key you can use it in other parts of your UI instead of just having your video right on top of the screen and nothing else but yeah so if you very very carefully use the API and make sure that the API makes sure that your order of operations is excruciatingly correct so you always start video and make sure it's there and then paint the color key behind that and then when you're done always paint something other than the other big color key that you want to show and then you stop your video then you can almost not show a color key I say almost because sometimes X will stop your video for you and the color key is there and there's nothing you can do about it so that would be nice to fix at some stage if someone video happy wants to look at XZ that's probably why you use a color key at this point frequently some people use a color key close to black and then you pop a dialogue up and your dialogue is sort of half a video it is a kind of ghetto transparency I guess but yeah most people really don't take much care in how they sequence it they just say oh cool XZ this is really easy start video, stop video and in the meantime you've got color key flashing everywhere and chunks of green over here so so tearing is starting to be a very big problem for us as well previously going back a few years you had kind of QVGA size screens and very much static QIs you pressed a button something would happen that was it but now we've got huge screens the N900 and most of it all of its predecessors are the N900 by 480 and everyone wants nice whizzy animations all the time and you know your background is kind of water bristles and you know you press something and you get a nice little animation coming up and everything's great but unfortunately this means that you're almost suddenly going to have tearing so your animation mostly consists of cute jagged ugly lines going right through and also you're probably texturing from the entire point if you press the upper left window switcher button on the N900 it'll do a very nice zoom out and a gradual blur over the background and it'll show you every window you have so it's actually starting to get complex and dynamic enough that we're hitting tearing quite badly so your two options are either you fix it or you can ever do a horizontal animation ever yeah if you happen to shift backgrounds with very strong vertical lines and then you can move them horizontally all the time turns out this is horrible and you should never shift that but luckily DRI2 makes it possible to avoid tearing properly with the swap buffers stuff which has just been merged to master the next release then you can sync everything properly to be one can't avoid that so that's been something we've been working on for a while and it's fully supported and working now in the Intel driver so admittedly that is only on your desktop but yeah you can avoid tearing now already just hasn't finished in fairness we're only 30 years behind here it's an improvement right so those are the sort of more more fundamental issues which we can't really fix without bumping the protocol or becoming way there's a bunch of much smaller little implementation issues which we can fix relatively easily so the first one being the initial presentation when you start the X server it's not the rootweave anymore we don't show you that nice little black and white X cross cursor anymore so we haven't got there but the problem is when you start something up it'll give you a very nice little swing animation there'll be a corporate logo it'll be very sort of relaxing and calming and then the X server will pop up and just smash black over the screen so fairly popular hack is to let X take an image file as a background and hopefully it's the same one as you've been using for the rest of your startup animation and we should probably more support for that I've got that on my to-do list because I've seen that at least 10 to 20 different path sets where people are putting support for having an initial background file again, same problem with shutting down this is partially not X's fault because people tend to kill the window manager and their next which is probably the long answer but shutting down X you tend to get undecorated windows all over the shop and then you get a nice gentle shutdown well I was thinking about the frame or the image thing what about grabbing the current image from the frame buffer yeah, grabbing the current image from the frame buffer is a very good choice as well it's a much better choice than the most impressive one I've seen and I'm not going to say who it was originally was it would start the X server which would clear the background and then in parallel it would sleep for almost exactly as long as the X server would take to start up and clear the background and then a tiny bit of slack in there and then repaint the same thing back over from user space it looked awful grabbing a from the user space does make a lot of, from the current frame buffer makes a lot of sense but then yeah, you wanted to actually be something useful rather than say the current X mode console or whatever so that's kind of a more difficult one so the side cursor rendering isn't great luckily we don't need to use cursors very often in mobile devices but it almost kind of works on the desktop but it's so enough on mobile devices such that especially if you're using a particular browser which likes to repaint its entire screen all the time you just get the cursor flipping horribly, that's quite a job so I've got a patch set which allows client-side cursor rendering so the compositing manager can just paint the cursor when it paints the frame anyway hopefully that's local yeah, it's also impossible to debug rendering issues so if someone looks at something and says shit black's popping up here or part of my window popping up here and it shouldn't be then how to get from there to actually fixing the bug is really not clear at all partially because the whole prime server separation with pretty much zero debunk tools for that whatsoever makes it a bit tricky partially because the server's hard to debug anyway and so there are xmon, xby, xscope and about a million others that will tell you exactly which commands to find sending over the wire so far I've not managed to make any of them work xscope I think will work for about 3 minutes and then randomly die so one of the things I have on my to-do list is xscope that actually works including telling you exactly what you're painting to whichever target you want and exactly what provoked it so yeah, that's the general frame for them as I said, yeah toolkits not great browsers, gecko is quite terrible it's I don't want to get into that but don't use gecko but then it's impossible to prove without any useful traces so I can show up to the browser guys and say, hey your rendering is rubbish can you please make it less valuable and they'll say prove it to me, show it to me exactly what we're doing wrong and then we can just leave the way with the tail between the lines I'm pretty much all I've got it is a short talk because in theory Eric Hanholt if he's somewhere he's going to be talking about COROGL I don't know where he is so that's great does anyone have any questions in the meantime yep