 So welcome back. Next session, probably the last one, we have from Daniel Scali, camera applications with the camera and pipe wire. Over to you, Dan. Hi, everybody. Thanks for coming, especially since it's the last talk of the conference. Topic is camera applications with the camera and pipe wire. You might notice I'm not hearing. And this is a talk that we're normally given by him. But unfortunately, he couldn't be here. I was volunteered to take his place. So the summary for his last talks, where he covered similar topics in ELC in Dublin last year, was basically this. Live camera exists, and it turns the new complicated world of cameras into being quite easy again. So please use it. Scope of the talk, I'll do a quick introduction and then cover off live camera, just in case people haven't heard of it yet. And then we'll look at some of the applications in development and using it. And then I'll try to demo some of them. I guarantee that they will all work perfectly. And then there'll be a summary at the end. And if you've got any questions, I'll try to answer them. So hi, I'm Dan. I joined Ideas Abroad in August of 2022, so not quite a year yet. And before that, I was part of the, well, still I'm part of the Linux Surface community on GitHub. And I was working with Intel IPv3 devices. So Intel IPv3 is an image processing, image signal processor that's in Microsoft Surfaces. And the cameras don't work in them or didn't work in them, and we were trying to get them working again. And I started that effort during the pandemic, mostly because I couldn't see my family. And I quite like my family, so I wanted to see them again. So I was working to try and get the cameras rolling so that we could do video calls, basically. So kind of what I am is an up-jump Linux user, really, rather than a developer. Before this, I wasn't in a related field. So what I was really looking for was something easy, like app to install magic camera stuff and everything works. And it didn't quite work out like that. So that's the service, or the brands are available. I think Lenovo does some, and maybe HP, can't remember. Anyway, it's kind of like a hybrid tablet desktop thingy. And the problem was complicated from the start because the firmware in these devices doesn't really describe the connections between the cameras and the CSI2 receivers, probably, which meant that from the beginning, it was really, really hard. On a sane device tree system, you get something like this. So you have a node for a camera, a node for an ISP, and then within them you have nodes for ports and endpoints. And the endpoints have a remote endpoint property that points to the counterpart. So the one in the ISP points, is this working? Yeah, the one in the ISP points to the one in the camera, the one in the camera points to the ISP. Easy peasy. Linux knows how to connect those. In v4l2 core, the CSI2 receiver driver scans down that chain and finds the cameras that are connected to it, and then it knows to look for them. ACPI, which is what we have on the surfaces because it's an x86 platform, it has a similar method of doing it. So it's not as clean as easy as DT, because ACPI is not as nice as DT. But you have the same kind of concept. You have the devices for the ISP, the camera, ports, endpoints, and then a remote endpoint property, which points across to the other one. So because there's a method to do it, you think that's what we had, but it's not. Instead we had something like this. So in the ACPI for the surface you have an entry for the CSI2 receiver, which is called the CSI2, CIO2, and then no ports or endpoints at all. And then for the camera, you just have everything that describes how it should be connected hidden inside this opaque buffer called an SSDB. And eventually one of the other guys found the struct that translates it all hidden inside an Intel repo. And from there we could kind of put it together. So the lanes field in here is the number of CSI2 lanes you're supposed to use. And the degree tells you which way the camera is pointing up, down, left. And then the link field tells you what port to point it at in the CSI2 receiver. So we built a piece of code that we called the CIO2 bridge that fixed that problem, basically. This isn't actually from it because we had to discover it at runtime rather than hard code, the connections. But the principle is this, basically. We use software nodes and with a node for the ISP CSI2 receiver, sorry, and for this sensor and then ports and endpoints within them and then properties that have remote endpoint properties that point across. This piece of code is, I haven't touched on it in a while, but it's undergoing a lot of change at the moment, actually, by Intel and Hans Lihuda. And they're sort of using it to support a lot more of these devices as well. So I've got patches landing less than five minutes ago to use this to support atomized P devices, which are in some of the services, especially the old ones, and also Intel using it to support IPU6 as well, so like the next generation. Anyway, that's the background of how I kind of came across to do everything. But the end result was this. This is the topology that we got once we'd finished. So this is two separate media devices, one up here. And this is for the CSI2 receiver. There's a subdev for each port with sensors connected to them and then a capture-capable video device. And then this is an unconnected port. So it's just left-hanging, basically. And then ancillary devices. So this is a lens controller. It supports this sensor. It should be connected. It's not, because I've forgotten to add support to media control for the ancillary links. So I need to do that. And then in a separate media device, we have the ISP itself. And then that's got output-capable video nodes, which are confusingly called input. And the idea is that you capture data from these ones, and then you queue that buffer that you've got into here, and then you can capture out of the output on the viewfinder. And that is way too complicated for what we were doing. But people were looking around, and they came across this camera project, and they thought, hmm, that looks like it might be able to do it for us. And it turns out it did, which is nice. So the reason that all of that complexity exists in the topology is there's a lot going on in a camera pipeline, right? And it needs to expose, media control needs to expose everything so that you can independently control all of the aspects in there. So like the lens, you need to be able to set the lens position. Sensors, you need to be able to set the output rectangles at the sensor, but you can change that further on in the ISPs. The sensor might have gain and exposure controls. Should have. It's not a very good one if it doesn't. And you need to be able to configure those as well. There might be more controls similar in the ISP. So there is a lot of complexity that's behind modern cameras. And so that's why the topology needs to be so complex. And that's why we need new software really to handle it. If you wrote Laurent's talk earlier, the slide will look a bit familiar. But basically this is the lib camera architecture. And lib camera kind of handles that complexity for you so you don't have to. So at the bottom of the stack is the hardware. And then above that we have kernel drivers for the sensors, for the CSI2 receivers, for the ISP and so on. Lib camera needs those to exist. It assumes that they're there, but they're not its problem, if that makes sense. So that's the kind of thing that needs to be done beforehand. Or I guess simultaneously, because if you're developing a pipeline handler for lib camera, you probably did the ISP as well. Then in core lib camera code, it's got representations of all the things that you need to control, like the V4L2 video devices and sub devices, formats, buffers, frame buffers, the cameras, controls and so on. Where the magic happens is in the green section and this is the pipeline handlers. So in lib camera there's several pipelines and they handle that complexity for you basically. So you need one bespoke for each pipeline. At the moment there's an IPU3 one, which is why it works for me. Rockship, Raspberry Pi, and then there's a simple one that can handle some pipelines that are just like A to B, not much going on in between. If you missed Laurent's talk earlier, go and watch it online. It delves into those quite a bit and it's really interesting. So after speaking to the lib camera guys quite a lot, we got it working. Hooray, working cameras on Microsoft Surface devices. As long as you like QCAM. QCAM is the test application that comes bundled with lib camera and it's not really what you want in a camera application like if you're a normal user. There's no way to control the resolution from the screen or brightness or anything like that. The capture button is kind of there, like up there, but not in an easy place. So the problem is although the cameras work, fundamentally they do work, it's still not a very satisfying experience until you have applications that support lib camera. Lib camera comes with a GStreamer plugin which makes things a little bit better because some applications that support GStreamer then work, cheese works, for example. So if you want to know them, that's nice. And we figured out a way to add really horrible hacky support. If you've heard of a V4L2 loopback module, it makes a video device that you can output data to and also capture from. So you can use GStreamer with a V4L2 sync to pour data into that and then capture from it from other things that just expect slash dev slash video zero. And that does work, but it's not configurable particularly well and it's just ugly. It's just ugly. So we need applications to have support for lib camera to really improve the ecosystem. There's a few different ways that you could add support to an application for lib camera. Oh, I should say, these slides are shamelessly stolen from Robert Maynard because it's just such a good overview. So in the old world, your application would point directly to the V4L2 video devices and try and control them. But there's frameworks that can kind of do that for you, particularly GStreamer, and that makes it a bit easier. And where lib camera lives is to just slot in at the bottom and make it easier for everybody. That's the idea. We've also now got Pipeware, Pipeware supports lib camera now and that adds a whole lot of other things because with Pipeware you get some stuff that lib camera doesn't really do, which is nice. So for example, with Pipeware you can share the video stream between multiple applications at the same time, like with the lib camera where you can't really do that. And also, particularly when you have the XTG camera portal, you can get fine-grained application permission. So when you start an application, it'll say, hey, something is trying to use the camera, are you okay with that? And that's a really nice thing to have for particularly desktop users, which is mostly what I came from. So given there's multiple different ways to do it, which one should you use? It kind of depends on your use case. So if it's a user-facing device on desktop, mobile, that kind of thing, really it probably wants to go through Pipeware or through some other framework. Embedded devices that want to do lots of different things probably ought to use a framework as well. So for example, if it's like an IP camera, you don't just want to capture the video, you don't want to encode it and stream it across the network, and perhaps not just video, but maybe even audio as well. And in that case, using one of the frameworks probably helps. But then if you've got a really custom use, or even if it's just something simple, maybe going direct to the API is the better way to do it. Kieran has an example camera application called SimpleCam that uses the APIs directly, and it's less than 250 lines of code, plus some comments, about the same amount of comments. Oh, yeah. And then there's some things probably want to have multiple ways of doing it. OpenCV particularly, for example, it's on desktop, so sorry, Tommy. It's on desktop, so having it be able to go through Pipeware would be nice, but also it gets used in a lot of places where you might not expect to have Pipeware. So it probably ought to have a direct implementation as well, and we have a GSOC project running at the moment that's trying to add that. So onto some of the actual applications that I want to highlight. This one first, just because it's so cool. This is a streamer that uses the libcamera APIs directly and just punts it to a web server for you, basically, which is really, really nice. If you're into your 3D printing and have heard of Octoprint, anybody heard of Octoprint? Yeah, OK. So this is what Octoprint uses as their streamer. Octoprint is a program that you can use to control a 3D printer over a web interface. And 3D prints go wrong all the time. It's really annoying, actually. They're fun toys to have, but you set off a print and it's supposed to take four hours, and then you come back in four hours and you find out that it broke three and a half hours ago, and that's really frustrating. But having a camera that's pointed at it that you can just check on your phone every now and again to make sure it's going well is a really nice thing to have. But there's a ton of other uses as well. Since finding out about this, I've stuck one out in my garden so I can sit and see what's going on in my garden from the living room, and so on and so forth. It's really, really nice. So this is the kind of thing you get. This is an upside-down image of my office because the stand for the camera was still on the printer at the time. Sorry, my office is messy. Anyway, it's dead easy to use. You just go to the IP for the pie or whatever. It doesn't have to be a pie. Lots of other boards support the camera. And you can get the video. The big one, though, and the one that I originally wanted is video calls. So there was a couple of different attempts at adding support to video calling software. Originally, I think the browsers were looking to add support for Pipewire directly, and then that effort semi-stalled for a while. And what eventually came about is that the browsers are adding support for WebRTC, going through the XTG camera portal to Pipewire, and then Pipewire, of course, supports libcamera. So with that stack, you can use your browser and use video call, which is a really, really cool, quality of your life improvement for people who've got those complex cameras in their devices as long as it's supported by libcamera. This is on the way upstream. For WebRTC, I think it is upstream. For Pipewire, it is upstream. Oh, sorry, ELC and Dublin last year in September? Yeah, we did a demo stand that showcased that. So this was here in surface rather than mine, and to a Raspberry Pi, doing a video call between the two. There were a list of a few problems at the time. Those are reduced now. It's still not completely perfect, but it does work, and you can try it quite easily, actually. For Chromium, the integration is really, really close. There has been, as you can see, a huge amount of work gone on for it. And it looked like there's two merge requests that are still not merged, or at least weren't merged when I last checked, which probably was a couple of days ago now. And the big one is this one, video capture, add back-end portal Pipewire. It looked like it was ready to merge, but then they found a crash that they need to fix first. So not quite, but really, really, really close. And that will be a massive, massive thing. Firefox did merge their version of it. So you can see a two-year effort. It says close eight days ago. I think it was June 15th, actually, something like that. So that went into, that's gonna go into 116. You can download and build nightly now and try it out, and it does work. It's at the moment behind an about config option. So if you're going about config, I can't remember exactly what it's called, but if you just search for Pipewire, there's two of them, and one of them is set to false. You flip it, and then it should work. And it does work. It's not perfect. I had some crashes when I was trying to use it. You can't change the resolution at the moment, it seems. So it's a bit fuzzy, because it's stuck at 480p, but yeah, it still does work. And going from, oh, sorry, I should mention, you need to have an up-to-date Pipewire. So depending on the version of Pipewire that your distro comes with, it might not work. I had to build from master. I'm on Ubuntu 2204, and it was just slightly too old. But yeah, the quality of life upgrade goes from doing shenanigans like v4l2 loopback to try and get your video cameras working, to it just works. It's really, really, really nice. And this is what it looks like when you try and go through the portal in Firefox Nightly, joining a video call. You get a nice dialogue that says, please can I use your camera? It says console rather than Firefox, because I launched it from console, but it's Firefox. Yeah, and you can tweak those permissions from, ha, now. You're supposed to be able to tweak them from privacy settings. They never actually work for me. But if you want to revoke these permissions, the way to do it is delete USR, share, flat pack, DB devices, I think. Other things coming on. So the Pipewire devs are adding support to a lot of other stuff as well. This is OBS, Open Broadcasting Studio, from Christian Shala. He's streaming two different webcams at the same time, and that's going through Pipewire and libcamera. So there's a whole bunch of different desktop applications as well as things that are more useful for the boards and stuff that are getting that support nowadays. So the ecosystem is getting better and better all the time. About two years ago, there was this mock-up published for a new camera app for Nome. I can't remember who... Sorry, I was going to mention the name of the chap who originally came up with it, but I've forgotten now. I'll look it up and shout it out afterwards. But yeah, this is exactly the kind of thing that you expect to have nowadays when you're using your camera, particularly coming from the world of everybody who has a smartphone, you know, an app that looks exactly like it would do on a smartphone is really what you want. And here it is in the flesh. I'm not smiling very much in these photos. Sorry, you're right, Laurent. I should have fixed that. Sorry? Well, yeah. I don't know why I wasn't already happy. I should have been happy, right? Yeah, so this is it running on the go. And it looks pretty much like the mock-up. It's exactly how a modern camera app should look, basically. You can switch into the gallery. You can switch to video mode. The video flip button is there. It doesn't quite work on the go, actually. That's one of the limitations I've found. For some reason, this app, which is called Snapshot, by the way, it used to be called Nome Camera. It got renamed to Snapshot. It doesn't let you flip back to the world-facing camera for some reason. The camera does work in Firefox. It works fine when Q-camera works fine. But Snapshot doesn't like it. But I haven't had time to look at that yet to figure out exactly why. And this is Snapshot running on a Pine Phone Pro, as well. So, you know, the same application that I can run on my go, you can run on a mobile, as well. And it's running through the same stack through PipeWire, through Lube Camera. Ooh, demos. Okay, let's see if some of these work. How am I doing with time, as well? Okay, well, I had. So the first one I wanted to show, just because it really is so cool, is the camera streamer one. So, you just get a nice web page and then, depending on the link you click, you get images. Ah, no, you don't. You get server error. Nice. Doesn't that just make sense? Okay, hold on. I'll tell you what, we'll skip that one for now. I'll see if I can fix it at the end. Bit annoying. Okay, so what I wanted to do was join a video call on my go to this. Just want those. Hooray. That wasn't too bad. Oh, it goes a bit crazy, though. It's watery? Ah, okay. Okay, well, anyway, fundamentally, there we go. So that's running Firefox through WebRTC, through the XDG Camera Portal, then PipeWire, then Lube Camera to some really complex cameras, and it works really well. Apart from the odd flickering. Like I say, there is still some hangups. I can't get it to get above really low resolution at the moment, but it's a real quality of life upgrade. So if you've got these kind of devices and you're a Linux user now, the world's gonna get better quite quickly now. Particularly because it's coming out upstream. There'll be a point fairly soon where this stuff just works out of the box, which is really, really nice. I also wanted to show Snapshot, which I can hopefully do by just showing my screen on here. It won't be too terrible. Let's see. Okay, so here's Snapshot, and there's the pop-up from the Portal. So that's Snapshot wants to use your camera. Are you okay with it? Yeah? And then we say yes, and hooray. We get working cameras on Snapshot. So like I said, on Lube Camera directly, if you try and stream from the same camera twice, it tells you it's busy. But through PipeWire, you can have both applications, both looking at the camera at the same time, which is really, really cool. And so Lube Camera's running all the complicated stuff in the background as not just streaming, but also the 3A algorithms. So auto exposure, auto white balance, and so on. Like if you trick it into dumping up the exposure for a second, it'll do that and then reset it for you. It's all handled really, really nice. Selfie time, hold on. Everybody say rust. And then from here, we can switch to video mode. It lets you take video. And then there's also, you know, drop into gallery. It's got everything you expect from a camera app. And you can cycle through, hooray. Okay, hang on, hang on. So if the downside is, for example, if I try and go into video settings, the browser crashes. So not quite there. Still work to be done, as with everything. The other thing is, on the camera sharing, what I've found is I can't use, like through Piper, I can use both cameras at the same time. But if I'm using one camera in Piper, I can't then use another camera using an application that goes to the APIs directly. It doesn't find them. So work to be done in multiple places still. Is there ever not work to be done? But yeah, as an ecosystem, it's getting better and better all the time, which is really, really, really cool. Okay, where was I? Demos, that was it. That was the end of the demos. So at the beginning, I gave a summary of the previous talks, which was live cameras there. Can you use it, please? The summary of this talk is, look, we've got a bunch of people who are all using live camera. And we think those people are really, really clever. Can you do that, please? I don't mean to imply that if you're not using live camera, you're not clever, but it is a good idea. And if it helps persuade you, I specifically checked and there are Rust bindings. So if you're that way inclined, you can try that. Okay, that's pretty much it. Any questions? How did I do on time? Terribly. Way too fast. Yeah, thanks. Awesome work and a great presentation. And it's great to see that Linux is starting to thread camera subsystem kind of with respect as well as audio subsystem, I would say, so you can share and have single sync, right? Or sources. My question is, you said that this is not yet upstreamed, but to my knowledge, on Wayland, on Sway in Arch Linux, at least camera subsystem works exactly this way. So it's WebRTC, then it's a XGD portal, right? Also with permissions. Is it using something else under the hood that it works already? I think at least for half a year it works. So it's WebRTC through the XGD portal and then PipeWire. So on my Ubuntu 2204, the version of PipeWire that comes packaged with it doesn't have a camera out of the box. So that's the bit that was missing. Like if I try and run the snapshot without running, without shutting down the default PipeWire and then running a PipeWire server that's built from master, it just doesn't find any cameras. Okay, I think, yeah, then that explains. So Arch is just using the most recent PipeWire which is probably has this ability, right? Yeah, I guess so, yeah, I guess so. Thank you. Okay. Anybody else? Homesign. Oh, one more. Oh, I don't think, I don't see. Oh, okay, sorry, do you run the camera on something closer to like DSLR, like professional camera to for taking still photos? I don't think so. And application for that? I don't think so, not to my knowledge at all. Okay. Thank you everybody. Thank you Dan, for the talk and that should be done. Thank you.