 You're right, it's starting, since it's 2.040. Hi, everybody. Welcome to our talk, which is called Open QA testing on hardware, automated full system tests. Really, what this talk is going to be about today is the journey that we've been on over the last few years when we've been really trying to focus on how to do end-to-end testing with Embedded Linux on hardware, i wneud o'r rhaid, i wneud i'r cwmhae i'r cyflwyno. A gweithio y ffyrdd QA yw ymdyn nhw'n rhaid i dwylo'r pethau peithio'n rhaid i dwylo'r pethau ar y gallwn gweithiol, ond oes yn y bwrdd y gallwn ei wneud hynny, ac mae'r dwylo'r cyflwyno'r cyflwyno'r cyflwyno, ond mae'n gobeithio'n fawr o'r pethau. Cymru, yw'r ddych chi'n gweithio'n gweithio'r cyflwyno'n gweithio'n gweithio'n gweithio. Byddwch yn rhwng ond ymwyl, mae'r rhan o'n Llorin. Rwy'n ymddysgu'r ymddangodd Cymdraidd. I'r cyfnod i gyfnodd byddu'r proje shopaeth Llingwyr i gefnod ymddangodd Cymdraidd, a'r cyfnodd cyfrwsod ffost. Rhwng am Sam. Rwy'n cyfrwsod myfyrwyr sefydliad Gynon, a'r cyfnodd Cymdraidd Cymdraedd, byddwch chi'n fobl yn ymddangodd cymdraedd? Pwysiau yma i fath byddwch We've both worked in automotive over our time of code, think a lot of the things that we're going to talk about in this talk are examples of from the automotive industry, but that doesn't mean to say that it's exclusive to automotive. So I don't want that to put anybody off. The things we're going to cover are quite generic and generally applicable to embedded Linux use cases. Okay, so outline of today's talk, I'm going to spend about 10 minutes trying to set the scene and put some background in place. I think it's really useful to understand why we think testing is so critical and so important and probably still kind of overlooked more than it should be. Then we'll do a recap on some of the previous work that we've presented on this topic and then we will talk about some of the new advances that we've made since then. So why testing matters? Probably this point doesn't really need to be emphasised too much for this kind of audience, but software complexity is increasing at an exponential rate. There's an article here from McKinsey and they were studying the automotive industry. I won't read it all, but they are basically saying that the productivity levels within automotive are falling way, way, way behind the complexity levels. And they particularly point out some of the safety relevant features in vehicles and some of the over there updates and the ability to upgrade. Whilst preparing for this talk, I did read an article that said 15 years ago the average lines of code in a vehicle was 10,000 lines of code and now that kind of makes sense if you think about vehicles that time it was a radio that ran on a microcontroller, right? But now if you look at AGL, which is not even a full system, it's a reference system, it's just shy of 200 million lines of code. So it's an exponential increase. Unfortunately, I couldn't link that article because I can't find it for the life of me, but I did read that. And then we've added a point here which is that most automakers are not really set up to support the software for the life cycle of the vehicle. And we're going to touch on that point in a little while as well when we start to think about how, you know, some products that are out there in the field are depending on things upstream that are supported for less time than the life cycle of the product. And finally, full system upgrades are often rare and avoided. So again, this applies to the automotive industry, but it applies to embedded Linux in general that complexity is really increasing. So a little bit about code think. We are a software services firm. We've got a long history in open source and a particular expertise with embedded Linux. And we have always really cared about and focused on the build test and integration story. So engineering around that, you know, it was often called software delivery life cycles and that kind of thing. So this is important because we see a lot of different automotive projects. So we work with lots of different tier ones and lots of different OEMs. So we have an interesting viewpoint in the industry. And we often see the same kind of things happening in the same kind of projects. So I think when we first come onto projects, it's useful to think in terms of the theory of constraints. So we're often involved in these projects probably a little bit later than we would ideally like to be. Things are often behind schedule or not meeting the quality criteria, whatever it may be. And you know, stress levels can be quite high. That's kind of always the case in automotive, it seems. Everything's always behind schedule. And then, of course, your next project is behind schedule because your current one is etc. So sometimes people have a very good idea of what's going wrong. Sometimes people have kind of wild accusations and point the finger at suppliers and that kind of thing. So the point I'm making is that we try to come at this with a little bit more of a systematic approach. So when we come onto a project, we're always thinking about where the bottlenecks exist. If you think about in terms of the theory of constraints, the maximum throughput of your delivery system is only as good as the maximum throughput of your bottleneck. So it's constraining the whole system. If you think about a motorway or a highway with six lanes of traffic for all these vehicles, if there's a crash in the middle or whatever it may be, roadworks, the maximum throughput is constrained by the bottleneck. So if you are able to alleviate that bottleneck, then you're not just improving the bottleneck, but you're improving the entire throughput of the entire system. And then conversely, if you're spending energy improving things either side of the bottleneck, that's a complete waste of energy. It's just an illusion. It's not really helping you. So we're always looking to remove bottlenecks. And as a result, as I said earlier, we see the same patterns repeating on these projects. And almost always, our involvement in projects will involve improvements to testing. There's a term that a lot of people use, which is shift left, basically trying to do more testing or more valuable testing earlier in development cycles so that we can catch issues as early as possible. The obvious thing there being CI and testing on every single commit or every single MR, whatever it may be, or PR. And then also, we're always trying to improve the reliability and the robustness of testing. And this is almost always done with automation of some kind, again, CI and CD, where you're really trying to establish absolute repeatability of tests, which is very important as well. So this is a diagram which some people may have seen before. It's a pyramid of testing value. I probably need to think of a better name, but there we go. At the bottom, we have unit tests. As everyone knows, they have their use, of course, but they're quite quick to run, cheap to inexpensive, and their value is on the lower end of the scale. Then raising up one integration test, where you might be simulating two different modules together. And then finally, end-to-end testing. So this is where we are really trying to focus our energy on end-to-end testing. We're trying to simulate the exact test environment that would be used in the field, and we're trying to simulate exactly how users will use the software. So naturally, this is generally slower to do, more expensive to perform, but obviously is the higher value. So we have a particular bugbear on a lot of these projects when people will try to say that we've got 95% code coverage of unit testing, or whatever it may be, and people will try and make that as a measure of the confidence that you can have. But I think, as we all know, there are many cases where you can have all your unit tests that pass, but if you haven't done your integration testing, it's not going to work. And this is just one obvious example, which is infuriating because you can't use half the draw. But obviously, if you're talking about critical applications or safety-critical stuff in vehicles or in nuclear power plants, whatever it may be, this can have very serious consequences. Okay, so as I mentioned, end-to-end testing is typically very expensive. And on automotive projects, the hardware is often a very prized resource, especially in the early days, people can really kind of fight over who's getting the access to it and that kind of thing. So if you're able to have a test set up whereby you can test in emulation first, you might have a release, especially on the early days of the project, a release that has a critical issue. It doesn't boot or it's in a reset loop or something like that. You can find that in emulation before you even bother to put it onto the hardware. So you're not wasting people's time in the hardware, you're not wasting that resource. And if we take that one step further, again in automotive, the vehicle is the most expensive testing environment. So you really don't want to be finding issues in the vehicle that you could have found earlier. It takes months to get them ready in the early days, these prototype vehicles. You have to have somebody with a special driving licence who can operate them, all this kind of thing. And in a very extreme example, I have actually seen this on a project. If you brick a unit within the vehicle, you have to phone Dave in another department and he's in another office, so he comes along 20 minutes later. He pulls out the seats, pulls out all the wiring and then you have to replace the unit and everyone's just stood around, your day of testing that you've had booked in is just a waste. So we're talking here about hidden costs and it's, again, it's these kind of bottlenecks that we're looking for. So in summary, the later that we find an issue, the more expensive that it is to fix. So we want to catch them as early as possible. That's a very well understood concept. So in addition, I mentioned automation. We are really trying to focus on automating repetitive manual tasks. These automotive projects, they always rely on masses and masses of manual testing in order to gain confidence. Normally hundreds of people are involved in validation. In big projects, we can be talking about thousands of people. And a really good example of this is the project that I'm on at the moment. I spoke to the test manager about the three week system validation that they do. And it's a very comprehensive test. I said to her, how many times do you plug and unplug the USB media device to test it? And in a three week cycle, they do it 800 times. So if we try and extrapolate that and even if we're quite generous and say 30 seconds per time because you have to make sure the folders load and that kind of thing, that's a full day of somebody's effort spent on that. So if we could automate that, we would be able to free that person up so they can do more interesting testing. They can use the system as a user word, try and find edge cases, stress the memory, whatever it may be. We could do the testing on the hardware overnight because it's automated instead of using it during the day so you're making better use of the resources. And also going back to that person testing it, we can save them a bit of their sanity. Plugging and unplugging this thing all the time. Nobody wants to be doing that, right? Nobody wants to be doing manual repetitive testing. Not to mention that the tests are more repeatable because it's not a human being doing it. Okay, very quick recap on some of the previous ideas and the efforts that we've spoken about. I've just realised I've got to do here, which I forgot to do that to do. But basically, we've spoken on this topic before, ELCE 2021 in Seattle, FOSDEM 22, and then earlier this year we've spoken about this as well at the AGL All Members Conference and then in Prague at the Embedded Open Source Summit there. The first few talks we did were about how we'd combined Lava and OpenQA. So I'm just going to spend a quick five minutes now on a very quick recap on what we did in the initial stages of this journey that we've been on. Okay, so this is kind of a really critical message that we want to articulate here. We have noticed that there is this trend in software at the moment. Modern software platforms are incredibly complex, as we've said, and testing them is often complex and time-consuming. You will often find that a product in the field is supposed to be out there for 10 or 15 years, but it's relying on a version of Linux that is only supported for two years. Now, if you speak to the upstreams, they will say use the latest version. Quite rightly so, that's absolutely their prerogative. And we can see obviously the news from earlier this week. The LTS versions and maintenance is going from six years down to two years. And I think that's, you know, let's face it, you have to fund people to do it. It's not particularly exciting work to backport old fixes and things like that. There's not a huge motivation for doing this and understandably so. So basically we have this gap where projects are missing out on the latest fixes and the latest features from upstream because of the fear of all of the manual regression testing that would be needed in order to upgrade. So we were saying mind the gap. And we thought that if we could help reduce the burden of end-to-end testing and make it more accessible, then we could help to reduce this gap. Quick introduction to OpenQA. So this is, OpenQA is a very well-established testing framework. It's a SUSEI project used on the Open SUSEI desktop. And what we really like about it is that it performs tests how people will actually use the software. So it automates touch events. And you can navigate through the software like that as a user would. You know, we're not interested in having a Windows laptop that has a simulation on it and running those simulations via a UI. As I say, it's designed for Linux desktop operating systems, designed for QMU and desktop hardware. And the tests are based on screenshots comparisons which uses a concept of needles and tests which I'll explain in the next slide. There's a central server, an OpenQA server which you can view the test results and it dispatches jobs for workers and workers provide a specific machine type that's one up per job. So what's a needle and test in this context? A needle is essentially a screenshot of the UI screen that you want to match against and then you have also a JSON file in the same needle and the JSON file is the coordinates. So you can do quite clever things like, for example, if you have a clock in the bottom left that is always going to change you don't want that to fail your test so you can mask that from the test. You can have different areas in the screen with different tolerance levels to do different tests. So it's quite clever in that way. And then separately from that you have the tests and tests actually run on the needles. So you have like a mouse click and assert where it needs to be, you have a timeout value and in this way you can run multiple different tests on the same needles. So you can navigate through the entire UI of the system using this. That's quite a powerful model. Very quick notes on the architecture of how OpenQA works. So you have the tests that are stored in a Git repo and they can be tagged with each release, which is useful for obvious reasons because you probably grow your test suite over time and you can look back to see what tests you run in each release. And then we're using GitLab because it's what we're very familiar with and this will work with any CI orchestration that you choose, whatever your weapon of choice is. You have a build info that launches the image and that spins up an OpenQA worker. In our case we use a GitLab runner. That can be on a laptop or a Raspberry Pi or a VM, whatever it may be. And then the OpenQA server exists. In our project specifically we've put the needles into the OpenQA server as well because they started to get quite large. So we took them out of Git and put them into the OpenQA server, but they can also be in Git. And I'll go into this image a little bit more in a few slides, but that's basically a snapshot of the test results web page on the left-hand side there. So, yeah, OpenQA server, for us it stores the needles and also it shows this overview of the tests. And then the tests for themselves run on QMU via VNC and then you push the collated test results to the OpenQA server. So this is what the results page looks like. It was pointed out to me last night that it looks like an awful amount of black screens here. They're not black screens, it's just a very dark UI that most of those are passing, as you can see at the bottom. We've got one that has failed. And it's really quite useful because you can dig into the test and see exactly how it failed, and I'll show that in just a minute. So we've added AOSP testing to this, Android Automotive, and that's running on a Raspberry Pi. And there's a blog post there to cover that. And we've also added this to the GNOME project. So you can see just on the middle here there's an orange line down the middle and you can move it left and right. This is called the needle viewer. And you can see the screenshot that you expected to get versus the one that you did get. So you can see exactly where the test went wrong. So it's very, very useful. So yeah, we've added this to the GNOME West project and that's NCI there. And Sam is maintaining that. So I'm going to hand over to Sam who can talk in more detail about it. All right. Thanks, Lawrence. So I'm going to talk a little bit about the journey we made testing in a virtual machine and testing desktop operating systems in a virtual machine towards testing more specialised automotive operating systems on real hardware. The simplest one. Well, I'm going to talk a little bit at the end about some custom hardware that we've made as well. So we do have some toys to show off. I'll leave that till later. The video says a lot more than a picture, right? So I'm going to play you a video that is taken from the GNOME Open QA test suite. This is public. And there's nothing special about this video. It's just every test that we run against the GNOME OS distribution records a video of installing it into a virtual machine and checking that it boots. So that's the installer. It's not in real time, by the way. It removes the frames where nothing happens so you don't get bored. There we go. The machine boots. The shell appears. We turn it off. And the test passes. So that's the kind of cool stuff you can do in a virtual machine. We don't want to watch it again, I suppose. Another thing that we've been doing, the kind of the next step, this is again a public project, is testing mainline Linux on different representative dev boards. So I think we have an x86 laptop, a Raspberry Pi and then a RISC-5 dev board. And we're using Open QA with Lava in this case to take a Ubuntu image, build the latest mainline Linux on top, boot the Ubuntu image with the new kernel and just see, does it boot, basically? Does it boot and get to a graphical desktop? And this catches all kinds of regressions in the kernel, right? Even we sometimes catch build regressions or failures in the drivers. Getting to a graphical desktop takes a lot of lines of code, right? Even my colleague Sudip is manually, this is a screenshot of a test which worked. My colleague Sudip is manually mailing these to the kernel mailing list when we encounter failures. There is a plan to automate that and use kernel CI. People think this is useful. But still, where we wanted to get to for our customer project is testing on hardware rigs. I'm going to ask you in the room how many people are working on embedded platforms? So you're testing on weird custom hardware some other time, right? You know the pain. How much of your testing is automated? Hard to indicate with a hand. Some. Between some and none, perhaps. OK, so I'm going to talk a bit about how we've adapted OpenQA for testing more specialised operating systems. The way it traditionally works to test desktops is it uses VNC, the remote desktop protocol, to control the keyboard and mouse and view the screen, which is fine if you have a GNOME shell set up, but a card doesn't have a VNC server and it would be quite an intrusive change to add one. So we developed a small piece of software called QAD, Quality Assurance Demon. If you scan the QR code, it takes you to the GitLab repository. It's written in C. The idea is to statically link it and inject it into the device, and it doesn't require any other changes in the rootFS. So you're not modifying the system under test beyond injecting a binary. And what it does basically, it uses UInput to inject mouse keyboard touch events and it accesses the DRM frame buffer to get screenshots, and then it runs a minimal HTTP server, or HTTPS server, and whoever is connected via Ethernet to the device under test can now remote control the rig. And there's two uses for that. We developed it for automated testing, and then we realised actually there's a more obvious use, which is sometimes you need remote access to a test rig. Sometimes you're at home and the test hardware is in the office, or perhaps it's in a different country, or even a different continent in the worst case. But if you have a suitable VPN connection, you have a VLAN with the test hardware on it, then all going well you can control it remotely from where you are. So in some projects this can be a game changer and it stops people from having to fly to Italy every week hopefully, or whatever your personal pain point is of hardware locations. But we're also using it with OpenQA, and the way we do that, OpenQA has pluggable backends. There's the VNC backend, and we developed a new backend which uses QAD to do the things that OpenQA needs, which is basically taking screenshots and injecting input events. At the moment we only use it with OpenQA, but because the API is quite simple, you could interface it with other testing frameworks. We're interested to hear about that if you do so. Please contribute the support upstream. So we already saw a similar diagram, right? But now we've swapped out the pipeline Lawrence talked about, and we've put some real hardware in its place. So the typical workflow in one of these projects is it starts with a developer, it always starts with a developer. They send a merge request against usually the Yocto recipes. That would be in the GitLab repository. That triggers first a CI pipeline which builds the full image. Hopefully everyone is doing this. Then we have another GitLab runner, which is a special runner on a PC connected directly to a rig. So there's a stage in the CI pipeline which starts the full test suite. This, because it's directed from the runner to the hardware, like it's got an Ethernet connection, a CAN connection, et cetera. It can flash the new firmware on the device, reboot the device, inject the QAD binary, and then start the open QA test runner and run a full test suite. If all of that succeeds, we can be pretty confident that, yes, we can merge the change and we're not going to ruin anyone's day. We're not going to have to phone Bob or Marjorie to come and take out the seats because we've installed a bad firmware. I'm going to show you a quick demo. I obviously can't demo any of the client hardware in this case, so this is a representative demo which is a Raspberry Pi with a multi-touch touchscreen, and it's running the AGL Automotive Grade Linux demo platform. What you're going to see is a very simple test where we open the HVAC controls, this one, and then we go back to the home screen. All it's going to do is inject a touchscreen event and then check that the correct screen opens. So let's see. Also, we're going to switch to a terminal so you can see, on the left-hand side of the terminal will be the EVDev output so you can see the input events and on the right is the open QA and the QAD logs. So there's a simulated touchscreen event. Should have pressed in the right place to open the HVAC control. Let's see if it did. Very good. And now we inject another event to go back to the main screen. And the important thing here is that this is the stock Automotive Grade Linux. The only thing we've changed is injecting the QAD binary. Everything else is exactly how it will be deployed. So there's a trap when you're testing sometimes that you mock everything and then you test that the mocks work and you go, yes, the mocks work, brilliant. But in this case, this is testing the real software just one binary injected to allow remote control. So we're getting as close as we can to end-to-end testing. So QAD is nice and we developed a web UI which this has just been released. We developed a web UI that works with QAD for a couple of other workflows that we had. When you're creating a new test suite you want to test all these different things and you need to take a lot of screenshots and we didn't find an ideal way to do that. It's not very fun to manually step through the device and manually write JSON files with coordinates. So I'd like to introduce Canvas. You can see what we did. It starts with QA quality assurance enverse. It's a very simple web UI just to index a pile of JavaScript which can do two things. Firstly, it can allow remote control of a device that's running QAD. It sends HTTP post and get requests. So it's a very simple remote control interface. And secondly, it gives you a way to create needles. So at the bottom here we have the JSON file which specifies the coordinates and then up here you can't actually see them but it would draw on where are the match like where is it supposed to click, which bits of the image need to match, which bits of the image so we ignore, etc. So if you don't like running curl commands manually now you have a web UI and it makes life much easier. So Canvas has just been released. Again we're keen to have contributions make it at least prettier perhaps. There is some future work planned on our side for QAD. Obviously more ways of injecting events, more types of device that we can model. We do upstream stuff to open QA periodically and we also have changes that aren't ready yet for whatever reason. And we also welcome contributions. So how are we doing for time? I think we have a few minutes left, which is good. This is an example of how we might test can events, so in the HVAC example when I press the fan control we don't have a real fan in the test rig but we can at least check that the correct can event is sent and then we could simulate a reply like the fan is now turning at this speed and does the UI update. That's another future plan. On to the fun part. I promised hardware, right? Lauren talked about a test where the test engineers had to unplug and plug a USB device 800 times. We can't automate that with software but do you want to hold this up? Or shall I? You hold this up and show it off. This is a mini USB-C switcher on one side it has a control port which you connect to a PC and it's controlled over serial so you can connect and disconnect the two input ports via software and then the other port goes to the device under test and it transmits either nothing or the device on the left or the device on the right. So this again opens up a huge world of new tests that you can write testing plugging USB sticks and unplugging them Apple CarPlay and Android Auto which are both very hard to get right and have complex specifications of exactly how they need to work so we can automatically test that. The switcher can power phones so even if they're not connected they won't run out of battery and although it only has two inputs you can chain them together and this is open hardware so we don't sell these individually but you can find a fabrication company link them to the hardware plans which are online and you can have them fabricate them 10 or 20 at a time for your own use. We are happy to help integrating them into your test pipelines though that is an area of expertise. So that's the first piece of hardware we developed for end-to-end testing around the start of the year and we're now on to a more ambitious project which is testing in a box and here is the box inside it contains testing I can see it in here but you can tell from the number of ports on the back that this is designed for connections so as we've built up our testing rigs we accrued more and more pieces of phones and cables until we got to this point which it works but it's vulnerable to a malicious cleaner attack where they disconnect all the cables in the interest of tidiness and it's not very easy to move around or to bring on to a new site so the idea is that by putting everything into one box we can make our lives easier and this is the list of what it contains bluetooth and wifi is an exciting one so you can in here is a wifi chip and a bluetooth chip you can use to simulate things like bluetooth pairing it can emulate input devices like keyboard mouse and it has a single board computer I think it's an arm chip of some sort so you can also run this you know we intend to have bundled on it already a really basic gitlab instance and gitlab ci runners and an open qa worker so that you really can, the idea is you really can go to a new site take the test rig connect it to this using as many cables as you can find and then set up a basic end-to-end test suite as quick as possible work on this is actively in progress we have a couple of people working full-time on this at the moment and there's some improvements on the way for setting up the ci with ci templates and a new revision of the hardware and again this is developed in public on gitlab and I think okay I'm going to wrap up quickly and then we have a few minutes for questions I think so why are we doing all this? partly for business reasons and to make our lives easier but it's also very important for the wider open source community because of the gap right? what we see a lot in the embedded world well and everywhere in the world is that companies will be running a kernel from five years ago because they're too scared to upgrade it they say this one works, we've tested it extensively and if we upgrade to a new one things might break so let's just not upgrade, it's fine you can backport things it's not necessarily that they don't want to upgrade it's that it's too difficult to do all of the retesting so one of the goals of this effort is to really push end to end testing as a way to keep your device firmware up to date the days where you could have one static set of software that doesn't evolve over the lifetime of a 20 years of a device is it gone? we need to be updating devices for the lifetime of the device and better end to end testing can help so that's all I wanted to say do you have anything to add or should we open for questions? well yeah just what it says on here really you know end to end testing is very expensive and it's hard to get right and we're trying to make that more accessible reduce hidden costs and with the right automation and the right tooling it can be a lot easier and we've developed these tools and they're all open source and these custom devices as well to help people with that well that's it alright well thanks a lot okay we don't have a compare I think do we have time for questions? we've got four minutes here if there are any questions okay okay I can see one at the back okay the question was what about devices that don't have HDMI right? ah which would okay good question we don't really work on those but so what interface would they have? okay yeah there is a serial connection in this box so you can an open QAE will yeah open QAE has support for running commands over the serial port so it's possible yeah if it doesn't have a human machine interface then I guess perhaps screenshot testing isn't the first approach you'd go to right because there's no screen but you can definitely still use some of these approaches such as the USB switcher for example if you have USB ports but yeah open QAE itself is designed for screenshot testing first and foremost so it's not always the right choice if you don't have a screen over there okay yeah the question was how do you ensure the screenshot tests stay up to date while people are developing the UI which is an excellent question there's no magic solution to this right? and the design of open QAE accept that screenshots will change so one way they solve this is that you don't actually specify a single screenshot you specify a tag so you could have a tag for example home screen and you could have multiple different screenshots with that tag and it will pass the test if any of them match so one thing you can do is have multiple possible matches there's also it doesn't compare pixel by pixel it uses open CV so you can put a threshold of how many percentage change threshold and you can set that to 95% or 90% and that gets you past the problem where font aliasing has changed slightly and now all the screenshots are off but it also has to be easy to update them right so a key part of this is having the web UI where updating a screenshot is a point and click endeavour and it's low cost to keep them up to date and just to answer that's one of the key reasons why we started to develop canvas to make quality of life improvements for updating the needles and I think the ideal workflow on most projects is that if you have a merge request which breaks the UI but it's not a bug it's an actual UI change you can submit a new needle and a new test with the MR so the test is always green and actually there's a whole etiquette for maintaining these needles you know we've realised all sorts as we've gone through using OpenQA you know we used to create a new needle every time and now we've realised it's a lot better to update them so in a way you almost have to have some maintenance of the needles as well but yeah it's definitely doable but it does need some oversight alright do you have time for one more no I think we've finished right it's 20 past well thanks a lot for coming we'll be around with the two boxes if you want to come look at them or if you want to chat to us about anything else thanks again thank you