 I think we can just get started on time. You can hear me fine like this, I guess. Yeah, wonderful. This is a talk about our gateway. The gateway we built for Gardena during the last two years. And we saw in the call for papers that there was a topic war stories. And we thought, yes, this fits perfectly. Because from our perspective, this is pretty much our war story. And I hope you enjoy it. If you have any questions, feel free to ask during the talk. Or I'm pretty sure we'll have time after the talk. I'm Andreas Müller. This is Rita Schneider. So I'll give you a brief introduction about the company, about us, about why we built the gateway. Then we'll show you some of the technical background. Not because we think you can learn a lot from us there. We're pretty much beginners with the Octo. But because it's important to understand the rest of the talk. Then we go into what we did with open source, how we open sourced our stuff, how we also plan to keep our products open for our customers, and what we achieved by mainlining things. That was pretty much a big gain for us, that we mainlined things. So starting right with the introduction, Rita Schneider and myself, we both work at Gardena in Zurich. We both have been doing embedded development for quite a while, but more with microcontrollers, not so much with Embedded Linux. And both of us had very little experience with Embedded Linux and no experience with the Octo. So that was pretty new for us. I chose this picture to tell you about our company, Gardena. Gardena is a pretty traditional company. We manufacture gardening products. And gardening products like rakes, like watering products. Gardena has pretty much no experience with electronics, very little experience with electronics, and until, I think, five years ago, no experience with software development. So you can imagine, usually at Gardena, when a project is done, it's done. And we no longer worry about it. This is quite a stark contrast to something like a gateway where you finish it and then maybe you maintain it for 10 to 20 years. That's quite a new mindset for Gardena. But now the new Gardena product line that we're partly responsible for is the Gardena smart system. And I chose this picture to represent it to show you on one side. So we have all these products in the garden, like a mower, like watering products, like a pump. And we have the smartphone app to control these. We have the servers for things like weather forecasts. And in the center of it all is the gateway. And the gateway isn't something we really want to sell to our customers, but we just have to, because we need to communicate with the devices. And there's no standard sub gigahertz IoT technology yet where you can just buy a gateway and communicate via that. Or if you know which one, I mean, there are many technologies. There is SIGP, there is Bluetooth, there is Lora. But none of these has succeeded enough to know, OK, we can use that and just no longer worry about the gateway. Or if you think you know which technology is going to win, please talk to us afterwards, because we would be very interested. Maybe a bit of background about the project. The main goal was to lower the hardware costs. We had to have maintainability for at least 10 years. So we want to sell the gateway for maybe five years. And after that, we should maintain it. We usually calculate with eight years, at least internally. And we wanted to have open source compliance. Rator is going to tell you more about that in a second. Challenges was that we have quite a distributed team. We have part of the team, a large part in Switzerland, in Zurich, part in Germany, in Ulm, part in Sweden, because Husqvarna is Swedish, and part in Denmark. We had no backup plan. We needed to have this project successful. Otherwise, I think we would have been in quite a bit of trouble. And we had a very short timeline, one year from a project start to sale. Yeah? Oh, yeah. And important to know is we already had one gateway out. And that was the first generation. But we wanted to make the second generation more technically sound and cheaper. And what we want to achieve with this talk is we want to show you how our gateway works a bit on the technical side. We want to tell you about our open source journey, how we went from being a company that had pretty much no idea what the GPL is or what open source means to the state where we are now. We want to tell you how we benefited from mainlining our stuff. And hopefully, you can show this talk to your boss if you want to mainline stuff as well. Because from a risk perspective, it was very useful. And if we have time, we also have some slides about the pitfalls we had and how you can tinker with our gateway. But if we don't have time for that, it's just like for reference on the slides. Yeah, and we want to say it's not done. So we know that we still have a lot to open, a lot to do. We're not perfect. The system is not perfect. But we're on our way. That's basically what we're thinking. And neither of us are YOKTO experts. So if you think we could do something better, talk to us. The code is now also on GitHub, so you can send pull requests. Yeah, let's get right into the technical background. For the hardware, we have a MediaTek MT7688. And in the future, I'm just going to say the MediaTek chipset, because it's always so complicated to pronounce. We have 128 megabytes of RAM, 128 megabytes of NAND flash, and 8 megabytes of NOR flash. And we have a radio module that's for communicating with the devices via sub-kickerhertz. We need sub-kickerhertz, because we need quite a bit of range, because our products are in the garden. And the gateway is usually inside the house, so we have to communicate through the house, into the garden. Maybe even a garden could be on the backside of the house. Maybe there's a tree in between. So we need quite a bit of range. On the radio module, we have an ARM Cortex M3, and a sub-kickerhertz transceiver from SILAPT. So it's pretty standard hardware, I think. Most of you might know MediaTek. It's very commonly used. This is what our gateway looks like. Or actually, I'm just going to skip to the next slide, because then I annotated what we have where. We have the Linux module, a big part. We have a power supply, the radio module on the right. We have several antennas. You see the PCB antennas. And we have the UART. And the main thing you should probably remember from this slide is where the UART is placed, because if you ever want to play with our gateway, this is where you need to solder on a connector. Is that already you? Yeah, for the software aspect, this slide is here not because it's extremely interesting, mainly because of the unused space. So I explain the rest first, and then you maybe you should get to talk later on. We have two flash memories on that device. The first one is in all. We boot from it using U-boot. We have the environments there, and the factory calibration data, plus like seven megabytes of unused space. More about that later. And on the spine end, flash, we have two root fs, two kernels, and an overlay where the user data goes into. And the update mechanism just toggles between the two kernels and root fs. Then the build system, it's 95% of the shelf, really standard stuff. What we did was having a layer for distribution aspects for some custom packages, another one, then one for the third party packages, one for the BSP for the MediaTek, and another one for the firmware, which are running on our devices in the garden. Now to the more interesting stuff, probably, the open source from scratch. Not in the sense that we programmed everything from scratch, but we started really from the backside. Like when our system got released first time in 2016, pretty much immediately a German news site called us out that we are violating the GPL, and they asked for the source code even, and custom support, like basically nobody had got even knew what we had to do, and the news site did not get what they wanted for, and that stayed like a year or so. Then when I joined the company at the beginning of 2017, a month later I filed an issue. Because of that, I really tried hard to describe all the details, but apparently it was a bit too much, too complicated, did not really get prioritized as I wished, but after a while, a year later, then like 2018, this McCarty, he went to court and got in ruling that the company was violating the GPL has to pay either 250,000 euros and all going to jail, like the CEO of the company. And this helped me a lot. I could write the release. Yeah, the exact details are censored out, but basically it was about CEOs going to jail or paying a quarter million. Yeah, and then suddenly got some traction and we were allowed to work on, and basically what happened is like the new project we did, we just did it right from the beginning. Then we went parallel to that, to our supplier, told them, hey, you should have told us about GPL and so on, and now deliver us the source code, so we are able to hand out the source codes if somebody asks. And new gateway, we put it on GitHub already, like upfront. Yeah, so that was more the software side, and ideally we also want to keep the hardware open. And for me, for instance, having open hardware means if I buy the device as a customer, I should own it and I should have the right to log in and to have access to it. And that's, I mean, traditionally for Gardena, if we sell a gardening device, the customer owns it, so it should be kind of familiar, but still for the gateway it was a bit more difficult to convince Gardena that our hardware should be at least as open as possible. And originally, this is again, the original gateway, we did not allow our customers access to the gateway, we just closed it up or the company that was actually hired to implement it closed it up and we didn't allow any access. But of course, it wasn't done very well and some customer, we don't know exactly who this is, but this is a description from GitHub, someone found out to route our gateway. And this is actually quite an easy way. I know there's another way, but my way would be more complicated, so it seems this way is the best one. So it's quite easy even for our original gateway to access it and to reverse and engineer things. But of course, from our perspective, as I said, we think you should own it if you buy it and we also realize locking down hardware like this is quite difficult and especially if you want to do it right, you have to have hardware support, you have to have like, even from boot to the shell, you have to sign everything and to lock down everything and we don't really want to do that and it's also not really important because if you own the gateway, say if you have physical access to the gateway, as I mentioned before, the gateway is inside the house. If you have physical access to the gateway inside the house, you don't have to hack the gateway to turn the water off in the garden, you just can just much more easily go into the garden and turn off the water there. So really we thought, why don't we just leave the route access open for the customer at least via UART and we kind of proposed or we actually just did that, we implemented an open route access on the gateway and then we got this issue from our quality department. There was some confusion there. I should really stress, we never had the intention of leaving access via SSH open, it's always just been access via UART and you have to solder on the UART connector to have this access and I think that the main discussions we had with our product quality department was to explain exactly this point, that it's not about you go via internet and you just log in and somebody else log in, no it's you open the screws on the back of the gateway, you solder on the connector and then you have access. And we had this discussion quite a while, they also were worried about service department, what if someone accesses the gateway and later the gateway ends up broken and we have to replace it, for that we kind of discussed yeah, well there are those stickers you can place on the screws and then you can say, oh, but you opened it and in the end we kind of got an agreement and what we did, we just have this one banner but now if you buy our gateway, you can just log in via route and there's no password. So our customers are free to log in to their own gateway which is what we certainly think how it should be. Yeah, so quite a cool result from my perspective. The next point is about mainlining stuff. When we started the project in April 2018, around then we decided to go with the media tech mainly because of the financial, like the price, which was really attractive. And on the software side it looked a little bit less attractive, what media tech is offering is a U-boot which is based on the mainline one from 2005, a kernel 2.6 something and or the open WRT 3.10 which is also like already four or five years old. And upstream wise, there was not that much support. U-boot had nothing at all, Linux had some drivers but not really this SOC and open WRT had support like without of three patches. So our plan was to hire U-boot maintainer, like thanks, they implemented support in U-boot for our SOC which then allowed us to have like proper scripting capabilities plus this was quite relevant to access the spine end, not just the spine or, which then allowed us in series to put the Linux kernel on the spine end, which is much cheaper and we could reduce the spine or and save money. And this is what we pitched to management, we reduced the spine or from eight to one megabyte, we for that to work in implement the spine end support, luckily it was already really close to mainline at that point, we just had to write the spy drivers. And yeah, this cost way less than having this price difference times the gateways we intend to sell. In the end, we did not really change the spine or, but yeah, it's convinced management in the first place. So it worked really well from our point of view. And the Linux side we intended to basically just bring over whatever OpenWRT has and use it on our MediaTek BSP layer. So in November 2018, the state was like that, that we U-Boat support worked really well, our collaboration with Denx was really going well. We, it went so well that we told Denx to also update our Linux kernel to Linux 4.19, another LTS version, and then also to upstream stuff so we can reduce our burden in the future. And the upcoming dates were the manufacturing start, so we have like beginning of the year 2019, so we are ready for the gardening season 2019. Then this, from our perspective, was quite challenging time-wise, and when one thing was to get all the certifications, and this one especially is the one for the Wi-Fi line, which you need in order to get the home kit approval from Apple. This is a prerequisite, a requirement for that, and that this time it really helped us to be close to the main line. In this case, for the MediaTek Wi-Fi driver, the MT-76, we, or I sent a gateway like late November to Spain for certification, for Wi-Fi Alliance certification, and then one morning I was really happy that I got it out. One morning I got the first results, like five out of 42 tests have passed, the rest is not going like so well. A day later it was around 12, and they filled around a bit, but yeah it's quite far away from 42, and we really had to get this in time before end of year in order to get all the chips from Apple and so on, and continue with the production. So yeah, then we basically just called up the mainliner, Adam, sorry, the maintainer of the MT-76 driver on the phone, we agreed that he should help us, and we will sort out the details later, and this was on a Friday, and Monday he was already working, already had some fixes pushed, and yeah, 45 men hours later the driver was in the state which passed the Wi-Fi Alliance certification. Another example of how it helped us greatly to be close to mainline was the problem we had when we first started to produce some more gateways, 250s to be exact, 30 out of them did not work at all, failed our testing, turned out the spine ends had some bad blocks which is to be expected, but what we did not know was that for whatever reasons I didn't dig into it, the physical, the erase blocks, once it detected the bad one, it got moved to another one, and for some reasons it got moved back to the first again, and this basically ate up the whole CPU time. Luckily it was fixed in like a half a day which has upgraded to the already prepared 4.19 kernel, and I'm sure both of those not being close to mainline would have been a huge problem for our project and our timeline. Nowadays the current state is that the project went well so far, it's being sold now, we don't know about any bad problems or the like, we have the empty, the media tech support is upstreamed in Linux and Uboot, 5.4 has pretty much everything in it, 5.5 I think it's just like two or three patches left which we on purpose do not try to mainline, and same goes for empty 76, everything is mainlined, and I know of two other companies who passed the Wi-Fi line certification after us, and from one I know it was really easy for them to do it, just worked out of the box. And for us it means that we will port this new, this new architecture we have back to the old gateway and we will again upstream as much as possible, but I have to say there is the situation is much better for the different chip which has great upstream support. So for us the conclusion is that this money was, basically we paid an external company to the mainlining stuff, was spent really well, cost way less than 10% of the project budget and it saved us at least twice, and I'm quite sure we would have had big problems, not doing that. And we are also hoping that in the future it will help us greatly because we are just like two developers, we will move on to new projects and maintaining this should be much simpler when we can just upgrade to new kernel version easily. And what we also, what we had to learn, or what we learned is that, first we try to do our own BSP for the media tech, basically by taking over patches from OpenWRT and we had to learn that, yeah the real experts out there, they are much quicker and much better at doing that work. Yeah, so thank you for your attention, we have I think some time for questions.