 Welcome to our presentation about testing a wireless IoT product. My name is Reto Schneider, and this is my colleague Andreas Miller. First, we're going to start by giving you some background about our work and the context we did it. Then Andreas will tell you about the building blocks we used to test our system. After that, I will show you how we used PyTest to assemble all the building blocks into our test system. And then Andreas will wrap up with a brief conclusion. So short warning, we are by no means RFX birds. We are just regular embedded developers who got task to implement the SAP Gigahertz IoT stack. Sorry for the marketing slide. It's just one. But I think it's useful to help you understand why we are doing things and what we are doing. What you see here is the Gardena smart system. It consists of several garden devices. They are IoT enabled. They communicate with a gateway in the house of the customer. Communication is, as I said, based on SAP Gigahertz RF communication. And the protocol is proprietary and called Lemon Beat. This is the system we are on the market since 2016. And in the last few years, we put some effort into making it more modern and future proof. What we wanted to achieve is to be more independent from suppliers. We wanted to no longer be locked to a single MCU. And for this, obviously, we chose to move away from the bare metal system and to Zephyr. What else? We also, on an organizational level, put focus on making things ourselves instead of just buying from external companies. And not just making ourselves, we also improved the collaboration. For example, instead of buying from an external company, we got to work together on a shared code base. What we also had to keep in mind is that we need to be backward compatible. So despite basically redoing every part of the system, we did not want to betray our most loyal customers, the ones that are staying with us for many years. We didn't want them to force to buy a new gateway just because we released a second generation of our system. For this reason, we tapped one part, which is basically the RF communication channel, the Fire and Mac, the proprietary one we used before. We re-implemented it for Zephyr. It's still proprietary, but now we have total control over it. And we are not breaking compatibility. You can see some specific details about this protocol. The details don't matter that much, except maybe for the TX output power and the sensitivity. When future-proofing a system and rewriting basically everything, testing is quite an important aspect. Otherwise, I guess it will become instant legacy, instantly. And what we are telling you today about is just a device-based testing, like with the actual MCU and the Transceiver. There are many other aspects, but we're not going to have time for that. And the focus of our testing was to verify the features we released or we implemented. We wanted to make sure we're not breaking stuff we already implemented before, that's like avoiding regressions. And if we messed up, we also implemented additional tests to prevent regressions from the regression. And also quite important, we used this test system to evaluate the performance, maybe different settings for the Transceiver, see how well it works, how well the signal could make use of a certain signal. The main challenges we encountered were that tests really should be repeatable and reliable. Test system, that it's showing some red CI pipeline every day with another false error signaled and requiring people to investigate every day. It's degrades and it's not going to be useful at all. People will just start ignoring it and not really make use out of the system. And number one is the enemy of the repeatability, in our opinion, is the RF interference. And that's where I hand over to Andreas. Yes, indeed. I want to talk to you a bit about what we did to avoid this RF interference. But first, I want to actually go a bit into detail. What is RF interference? Or how does it manifest for us? And I guess when we started about like two years ago, like probably many of you, we were in home office. And in home office, this is what the spectrum around me, the RF spectrum in our band looked around me. And that's on a good day. What you see here is actually a screenshot of an application called GQRX. On the top half, you just see the spectrum, like you would see in a spectrum analyzer. You have on the x-axis the frequency from about 860 to 870 megahertz. And you have on the y-axis the amplitude. On the bottom half, you have what is called a waterfall plot. You also have the frequency on the x-axis. On the y-axis, you have time. And the amplitude is in the third dimension, is encoded as color. And here, it's pretty clean. But then towards the evening, it often looks more like this. And we see some things here. I don't know what they are, or maybe it's difficult to say, but I try to annotate. So the one on the left, that's just my neighbor. He's using a wireless headset. I don't know which neighbor exactly it is. I just know he's watching Italian TV a lot. And I also know this is kind of the channel two that we use, or it's very close to our channel. And I think that was the main reason why when I was developing often, I was working on some feature. And it worked quite nicely in the afternoon, but then towards the evening, I suddenly started getting more and more errors. And I didn't know why. And then I fired up this application. And I said, oh, OK. I guess he's kind of a nice neighbor. He doesn't produce noise, in audible noise. But for me, that would have been preferable, because now he's producing RF noise. So yeah. And I also have some other interference. This, I think, is like an unwanted emission, but I don't know what it is. And here on the right, we have some emission that might be like from a wireless weather station. So that's quite a lot of interference. And then at some point, we also moved back into the office. And the situation is a bit different. We don't have too many neighbors that need, at least not close by. But what we have are four different teams who are developing things, who are developing features. We have more than 20 gateways and more than 50 devices. And all these devices are sending things around the RV spectrum. And ideally, once the protocol is finished, it doesn't matter much. But when you're developing, you don't have all the safety mechanisms. And then it becomes quite difficult. And we could do things like, we could say, yeah, we just test in the night when it's not so busy. But that's kind of a half measure. And we thought pretty quickly, the real option is we just need to have a shielded test system. Now let me explain how we went about that. But actually, first, I got to give you a bit of background. I hope it's not too quick. But it's kind of difficult to explain. You might have heard about decibel also in the context of sound. It's also used for RF. The bell is actually a logarithmic value. It indicates a proportion. So it doesn't have a unit. The decibel is just 1 tenth of a bell. So for instance, if we say there's 120 decibel, it's just 10 to the power of 12. It's just a number. In contrast, the DBM is actually a power value. It's decibel with a reference of 1 milliwatt. So for instance, minus 20 DBM would be 10 microbots of power. And the nice thing about using or why people are using these decibel values is that the logarithm maps multiplications to additions. So if you have decibel values and you have a certain output power and some gains and some losses, you can just add up all those values and you get your resulting power. And that also brings us to the link budget. That's really oversimplified, but I think it should be enough for this presentation. When we talk about link budget, we're interested in what kind of gains and losses do we have, which kind of power and sensitivity, and then we know how much power can we lose along our paths and still receive a signal. And in our case, it's more about how much do we have to actually do shielding. And with the numbers we had on the slide, the data already showed you the 30 DBM output power and minus 104 DBM sensitivity, we get a path loss of about 117 dB or 100. So that means we need maybe around 120 dB of shielding. And as Rito also mentioned, we're not really expert. So we kind of looked at up a bit, OK, so for RF shielding, you need to have it fully enclosed. You don't have too high openings. And actually, the higher your frequency, the smaller your wavelength, and the smaller your largest opening needs to be. So that's kind of the relation. And we need a lot of shielding. So we need really a fully enclosed case, and we need also filter connectors. But still, we just kind of tried it. We just tried using this kind of aluminum box. It's not really made for shielding, although it does come with kind of an RF shield kit. And I'm not going to spend too much time here. I'm just going to tell you it doesn't work at all. It gives you like less than 20 dB of shielding. So that's, I could say it's 1-6 of what we need, less than 1-6, but if you think about it, it's a difference of 100. So it's more like 10 to the power of 10 less than what we need. And our second try was similar, just using these kind of extruded aluminum exposures for shielding single devices. This doesn't work at all either. But then we kind of thought, yeah, maybe let's try something more professional. And we found this commercial shield box. This is specified to have at least 60 dB of attenuation. And we measured for our frequency because we don't have such a high frequency. We measured better than 85 dB. And that's quite great, although it's still not enough for us. And these also come with filter connectors. So you can have like a USB connector to feed through USB connection without getting the RF interference. You have Ethernet, all these things. So we kind of bought some of those and thought, okay, how do we proceed from here? And then we thought, well, maybe our solution is two-state shielding. And that's what kind of actually got us where we needed to be. So we have the inner shielding, this box from the previous slide, under the outer side we have a shielded server rack. These are usually, or in many cases, sold to protect your data center from nuclear blasts or EMP pulses. In our case, we just want to be protected from our colleagues who are testing with their devices. But still, that was kind of the right solution for us. And I've drawn it on the right side. You can see from inside to outside, you either have the inner shielding plus the rack, or if we're talking about interference from one box to the other, you go out of one box and into the next. So we always get more than 140 dB of shielding, at least theoretically. And it also works quite well in practice. So that was our solution. But before I continue, I have a slide digression. Or actually, I have a picture first. So that's how it looks like, our test system now. You have the rack and the shield boxes. We have three by now, and it's growing quickly. And our takeaway here is it's best just pay for a commercial solution in this part. There are some cheap ways, but they're quite difficult. And yeah, I wouldn't recommend it. And the digression I mentioned is that after we already had the solution, I read somewhere, yeah, you can just use a microwave. And I wasn't really planning on doing that, but I kind of wanted to know if it works. And so, a microwave, it has to be built to shield at 2.4 GHz, otherwise it wouldn't be legal. And it should then also work for lower frequencies. And we have this really scientific test setup. We just have this Nordic SDK with our shield and our sub gigahertz transceiver on the left, with my smartphone to provide power. And on the right, we have a spectrum analyzer. And then we put it in the microwave. Don't worry, we didn't turn it on, we just wanted the shielding. And you can see in comparison, the peak is kind of lower, but if you look closely, it's only about 30 dB of attenuation. And luckily, I also tested it at 2.4 then, and there it's better, it's about 50. And I'm quite happy about that, otherwise I don't think I would still be standing in front of it when I'm cooking food. But still, I wouldn't use it for actually, like, shielding purposes. Yeah, so that was about shielding, but now we have a second problem. We just put all our devices in this box, they have antennas, they kind of send a really strong signal, the signal will be reflected inside the box, you have kind of weird RF pads. And you'll get like, if you move the device even a little bit, it will change the whole situation. You get weird reflections, I don't know what happens. It's just quite inconsistent, inconsistent. And one kind of partial solution would be to use RF absorption foam, but in my opinion, that's just a really small part of the solution. And the better solution is rather than using antennas because you don't want to send out things. Anyway, you use cables, you connect the devices via cables. And I don't wanna spend too much time on that, I guess we kind of wanted to tell you about a lot of things and we have more things in the slides than we can tell you about right now. So I have a lot of things that I just have in there. For reference, if you wanna look at it later, also the slides we uploaded are a bit longer than what we have here. But the takeaway here is, if you don't have the medium, the air and the antennas, you need a replacement for that. And that is a power splitter. And most power splitters you see online are asymmetrical, we actually need the symmetrical type here. They are available, for instance, from this company called Mini Circuits. I quite like this company, they sell these small devices. These are just like RF building blocks. It's like Lego bricks for RF, is how I think of them. You could have an amplifier, you could have a filter and you also get these power splitters. Unfortunately, it's not that common to find power splitters with many ports. In our case, we need like at least 10. And they also get quite expensive. So we did find some with 12, but it was like $1,000. And we bought one, but we also wanted test things at home and our colleagues wanted to test things. So we didn't wanna like buy 10 of these. And instead we thought, yeah, why not just do it yourself? Such a power splitter is really simply on the inside. It's just a few SMD resistors. And then we just started, yeah, let's just design that, let's build it. And then you also have the advantage you can add on other components. Here we also have the attenuator. And that's quite simple. And we have like these prototype manufacturing services. You can just manufacture like five of them for $100. So it's like 50 times cheaper than if you just buy it from the commercial vendor. And it works for our purposes pretty much just as well. Then if you connect your devices, you'll also need some attenuation. Otherwise you would either damage your devices or if you just connect them directly or you would overload them. And then you don't get a good signal because it's too strong. So it can be both too weak and too strong. And ideally we wanna have about 40 to 80 dBs of attenuation. If you use more than 80 dBs, it actually doesn't make any difference. Your signal will just couple directly from your T export via the air to the R export. And then you have the situation you think you have a test system and you can unscrew your connection, take out the cable, it just makes no difference. You get a signal anyways and you can transmit your packets even without the cables. And what's nice about these attenuators is you also have programmable attenuators. And with these you can actually do things like test the sensitivity by just increasing your attenuation until it's kind of at the limit where you can no longer get a signal. And that's kind of like not the same but similar to doing a range test by just walking away with one device and increasing the distance. What you need to be aware of, of course, is if you wanna do that, you need to have your two devices in separate shield box or otherwise you get this direct coupling I mentioned. Yeah, and that's how it kind of looks inside or that's an early version of our test system. You have in the middle this power splitter, the attenuators, we have on the right, we have like here, we have six devices with our transceiver as a daughter board. And we can kind of communicate at the same time to do test different things. We also have some measurement tools like power measurement and STR hardware. And here again, that's just for reference, the block diagram, unfortunately we don't have time to go into that too much. Yeah, so that's the shielding and the connection. And now I wanna go into one topic that it's just one particular aspect, the STR and the GNU radio, but it's one thing that I kind of feel like it's from my perspective a really great tools and really helpful when you're doing RF work and it will be kind of nice if more people maybe knew about this. So what is software defined radio? That's kind of a complex topic. There are actually conferences on this topic alone and I'm just boiling it down to one slide, so sorry if it looks a bit overwhelming. But the basic idea is just rather than having hardware that does your signal processing, you just build as much as possible in software. And in an ideal world you would just have an antenna, an analog to digital converter and then process all the samples in the PC. And if you wanna transmit, it's just the reverse of that. Unfortunately in reality our analog to digital converters are kind of limited, limited sample rates, limited bit depth and for that reason you need analog front ends and you need specific hardware. But the good news is you don't really need to know about all that, you can just buy this hardware and work with the signals. And here I have two examples. On the left is kind of the low cost option that you can use just for receive. That's really cheap and you can, like if you just wanna try it out that's great for doing some basic experiments. And on the right is the hardware we actually use in our test system. There you have two TX and two RX channels. You can also do full duplex, transmission and reception and we really like this hardware and it's kind of not too expensive for what it does. And what do we actually use it for? We use it for several purposes. The screenshot I showed you in the beginning was done using this hardware from the last slide. You can check is there any interference and is your neighbor actually watching TV again or is it really my code that sucks? And you can also generate interference because of course now we have the test system but we can't expect our customers to have such acquired condition. So we need to, but we need to do it in a deterministic way. We need to reintroduce this RF interference. And that's what this hardware is also great at. You can reintroduce all kinds of interference. And then what we also did and if you have a less proprietary protocol that's even easier you can just go online and for most things like, I don't know like FSK like Wi-Fi you will find an implementation of that for SDR and then you can actually receive and send packets. We had to implement our protocol ourselves. We did the physical layer in GNU radio which I'm going to explain in a bit and the upper layers in SCAPI. And with that we can not just send actual correct packets but also we can send wrong packets. We can do kind of fast testing. What you can also do but that's kind of where the limits of these devices are you can also do basic qualitative measurements but you have to be where they're not, these devices are not built for that. In that case, using a spectrum analyzer would be much more appropriate. And the second part you need if you want to use this kind of hardware is software and my preferred software toolkit for this is GNU radio. GNU radio is a toolkit that allows you to implement RF processing blocks and also to wire them together either using Python or using a graphical tool into signal processing applications. And these signal processing blocks for me they are kind of the software equivalent to these mini circuits blocks I showed you before. And you can use that to just quickly build such flow graphs and I'm just gonna show an example that I think explains it best. So we have again this screenshot of the RF interference and now we wanna for instance model this wireless headset. And we could do that using a signal generator that's not a bad option because it gives you a precise power level but it's also kind of big and not too flexible. You could use an actual headset but that's also kind of yeah, I don't wanna add an actual headset to my test system. So in my opinion, the best option is just to use GNU radio and SDR hardware and then you have this very simple flow graph. You just have this audio file with some music or some radio show. You use a frequency modulation and you just pipe the samples to your hardware. So it's really as simple as that. And from this code, from this graphical file, you can also generate Python code and integrate that into your test system because we actually use Python and I guess that's your turn again. Now that Andreas has showed you our building blocks, I'm gonna talk a bit about how we put things together in our test system. We used PyTest, I'm not gonna advertise it right now because pretty sure most of you use Python and you do, you need testings and chances are that you know it already. But I want to highlight a few things. One is the fixtures. The fixtures allow to set up context for a test in a really brief way. Like the whole complexity of setting up for a test does not go into the test itself. The only thing needs to be done is to declare that the test needs a certain fixture and all the complexity is in the fixture itself. It can be reused and it's kind of really flexible. Like it allows a certain fixture to be initialized once per session or per function, per module and PyTest takes over like or manages and sorts the testing in a way that is optimized. We have a small example here. Let's say we have a fixed shock hold power supply which doesn't do a lot more than initialize your power supply and yields, well, let's say returns it to your testing code where the testing code can do whatever it wants with it. This makes sense to do it in a session scope so it gets done only once per test run. And on the bottom you have a second fixture. It actually reuses the fixture from above but it also enables the output power sets a defined level and once the testing code no longer needs the fixture it disables the power and it does do this for every function in our case for every test. So every test that uses this fixture gets a well-defined state of the power supply. And yeah, we found this really, really helpful and we have built fixtures on top of fixtures and it's really a breeze to work with it. Then to actually integrate PyTest or integrate PyTest with Sephir, we decided to go for the awesome shell and we wrote a bit of code to be able to access the shell of Sephir, execute the shell modules we need to inspect certain aspects of the product. There are so many shell modules available like for the net subsystem. And wherever we felt like we need to expose or test something that isn't yet there, we just wrote our own shell module and the counterpart in the PyTest fixtures. What also was really easy to do and helpful to simply parse the K-config files of the Sephir binary and this allows us to skip certain tests or ensure that certain features are enabled. So we don't test for things that will never ever work. There are some other very useful Sephir things and for us it was the possibility to stop the RF link by using internet. Over USB it allowed other teams to advance or work on their features before we were up and running with our RF stack. Same goes for MCU manager. It's another area very useful. We found it to be extremely helpful for photo testing and also exposing the shell over UDP is quite helpful during manufacturing like after you no longer have access to the testing pins and shell access or whatever communication is not really possible, we can still do it using UDP. And now I'm moving on to another example. This is how we tested the sensitivity of our transceiver and some pitfalls we encountered. To figure out the sensitivity, we decided to increase the intonation step by step and see at which moment we no longer are able to receive the package we sent. And that allows us to calculate the sensitivity or verified sensitivity. And the very first approach, naive approach was to just do it in a loop and see once it no longer works. First result we got looked like this. It seems like at 52 dB no package can be received anymore and sort of implies that our sensitivity is minus 99 dBm. However, it's quite misleading because it was not actually the sensitivity that was limiting here, it was the data that basically just crashed at this point and gave us quite misleading results. Good thing about PyTest, it has a really nice feature called parameterization. It allows to invoke a test function multiple times and every time the test setup gets done by PyTest using the fixtures. So yeah, with this feature it's very easy to always get a well-defined state. However, there are still variables remaining that are not controlled and for this there is also something we found to be extremely helpful to just let the tests run in a random order every time. So the not controlled variables and the spurious correlation of those, yeah, they just cancel out. It doesn't give systematically misleading results. So that's pretty much at the end of our talk already. Our takeaways, it really helped to have a shielded setup and a cable-connected setup. It's just really improved our reliability and our repeatability. We found PyTest to be a great fit. I think on the homepage of PyTest they advertise it as it's great for testing software but we also thought it's really great for test for hardware-based testing. And also SDR and Knur radio just, if you have any RF work it just helps a lot. And of course also we're really happy with Sapphire. So that's also our contact information and if you have any questions I think we should have a few minutes. Yeah, I think, can you go to the microphone I guess? If you're using cables internally, do you still need the greater than 110 dbm shielding externally? Like you don't have antennas internally? It might be kind of enough to have one level but what for sure isn't enough is to just have the cable because that's the setup that I have in my home office. I don't have the shield boxes at home but I have the cables and that alone is not enough. With one shield box it might be enough but what you have to keep in mind is we don't just want to have the case where the stronger signal wins. We also have features like listen before talk where the transceiver checks is there any signal on air and if there's a signal it will not send. So even if it's just a really weak signal it might decide not to send and it's not the case where the stronger signal wins. So for that it kind of helps. It might be a bit overkill but we suffered enough to kind of want to do that, yeah. Or I can also repeat it. Okay so the question was like about the difference between the setup, the really shielded and nice setup we now have for the test system and the reality as it looks for our customers and how we deal with this difference. And there's actually several parts for this question, several answers. One part we didn't mention is we're kind of just the low level engineers doing the low level tests and we also have other teams dedicated to doing system tests and they do it in a more real worst scenario and they are more than interested in the whole system work. They're testing from device to smartphone. So that's one part to make sure it actually works in a real worst scenario. And for us it's more about does an individual feature work with specific interference and there we just make sure to introduce this kind of interference in the test system. We're never gonna be able to cover an actual real-world case and that's why we need all these different types of testing or actually we have three different teams doing because we then also have the team developing the actual application. They test more on application level and we have the final test team that's an independent team doing the whole system test. So that's kind of these two parts that we do. So the question is if I take the test from the box and then random outside, how many will fail? And the answer is it depends and of course because I'm developing, we still have a lot of home office and when I'm developing at home, I don't have the shield box and it's again, it depends if it's like the first screenshot how my environment looks then pretty much all of them will work but then towards the evening maybe one third of them will just fail. So that's like really impressive to see the difference between okay, all my neighbors are at work and now they're back home and using the spectrum. Yeah. What I see from testing perspective that you are trying to solve two things with one testing setup, namely your channel capacity. Yeah. What do you call it? The channel. The performance, yeah. As well as the functionality of the software stack. Why didn't you choose to emulate the whole channel by new radio without an RF path? That's a fair question and I actually did that in some earlier projects and that's another thing I really love about using new radio. You can use the same framework to test poorly virtually just with a virtual channel model. You can model multi path fading. You can model a noise. You can model everything and that's really nice but here it's also a lot about testing our specific transceiver. It's about this specific hardware and that's why I think it wouldn't be too helpful here and also what we don't have yet is kind of the actual interface from Sapphire to GNU radio. If we talk about sending packets from GNU radio we do that from Python. I guess we could somehow use the Sapphire native build and that's actually something I thought about and I kind of want to do but I just haven't gotten around to it but like having a Sapphire native binary and hook up the network stack to our GNU radio interface. That would be a really interesting challenge. Well, someone needs to be the guy who asks what is the cheap or ugly solution for shielding? Yeah, so we had one external supporter and he's the actual expert with RF and what he did is it just took several PCBs and he soldered them together and he also has a filter connector but then he told me, oh, I forgot that I need this comment and what I have to do now is I have 24 screws on top to make sure it really fits tightly everywhere and I have to unscrew those 24 screws actually put the device in the right state and well, re-screw those screws to make sure it's really well connected. So that's really cheap but you're not going to have too much fun. Or another option might be because we have the filters here you could use those filters and use kind of cheaper metal boxes. That's something our colleagues from another team are doing that's kind of a middle ground. Are you able to give a ballpark figure on what those enclosures sort of professionally made and shielded cost? Yeah, so the inner shield boxes it depends on what kind of filters you have but it will be like one and a half, two or three and a half thousand euros, something like that and the outer, the shielded rack there we got several quotes from several companies and it was like most of them were like eight to 10,000 euros. The outer shell though, when you had a photo where you had like a server rack it's a refrigerator basically. Well, it's a really expensive and really heavy shield. That's the outer shield, the refrigerator thing. Okay, that's my wife will be very unhappy. Yeah, and I can tell you it's about 300 kilos so it's not too easy to get it into your office or living room. Just one addition to previous question. Is this 35 micrometers copper layer sufficient for your demanded attenuation? Sorry, I didn't quite acoustically understand it. You told that your colleague make attenuation box from PCB board and if it is sufficient. Oh, good question. I don't know how much shielding he actually got out of it but I think the way he did it, it might be because it's really well connected and he did also very similar tests to what we did, this kind of sensitivity test. He did a bit of a variant where I also added interference but I think it should be. Yeah, if it's done very well, if it's sold well, if you don't have any opening it should be possible. Yeah, and I guess we're pretty much through with our time also. So thanks for attending and if you have any more questions you can find us outside. We're happy to talk about it more.