 So let's start. So I'm going to talk about Osmojizen tester or how are we Testing end-to-end all the Osmo Com stack or at least part of it for now Yeah, let's I need to read though. So it's fine, but then I'm on the middle, right? Okay Good so this project was started as far as I know by Nils and then it was handed over to me I'm mostly the one maintaining it now or adding new features So what's the idea behind it as I said we want to well it's a bit problematic to test all these networks, right? So we're after the conclusion also yesterday Osmo Com stack is becoming quite complex There's a lot of parts to test hardware software different setups So, yeah, I mean we are still improving. We're already improving the testing We are using to DCN3 now. We have some unit tests and yeah, that's great, but still No, not all the world is running Osmo Com yet. So we still need to interoperate with all the people So that's one of the main reasons also for using Osmojizen tester so we can interoperate with other Producers let's say or manufacturers whatever so we can use modems real modems This kind of stuff Yep, so yeah, as I said We care about interoperability here also another Good point. We want actually to have continuous integration with all these stuff it's fine testing from time to time but It's not that fine that you find there's a bug after two years of adding it So it's I think it's nice that we are also moving towards these continuous integration platform Let's say or way of working It's also fine that we can repel use these bugs So, I mean, I'm just listing the objectives we had in mind when we started this this project, let's say So we wanted yet to be able to repel use all these all these bugs because it's sometimes a bit difficult to Fix bugs when you actually see them in production already and you only have some traces But not all the information. So it's all still good to have as much information as possible That's what One of the why one of the objectives also it's to yeah store as much information as possible during the test run and be able to Yeah, find the issues As we care about interoperability, so we want to test with different hardware different software and For that reason as we have to support all different types of software or hardware We want to have test which we can reuse as much as possible without actually caring about What's underneath or less? Yeah, not as much as possible. So So I'm moving now towards a bit on the architecture or how it usually looks like when you are using osmosis and tester, so it all starts with Jenkins or some developer basically Running osmosis and tester. So running some tests So most of the work actually happens in when we call what we call the osmosis and tester main unit Which is just a machine running osmosis and tester, which by the way is written in Python and Then you need to pass some input to osmosis and tester to enable to at some point, of course Output a report which we currently report as a J unit XML Which kind of stuff Do you do you pass to osmosis and tester? So you pass a trial. What's a trial? so basically it's Set of let's say compiled programs or rootFS. So basically if we want to test I don't know Osmo VSC We have some scripts which are Jenkins jobs Which basically take Osmo VSC code and all the dependencies from Osmo VSC like lip Osmo core lip blah, blah, blah and It basically builds everything and generates a let's say. Yeah target set with or an archive with All required dependencies and binaries Required to run Osmo VSC and we do that for each Osmo com project or Yeah process we want to run in the test. So Osmo VSC Osmo theory Osmo VCU, whatever And of course we do that for different architectures because for instance If we want to test a sysmo BTS, of course, we need Osmo BTS sysmo compiled for sysmo BTS platform instead of Anx86 platform being used for the main unit So we end up having like yeah 15 Jenkins jobs building this this artifacts, let's say What more do we have so Of course when we run some tests we want to define which test do we want to run and As we said we have we're supporting different hardware, right? So like sysmo BTS Osmo BTS theories with running with I don't know It was with 200 sysmo cell 5000 whatever so we need to select which hardware do we want to run for the test because actually we want to test Same as scenarios for different tests, right? so What we pass the test with plus a scenario so it's actually two different things the test which is actually The test you want to run for instance. I want to send an SMS from one MS to the other and then We pass an s scenario which basically defines which kind or Or details which kind of hardware or resources I want to use for that. So for instance, I could say I want to run this test which sends an SMS But then I want to run it using a sysmo BTS or I want to run it using a to speak to handle for instance And of course That information is used to Handle or handle all these hardware resources which are attached to the main unit again all these Different BTS is also different modems That we have connected to this main unit by means different means like ethernet or whatever and Of course this hardware cannot be used by several tests at the same time so we have in the in osmosis and tester We have a resource pool which actually Checks that we are not reusing the same hardware for different tests in parallel or by several users at the same time So basically when you run a test and you say I want A sysmo BTS osmosis and tester will look at resources pool, which is basically A description of the hardware and Check which one is in use which is not and then it will allocate it temporarily for it and prevent others from using it So more On architecture point of view more like not in pipeline but the architecture point of view. How does it look like so? You are usually interested into Like writing a test we will see an example later on how this test looked like but I mean of course As it's in python you can imagine just a simple python script which Basically orchestrates all these all these resources As we said we are defining which kind of hardware do we want to use for this test And this is presented to the test by osmosis and tester As a set of different objects that the test can request Generic objects like a bts or a bsc or a modem and Each of these objects has like As generic as possible Interfaces or apis that can be used for instance modem As an api to register to a network That's an api to send an sms an api to call some ms An api to wait for receiving call This kind of stuff for the bts similar stuff There's a There's an api to start the bts to connect it see if it's connected or not, whatever So and of course each of these generic Apis used by tests are implemented using different Classes or implementations of this interface. So for each bts, we have a different python class actually handling this. So for instance You would have a sysmo bts class that actually what it does. I mean some of these implementations are a bit Hucky still But it would for instance connect through ssh to a sysmo bts Copy all this Trial that we were talking about. So basically all this osmo bts sysmo binary with all the libraries It would copy it through ssh to a sysmo bts that it's connected to the osmo gsm tester Main unit and it would start the sysmo bts there configure it We will see also how do we configure everything to work together later And uh, yeah, well it could it could manage it the same for the bc for the bsc We also support like a simple esme Which is written using python smpplib. So we can test for instance If we want to send a An sms from one modem to the esme If it can read it correctly or not And yeah, well as you can see basically it talks to different hardware In the case of for the modems we are so far using of ono To handle the modems Yeah, we may Later and also maybe add support for osmo.com bb or i don't know like batches welcome of course So that's how the hardware setup looked like basically, uh, it's not like the latest photo, but I had it there so Just to give you an idea on how it looked like so, uh You see the modems in there so the one without the box, let's say The main unit is the one on the on the bottom and you see how all the rf network is connected through a wiring and In at this point, there's only like 2bts that they are labeled there Just to give you more like of Real world hardware hardware perspective So, um We will go through different, uh Objects I told before I presented before and just give you some insight on how How it's everything configured, right? Um So, uh, as we said we have a set of resources And of course, we have to define them so that osmosis m tester knows, um, how to use them um This is a sample resources file. It's not the entire resources file we are using in the production setup, which again is the one I showed you here Um But it's some of the objects there. So basically we have ip addresses Uh, because we have several ip addresses set on the osmosis m tester main unit and For simplicity what we do when we run some process. We just uh, yeah, we need some ip address to bind Sockets or ports whatever to it. So, um Yeah, we need some some way to, uh, allocate these ip addresses So we don't reuse them for several tests at the same time So they don't collide All the types of objects. Well, as we said, we have bts. You can have like, yeah, you can see an example here The first is a sysmo bts. We're defining. We have a sysmo bts Uh, and we provide some information to configure it like I want to use this ip a unit id The sysmo bts is actually connected to this ip address um It's using this band or I want to use this band Uh, I want to configure it to use a direct pcu Uh, and basically I can state, uh, I state here this bts actually supports this kind of Cyphers Because then we can match. So the idea here is that actually Um, we may want to match, uh, against scenarios. So for instance, if I want to test, um If I 3 in a test, I of course need a bts Which of course, uh, supports it because otherwise it makes no sense So what I would do to test this I would create a test But then of course, I want to run this test with a scenario that explicitly say I want a bts which supports Uh, this cypher um So that's why it's there because then when it tries osmosis and tester tries to find a bts that matches the scenario It will look at this information and then it will find a match and it will take the bts to use for the test Um One more, uh, yeah ARFCNs are actually there, but they are not, uh Currently being used correctly as a resource. Um, I will talk about it later. Uh, it's one of the two things Uh, and then we have different modems. Um So you can see basically we identify the modems by, uh, uh, a path which is provided, uh, since not that long ago by your phono um We then provide the ki, uh for the sim card that is, uh, uh, yeah set up on the on the modem so we can basically and dynamically, um, register these, uh, imsies into the hlr when we do tests And again, we provide also some features that the phone supports Again, like you can see which cypher supports, um And which features because for instance if one of yeah, we had a lot of that I'm going to talk about that later too, but we found a lot of issues when using Ophono Uh with modems like different modem support some stuff, uh, or they don't support it correctly so We have to also be careful when we want to use some Ophono feature or some modem feature to actually select the good modem because If I want to test to send an sms from one modem to another But one of the modems, uh, doesn't support, uh, correctly sending an sms or receiving an sms Of course, uh, I have to select the correct modem which supports it Yeah Yeah, I'll I'll I'll I'll go I'll go for that at the end. But yeah, it's one of the conclusions. Basically we spend a lot of time, uh, yeah Trying to work around the modem issues um And uh, sometimes I have the yeah, I have the feeling they should be less smart. I think they try to be too smart and, uh Yeah Okay, so, uh, let's move on. Um Next thing, um, we support configuration templates and, uh, that's actually what we used to configure the, uh, the different, uh, processes or, um, resources that we are using so, uh As we said, we are allocating ip's, uh dynamically for instance Um, so that means that of course we have to set up all the vty configurations to match each other so they can, uh, connect each time, uh, so, um Yeah, we have, uh Templates, uh, for each of them and then we fill them up as, uh, as we see so some stuff from here Actually, like the ip a unit id if you if we go back actually you can find it there So it's already configured here. Uh, some of this this stuff is, uh, uh, dynamically, uh configured, let's say Or like can be added by the test. So for instance, if you are testing an is me, um You can add an is me on the test, uh on on the on the msc on the smsc And uh, it will it will end up there. So we can generate, uh, yeah templates on on On time, let's say, um So here's an example of a suite and uh, and a scenario Uh on the left you have the uh The suite so in here it's uh The a over a p sms suite which basically only contains so far one sms. Uh, sorry one test and the test again It's to send one sms from one modem to another modem And then we check that it was received correctly and we check that the content is fine this kind of stuff Uh, what are we defining here? So basically we are stating Look for this, uh, suite. So this set of tests, uh, I require this this set of resources, right? So I will need a five ip addresses Because I need to run as you can see an msc, a bsc, hlar, stpmjw and I want to assign an ip for each of them Uh Then, uh, I will need a bts of course, uh, because I need to connect the modem somewhere Um, and I will need two modems, of course. Um, as you can see, we are not defining bscs or mscs or whatever because so far we are uh We are I mean, we don't need to, um Lock them because they are unlimited resources in the sense that we don't need hardware for it So I mean, we don't really need to uh, we don't have a limited amount of bscs that we can run so far So we are not handling them as resources. It may change in the future Um And then, uh, on the right we can see two different scenarios as I was saying so like, uh, the first scenario It's saying look, uh, when you have when you have to handle a bts It has to be of type osmo bts sismo and the other one the scenario turex b200 it's saying basically almost the same It's saying, uh, it has to be of type osmo bts turex And I want to take the one that the label is ethos b200 and then if actually we go back Uh, to the resources definition, we will see that, uh, of course you will match those for the first, uh For the first scenario the first bts would match for the second the second would match And The the blue letters in there basically show you Um, the blue letters in there show you how How it would work when you run osmo gsm tester So basically here you you're running osmo gsm tester minus s means a suit and you can pass Several suits to it. So you can ask it to run several suits in the first case You would be saying, hey, I want to run it, uh, with the sismo scenario So that would run the a over psms with the sismo bts And then once you're finished with that then run it with the turex b200 So that's a that's a sample test. It's again, uh, we are taking the sms some, uh, test So one ms sending, uh An sms to another one Uh, we can see three major blocks. So the first block it's basically Requesting, uh, the resources or the different, uh Components that we want to use for the test So we are asking the the suit. Please provide me with an hlr. Please provide me with a bts And the suit we would have yeah has had a Allocated those from Looking at the resources and the scenarios. So it will take the the required ones Um Second block basically we start everything So in the case of the hlr, uh, the implementation would just run the osmo chelar process Uh, for most of them it's only starting a process So you can see that, uh, then we are telling the bsc to, uh, add this bts So this is how we dynamically basically create all these structures and then we generate, uh, the vty configurations from the templates we saw before and On the right side, we can see already like the test doing really something So, uh, first of all, uh, we are the test is waiting for the bsc to state that the bts is connected um Then we are adding the subscribers from the two ms From uh to the hlr Remember we had the ki on the configuration so we can do all this kind of stuff um Then we are telling the modems to connect to the msc we created or to the core network um So we will basically telling Ophono, please tell the modem to connect And um Yeah, then we Wait until the modem state that they are connected And then again, we actually wait for the meas for the msc to state that the ms are connected So basically we are cross checking here that both sides state the same like the ms says i'm connected and uh The other ones also because as we said we are having some issues with Ophono So it's also nice to sometimes see if actually it's our fault or if it's Ophono or this kind of stuff um So and then once finally we have the two ms registered to our bts plus core network um Yeah, we basically tell one ms to send uh an sms to the other one and then Uh, the test waits for the second ms the empty ms to receive the sms And actually this sms was received as you can see there's the sms that was sent was passed as a perimeter So it will basically also cross check that the sms received was the same that was sent Or it has the same content um So now we're looking at some other features that we had we have in osmosis m tester Um, so as I said, we are outputting uh j unit xml Um, so this is an example of a test failure in osmosis m tester So I don't I don't know if you can read it from there But in this case for instance, uh a test failed because um A process a process that we are uh managing like an osmo vsc for instance, uh and prematurely So that means for instance Like yeah nowadays we have more more output like we We output which which of the process we was ended so this yeah, this photo is a bit uh out of date But uh anyway, so it provides us like osmo vsc Ended that prematurely that means it finished before we actually told it to stop the process And that of course usually means uh, there was an error. Uh, there was a sec fault Whatever so basically what you could do in this case is uh Of course, we are logging all the output from all the uh From all the processes we are storing the core dumps, uh, also so it's fairly easy if uh, uh One of the processes crash or there's an output failure You can go there and debug it quite easily. Of course, we are also Storing pickup files for all the processes so It's quite a lot of information so What's the currently status the current status it's uh We are running several tests. Uh, as we said, uh, mostly hourly Um, this is the kind of test that we are running So we test a network registration We test different encryptions Um, we are testing that we can send or receive sms Um, we also involve smpp in the process sometimes. Um We test the ussd code to receive your phone extension But I mean that's good. That means we are testing all ussd, right? Yeah, it's fine Um, so we are testing gprs. Um only the signaling though. So we are not testing real So data plane. Um, so we are testing that actually the modem can attach And that the pdp context can be established But uh, yeah for the day we we are still not testing it because we have to set up all the routing stuff So basically we have the this issue in which, uh Yeah, the routing is a bit difficult because then you have to send your packets through the Interface created by the modem, but then at some point you will receive it On the ggsn, which will try to route the same packet again somewhere and of course it will route it Yeah, it will drop it. So we have to create different namespaces or Uh, yeah, I have to come out with a good solution I have some ideas But I didn't implement it yet because again for to do this kind of stuff usually need a route access Uh, so it's a bit more difficult to handle because I I need to escalate powers and yeah, right and do this kind of stuff. So It's still thinking about the best solution um For voice calls actually Uh, also we support signaling no no data yet. Uh, so we can actually establish a call Uh, keep it for five seconds and then close it and check that everything's fine um I think the reason for actually not supporting data here is because uh, we are also having some issues with um Yeah modems handling data. Uh, so maybe harold knows a bit better the situation. I think he did some research Well, the problem is that you can hardly find any, um, usb or um Uh mini pcie modem that does voice to begin with So it's very very few modems that you can find and if you can find them then sometimes it's an experimental firmware It's not officially supported and it's broken in many ways and if you find it at all then the audio is uh Never exported as usb audio. Uh, some do some proprietary gsm voice frame over urt handling Which you then need to implement and the only interface that you can find is PCM bus and that's actually why we are now building uh hardware That can interface a pc m multiple not synchronous PCM slaves because each modem wants to be a pc m master So if you have multiple modems you need basically n number of pc m slaves that can get the the pc m audio And then feed that back to a pc and we're building specific hardware for that because it's not possible any other way Okay, so there we go. Um, then which hardware are we currently testing? Um, so we have a sismo bts Uh, I think it's yeah model 1002 for the etus. Uh, I think it's the model b 210 And then we are testing a sismo cell 5k um We are testing a 2 nano bts. So the difference between those is basically the band on which they operate um And then we are testing with an octo bts 500 So and for the modems, uh, we actually have uh three modems there. Um, three different types of modems Uh, we have this sierra wireless mc, uh, whatever Modem which is actually the one we are using uh for the test because um, so right now, of course, you have a list of modems and then um We only use up to two modems in one test and only so we put the this sierra wireless the first two in the list So they are always picked up and we did that on board posts because the other Two modems like the quick telly c20 and the goby 2000. Um, we are having some issues or they are not supported supporting all the Features that we would like like some of them have some issues are receiving some sms Some of them with voice calls. Yeah, different issues. So for some of them, we have some issues open So if actually if you go, I will show you later. Um On the osmosis and tester red mine, there's a few tickets related to ophono So if also somebody wants to contribute to ophono, it's uh, It's fine for us. Um so We talk about the current state. Let's uh, just list a few Things about what are we going to or planning for the future or could be nice for the future? um Of course adding more bts hardware. Um So, uh, we are planning to add lime sdr. Uh, so probably an open cellular at some point Would be nice. Um Then Again, uh, better ms support. So that means uh fixing some stuff in ophono um So for for instance, uh, just to give you some examples when you now create a pdp context You can have a pdp context before v6 or v46 and You can state that on the ophono api, but actually on the implementation side They are not correctly doing or sending the qmi commands correctly to To the modem. So basically even if you state i want a v4 Uh context or a v6 context, uh, you will see that actually it asks for a v4 v6 context or for a So it actually tries several ways and uh, uh, that's That's okay if you are running, uh, your phone, but of course if you want to test some specific feature, uh, it's not that nice um Yeah, some other idea regarding mess would be again to add, uh, an osmocon bb related phone like a Motorola one so we can uh Yeah, we can support it on the test. So basically would be Taking the modem api and implementing it using uh, yeah osmocon or whatever um Uh, so some next features, uh Yeah, adding data plane for the voice calls Adding data plane for the gprs, uh or back switch journal, um Adding a 3g testing so Uh, I think there's nothing really stopping Me from adding it now just basically lack of time Uh, but I think we have even another 3g there already set up on the on the osmocism tester main unit. So should be Hopefully straightforward to add it there um And then what we are missing to it's uh An airfcn resource allocation algorithm. So that's something it's been pending for a while and uh, basically because I have to invest a bit of time on it um So we have this issue in which actually The idea is that if we we run several tests in parallel, of course, we don't want to uh Different bts is to use the same airfcn because of course they could collide between them. So we have to Um, yeah allocate different ones Uh, but that turned to be a bit more complex than than I first though Because you have to take like different variables into account. So there's a red mine issue about that. Um But yeah, so You need actually to write some kind of algorithm to really pick Which airfcn you should be using or not or whatever. I mean, I don't remember all the cases I found but uh, yeah It it's in the description and yeah, it it has prevented me from from doing that now. So that means Uh, I think right now we are almost always using the same airfcn Uh, so it's set up quite a hard coded more or less or in the configuration Uh, which of course that means that we are not running tests in parallel. So so far We are really only running one test after the other And then that means that uh, the full test should Take like 45 minutes nowadays to run. Um, it's 45 tests more or less, I think But yeah, I mean you have to think that each test we run that means uh, you have to start The sysmo with the well don't start sysmo with the s but osmo with the sysmo You have to copy stuff to osmo with the sysmo. You have to power on the modems You have to wait for the modems to scan and and find the network You have to wait for the modem to connect to the network. Um all this kind of stuff Actually, so regarding the modems also we had really issues with that because We saw that modems were stating that they were connected to the network when they were powered up But they were not connected to the network. But uh, so that's what the smartness I was talking about So sometimes I wish that they were not so smart because uh, then of course if you are testing this kind of stuff And then the modem says it's connected, but your vts says it's not connected And of course you check the the pick up traces and you see that it's really not connected But the modem states it's connected then it's you have an issue there So, uh, I think we fixed that basically by incrementing the location area code on each test we run So every time we power on the the modem we increment the lack and then since then I think the issue is fixed um Yeah So some final remarks about it Uh, so it's proven to be useful. It really helps. We found uh, like some interesting regressions there uh, for instance, uh we A few days ago, we actually enabled address sanitizer and we catch like I had to fix four or five bucks in One day, um Sometimes it's nice because you find these Bucks, which you usually don't find if you don't have a specific configuration So for instance, somebody changes something in the implementation of lip osmo core But for god's that or forget that I don't know osmo bsc is using this Dysfunction, but then it's taking into consideration some implementation details And then of course suddenly it doesn't work. But yeah, this kind of stuff that sometimes happens um Or somebody removes vty configuration from the vty Code and then of course, uh, you have to remove that from, uh, I don't know the configuration. So it's it's good to check this kind of stuff um And the good thing is that also as it's real hardware, we don't have control over it It fails sometimes in unexpected ways and that's good because then it lets us test some paths That we don't usually control or when we are like manually developing or testing We don't usually take them into account like for instance, uh, I think like nil's Fixed an fsm bug which happened when some of the phone calls sometimes was not handled correctly or something like that or was released or um So it really helps with this kind of stuff um as I said, uh Controlling the modems is really hard. It took A big amount of hours, uh, really to have it working more or less stable um It helped. Yeah, we we we had to contribute to other, uh, projects which are not, uh directly related Or inside osmocom like ofono, uh, python smpp leap or pi debas Uh, I think it's actually some stuff. It's uh, quite reusable for all the projects, uh, and it has been found already like for instance, uh, holga Reused some of this code to test, uh, like, um, let's say massive parallel Uh, location update for modems using this Erlang interface um Yeah, yeah, I'm finishing. So uh, and yeah, basically the idea is to have some kind of upstream collection of tests With that we can share between all of us. So I mean, maybe we don't have all the hardware But somebody wants to do a setup so we can share the test at least um So yeah, here you have a bit of information so you can find the issue tracker. Yeah, the code manual We have some ansible scripts. So if you want to set an osmocom tester main unit with all the dependencies required, uh, it's there And uh, yeah, you can look also for the janking job, uh, to see if something's failing or not. Uh, yeah And that's it