 Let's look a bit at the Osmo-com test suits, how they work and what do we do there, not at the code, we wouldn't have time for that. But okay, who is taking the notes about the start and beginning of the, okay, both of you. So we have failed, so we have failed over even. Somehow the two French speaking people, okay, yeah, the test suits, we started to develop them last year and also this year lots of work in them. We used Eclipse Titan of course and we only use the command like compiler and make files. So there's no Eclipse other than that it's released under the Eclipse foundation. We run those tests containerized in Docker. I'll explain why and also why it's still annoying and we execute that from our Jenkins continuous integration. So bit of terminology, ATS is the abstract test suit, MTC is the main test component, PTC is the parallel test component and IUT is the implementation under test. So they have many more acronyms but I'm going to use ATS and IUT here. So ATS means the test suit and IUT is the implementation under test. These are the two important parts. So the test suit philosophy is a bit that we want to normally test one network element which is our implementation under test, let's say the PTS or the BSC and we want to test the external behavior of that. So mostly, I mean sometimes we violate that because we use the VTY or the control interface to also check some internal state but primarily we want to test the externally visible behavior on the external interfaces that's GSub, ABIS, AMNCC, PCU interface and so on. So we emulate the entire environment around this program from Titan and from TTC in three and we don't reuse any of the OsmoCom C code in there intentionally to avoid the problem of testing one implementation against itself and both doing it wrong. And we test as if we can against the independent implementations that have been published by Ericsson of the respective protocol. So there we get already some interop testing because if Ericsson thinks the message has to be structured some way and we also think that way then we have more confidence than if it's just us who think that way. So what do we test? What do we want to test? Successful cases of course of particular transactions or procedures. We want to test erroneous cases like no answer timeouts or negative acknowledgment cases which many of them are difficult to reproduce in a real setup. I mean we have Osmo GSM tester of course to do end to end testing or even if you sit down with phones it's very hard to make the phone send a negative acknowledgment in some case you cannot do that so but we need to test those cases and that's what's really nice to do in the TTC in three test suits. We want to test some load and resource exhaustion situations, spec compliance and focus on the functionality that's relevant to that particular implementation under test and not about other bits of the system. Well, why Titan and so on we know that now. So for what components do we have test suits already basically almost everything now? What is missing here? Yeah, but we have other tests for that. We tested that's actually Osmo BTS is being tested this way both with real phones and with emulated layer one that's part no Osmo PCU is missing. Yeah, it's not a part of CNI. It's out of spec, out of scope. Anyway, yeah, so Osmo PCU is missing. I started a test suit. It has I think two test cases and it's not automatically executed yet. So that's yeah and a BSC net is not tested. That's correct. But yeah, that's, well Osmox is part of Osmo BSC net and so it, yes. Okay, so the diagrams are you know I don't do diagrams. This is just a dotty graphs, they're a bit confusing maybe but still it's a graphical representation. So if we test the BTS, how does this work? So we have our abstract test suit, that's our BTS test dot TTCN. And we have our IOT that's Osmo BTS tier X in this example. So in the abstract test suit, we do not implement OML. We only implement RSL so far because OML is another layer of complexity and it's annoying and we wanted to get some tests going. So what we do actually is we violate our philosophy by running another program from Osmo Kong which is Osmo BSC but we only use it to terminate the OML to bring up the BTS. So basically we tell it the RFCAN to broadcast on the time slot configuration and so on. And the RSL we establish basically here directly into our test suit. So the test suit behaves like a BSC towards the BTS at least on the RSL side. And for the air interface simulation we have basically the Osmo BTS tier X sends bursts over UDP to fake tier X which sends bursts to tier X, which sends bursts to L1 CTL, this is actually a copy and paste error, L1 CTL to the test suit. So we see the APIS side here and we see the UM side here and we can test between those two. And we also have a PCU socket connection. So we also test, we behave like the PCU here towards the BTS and we attach to the control interface of the BTS. So we basically anything that we can attach to, we attach to and then we test if our expectations are met. So let's say and also sorry I forgot the VTY here also we control the BSC via VTY. So we can for example in the BSC change the time slot from TCHF to TCHH or the PDCH or something like that. And if we do that and we restart the BTS we expect that on the PCU socket we get an PCU info indication that tells us oh there is now an additional PDCH time slot there. So by attaching to all these interfaces we can do these kind of tests. Correct yes. I left that out for simplicity reasons, it's completely intentional. Um, what we, I mean this is of course with simulated physical layer, simulated hardware but we can actually, we can not only but I've done this myself and it ran out of the box. You can do this with a real Osmo-com BB phone and a real BTS. So I used the sysmo BTS on my desk and I put a C123 next to it and I ran the exact same test suite and it works. I mean of course the receive levels are different than the simulated ones. Yeah that's for measurement reports there's some adjustments that need to be made because of course the real radio interface has more fluctuation and no deterministic loss but apart from that it's really we can do the same with real hardware and it's in the process of doing that also with all the hardware we support. So on the BSC tests it's a bit like the same. So here for whatever reason Dottie decided to, Dottie decided to here's the IoT on the left side and here's on the right side I don't know. Anyway so here this is the BSC that we want to test and here's our test suite that tests it again for OML. We don't want to test this, we want to test it but we don't have the time to implement OML completely. So we run something which we call Osmo BTS OML dummy which has recently been added to Osmo BTS which is basically only implementing the OML part of a BTS but not the RSL part. So we can take this out of the equation and Osmo BSC will be happy because some BTS has spoken OML to it and then it will talk RSL to the test suite. So we have ABIS here, we have the control interface, we have the VTY interface and we put Osmo STP in between because this is the only way how we can exchange A interface messages between those two. And again we can, let's say we can claim a Ratch Burst was detected on the BTS and then we expect that the channel is being allocated over RSL and then we can exchange more data and we can send so many Ratch requests that all the channels are exhausted and all kinds of things and that's the kind of tests we do there. But I don't have here on the, it's mentioned here but not on the graph that's actually missing. We also attach to the MGCP side. So the test suite also emulates the media gateway towards the BSC. So again all the protocols that the BSC speaks is tested. For MSC it's again the same story, it's probably getting a bit boring. So we have Osmo MSC and we have connections for GSub too. So the HLR is emulated, we have MNCC, so the external code control is emulated. We have SMPP, so SMS is emulated, we have control interface and VTY interface and we again use the STP for sending STCP between those two. So same story and again MGCP is stated up here but not down there. So also the media gateway control is emulated and then we can simulate that there's no response from the media gateway to some command and we can check if the MSC is properly resolving the transaction and properly cleaning everything up. For the media gateway itself it's just a little bit of the same. We have the media gateway that we want to test and we basically have in the Titan code we have an RTP source in sync and we have MGCP call agent code. So we can basically create connections on an MGCP endpoint here and then we feed RTP data into it and we get RTP data out of it and we can test if we receive all the frames or if there was some latency that we didn't expect or whatever else. For the HLR it's rather simple because the HLR only has, I think it has a control interface that we're not testing yet but it's VTY and GSub basically that we talk to Osmo HLR here. For the Osmo SIP connector it's basically SIP and MNCC, so we emulate the SIP side and we emulate the MNCC side and we use again VTY to control it. Not so many tests yet but at least some. SGSN tests while we simulate the GP interface basically the PCU side and we simulate the GP side which is the GGSN that we simulate and then we claim that there is let's say PDP context activation to the SGSN and then we expect that it will do a PDP context activation towards the GGSN and again we can do timeouts and negative acknowledgments and so on. For the GGSN again same story so we have the GTP side, we emulate the SGSN, we emulate the internet all of it of course on the IP side and we can test whether it does what it's expected to do. Now we take all these components and we dockerize them where we run one process per container so we don't do like an init or whatever inside the docker and we package either the implementation under desks such as Osmo BSC so we have an Osmo BSC container then we also package the abstract test suite so we have a TTC and 3 BSC tests container and then some other utilities that we need let's say Osmo STP or the TRXcon or Osmo BTSOML maybe also package that as containers and the main reason is that we have a setup that can be replicated on any machine without having to change all the IP addresses, port numbers, global titles whatever all the configuration so basically we have a script that is included in the test suite jenki.sh script which will first do a docker network create to create a network with certain IP addresses and then we start the individual containers with fixed IP addresses and we have a set of config files where all these IP addresses and ports are hard coded so with a single command you can basically start the entire test suite and get all the results in the end. Which gets us to the Jenkins integration so we have the four steps that you need to follow if you want to run it but that's basically what Jenkins does. It gets docker playground.git which contains the docker files for all the individual elements it will build those containers and there's some git magic in the docker files that makes sure that the docker cache is invalidated whenever a git commit has been made then we rebuild the well this is the implementation test the abstract test suite where we have the same so if there was a change in the abstract test suite the cache layers of docker images will be invalidated you run that script which then creates the network and starts all the components and collects the test results and one part that I didn't mention about titan it has the concept of output plugins so there's a junit xml output plugin for example so you can get your test results in junit xml which is something that Jenkins again can import you get the log files we also create pcap captures for each test case and if you look at the if we quickly look at the jenkins osmo.com org quickly yeah just slow so there's a ttc and increase the font size a bit there's a ttc and three tab that you can click on which shows you all the ttc and three tests now we can click on let's say the bsc test and we can look at last successful artifacts where you will get basically the the log file of osmo bsc in the config file that was used during the test execution but you that's the most important but the bsc tester part you see basically you get the for each individual test case named by the test case you get the pcap file what was written and the log file so if we look at for example this this log output oh yeah i should have clicked on view to see it in here that's basically the titan log file that it generates sec it's extremely verbose everything that you can imagine that's why you have this log formatting and log filter tools to post process it if you look at it and we have the pcap file that we can open in wireshark to see what messages were exchanged in the test and we get that's actually the really nice part in the test results analyzer we get these nice matrix where we see basically which test cases have passed or failed and if there's any change in a test case execution and so on so this is run once per night and as you can see it's right stable at the moment there's no changes in this we have a couple of them that have been failing forever and you get also here the the the graph basically so we have uh 67 past tests five failed ones and so on that's basically for the bts and the same exists for all the individual elements and you can you can check that and you can look at the results so for for analyzing the issue it's actually you don't even need to rerun the test you can just look at the pcap file at the log files and so on