 So I'm going to talk about Osmojism tester, but specifically some work I've been doing lately about adding gprs support to it. So quick recap, Osmojism tester, it's some software written in Python which actually is in charge of doing end-to-end testing with real hardware. So it just starts the whole network every time and then we write Python tests which do whatever we want and then we get some results from there. Yeah, and the idea is to test the different hardware. So yeah, which kind of stuff did we want to test actually? So regarding gprs, we have several KPIs we want to have a look at like latency throughput, actually check if there's some fair queuing between MS at the same time. Then also different scenarios we want to test which are yeah I mean they are usual in in real life and actually we get some reports that some of them don't work sometimes in real networks. So we had this issue for instance in which we've been told that if a user is not sending data then after a while then it won't be able to receive packets from the coming from the network. So this kind of set up actually helps us yeah test these kind of behaviors with real hardware. Some other interesting stuff about this project is being able to test some specific hardware like for instance nano BTS. So we support dynamic PDCH channel there. So switching from PDCH to yeah, DCH channels for instance and that actually allows us to check if we had any regression with any change. But yeah actually this kind of testing involves lots of different projects like Osmo Vita, Osmo PCU, whatever. So lots of stuff involved. So as we said Osmo Gizem tester which is on top it's orchestrating all the processes. I'll go later about that net and s stuff that box but you can basically see Osmo Gizem tester controlling all different processes. So from left to right let's say MS side to more like core network side on the other end. First of all yeah we have a client application in this case for instance we just run IPERF so we just want to send some data and see actually if it goes and like yeah what's the throughput we get. Then we use a phone actually to driving the modems which then we'll end up sending something over there in this case in Osmo Gizem tester we are actually wiring everything up and yeah well it will go through the core network and eventually will reach out the Gizem which will output some IP data which will end up in IPERF server and back. So you may notice I added there in the middle of Ophono and modem UDEF in this case so I'm going to explain later lots of issues we had and how UDEF is ready to do those we'll see later. So yeah going back here so actually all this is running in same host so everything's running on the same network setup let's say inside the same host that actually provides us with some issues so basically when we want to test one specific modem because we want to test like what we said this path if you have everything on the same host actually then it's just going to be eroded to loopback right I mean see if you have IPERF client process and IPERF server running on the same host it's basically going just to loopback and of course that's not what we want because we want it to be sent over the modem. So what's the problem there if we want to set up routing accordingly to route traffic through the modem interface it happens that actually we end up in some kind of routing loopback because you've tried to fit all data into the modem interface and then it gets out of ggsn but then if you route it again it basically will try to to route and then the packets will drop so it was not working at all of course you can fix that by using some IP tables magic like fwmark marking packets and then deciding on whether ftwmark is mark or so you can make some tricks but the problem with that is that actually that's a bit yeah it's not good to do that in case that we are actually changing network dynamically which is what we are doing in osmojiz and tested like we are bringing networks up we are bringing networks down so it's not really fit for here like this kind of complex configuration it's nice if you have just one host actually with some static or mostly static set network configuration and you are done with it so the obvious solution well obvious after testing or thinking about it for a while was actually to have different net name spaces in the same host and then actually move one modem interface network interface to each network name network name space and then just add default routing through that network interface so let's say you have three modems then you create three network name spaces you move the network interface announced by the by the modem so when you connect it as a USB then you get a network interface assigned to that modem you move that to the network name space then in that network name space you configure the routing to send all default traffic through that interface and then it will it will be sent we go back here so it will be sent over there routing will will actually when you run hyperfly inside the net and s it will take the default route then it will go through there and then it will end up in osmojiz gg sn which will output in the default network name space and then it will actually be able to route without having a loop there because the routing in the default network name space is different than this one so that's how the routing is set up so that's the conclusion we arrived here so yeah we decided we want one network name space for modem so now the objective is actually to yeah find a unique net name space for each modem because actually on top we want Osmojiz m tester to be able to infer the the network name space for each modem because like I'm Osmojiz m tester I'm managing modems so I'm gonna use this modem but okay now I need to know which network name space I need to use for that modem to run the tests because as we said we are moving one yeah the e-phase to one network name space so how we do it how we do it in Osmojiz m tester we identify modems actually by its system path so it's it's def path from udef or system path from slices why do we do it well you do we do that basically because of phono doesn't provide with persistent unique names of modems so basically if you list your your modems in a phono the modem names actually can change over time so the same physical modem can actually if for instance if it goes down it goes up or it crashes and so it's read registered the name showed by a phono will change so the only way to actually identify a modem is by using the system path property that also phono brings us actually because we send the patch to add that yeah maybe to illustrate that it's similar to let's say persistent or non-persistent tty usb names right you unplug and replug your serial converter and they get different names and it's the same issue yeah indeed I mean we'll see it later it's the typical you it's the same issue we have with a network interface is right that's why supposedly systems move to have a persistent naming of ethernet interfaces so instead of f0 with one we moved we'll go there we'll see that also have some limitations so yeah coming back here so we have one unique six path per modem and that means that since that that's unique we can infer a network namespace for that so actually the way that Linux makes up the paths if you see as you go to the right the directory names become larger that's because actually the directory name actually contains the whole path so that's like hops in the middle or whatever so it's like it's a tree so last directory actually contains kind of the whole tree so that should be kind of enough to identify where is a device located yeah so I mean then actually you see the net device in this case is ww10 so we decided that the net or namespace that we are gonna use for this interface is gonna be that 1536 which is actually just its position according to usb3 let's say so yeah reach that point we know the interface of a modem and then we know the net net network namespace name that we are gonna use for that modem to run the tests now I'm going to enter into some internals of Ophono, UDef and network namespaces to show you some issues that we can run into so first of all Ophono as I said Ophono uses UDef basically to find modems so Ophono starts up and says okay where are the modems it just starts checking on UDef uses UDef monitor API to do that and then it has kind of its own database inside just in code and then it checks like what what it finds there and then it does some matches and then from there it infers oh this is a modem I will create some objects for that and yeah at the same time it also takes this device from UDef and then it finds that oh it has a sub-device of a type network or net and then it infers okay this is the network interface of this modem so it kind of builds its own object right something important too which we notice is so Ophono as we said it has this modem object and then it infers it has this network object which is attached to this modem through UDef code if Ophono detects that this network device is dropped we'll see later why it can happen so it disappears from from the kernel so this net interface is gone then it will decide to actually drop the entire modem okay so that means that you cannot use that modem anymore then let's go about yeah let's go talk about UDef a bit UDef runs in the default network namespace that means that actually only sees and manages network interfaces or devices devices in general which are on the default network namespace what does that mean so so for those who don't know the default network namespace is the usual one you use basically if you don't move anything to another network namespace so if you don't know about network namespaces you are using the default network namespace UDef also does that so UDef actually talks with the kernel through a netlink socket and from there it receives notifications on the status of yeah the devices right so what happens when you move a network interface from the default network namespace to another network namespace well the kernel decides that let's say all processes on the default net running under the default network namespace should no longer be able to use or know about this network namespace so it tells UDef that this network interface is removed okay and then if you take this network interface which is in another network namespace and you move it back to the default network namespace UDef will actually see that it appears again so actually you can trace that with UDef admin or whatever so you will see a remove event and an add event we ask about this behavior in the github issue you see here in system D and well yeah Leonard Potter answered like yeah it's known behavior it's kernel limitation let's say so system system the UDef runs in the default network namespace and we cannot do anything about that so that's how it works yeah so what's the conclusion about about these two programs if we move a network interface to another network namespace which is not the default one UDef will send a remove event to its clients and that means that the phone will receive a remove event for that network interface and that means that Ophono will drop the modem out so you cannot reach it anymore and that's not we want what we want of course we because we want to use different modem net different network namespaces but still we want to be able to use it through Ophono so what what we need to do we patch Ophono to actually avoid this kind of behavior so we have Ophono patch in our Osmo GSM tester branch which actually prevents Ophono from removing modems if the network modem is removed if the network interface is removed sorry okay that's some of the limitations let's go now to see an example on how a test using gprs in Osmo GSM tester looks like like all tests basically the first block is common to all tests we import modules we start the whole network so we get the objects we start BDS start a GSM start whatever start then we have the MS object which is actually a modem which underneath uses Ophono and we tell the HLR to add the subscriber data from this MS then we tell the MS to connect to the MSC that's actually on a synchronous function so it just starts the registering then we also tell the MS that it is allowed to attach in gprs to this network that's also asynchronous and then in the test we basically wait for the MS to be connected to the MSC so to be registered and to be attached to gprs so once we are here we have an MS which is connected to the MSC and actually gprs is attached at that point we want to activate a PDP context so yeah we just tell the MS which APN do we want to connect to and the protocol we want to use IPv4 IPv6 whatever and next function is setup context data plane we'll see later what this function does internally and once that context data plane so basically just summary this this function does all these network namespace movements and setups and whatever and then we simply run a ping command in this case in this case so the the first case I show it was using IPerf in here I'm just pinging the ggsn address and if it fails then basically the test is gonna fail and then finally I deactivate the context so that's how the test looks like and now we're gonna see some internals of this test first of all that's what Ophono does behind the Osmojism tester API so just for you to understand green stuff in variables it's divas paths so you have a divas path for each modem and then you'll see context why that's another path which is created it's a context is a PDP context for each modem so first of all what I mean that's that's useful in case you want to do it manually with Ophono so first of all each modem has an interface called connection manager from which you can operate gprs related stuff so you can see example of get properties so you have some information like right now it's not attached beer actually is not implemented with QMI there's a ticket about that whether we are allowed roaming powered whatever so what do we want to do in this case so we power it on and we allow roaming in case we don't use the same MCC yeah whatever then we wait for Ophono to tell us that actually we are attached yeah because attached is only a read only property so we power it on then it should attach at some point then we add a PDP context and it will return the divas path so we can set it up now on this context we actually set it up like access point name username password protocol etc finally we activate it and once it it's activate we can get some properties about it and that's what the kind of properties we get from Ophono once once we are established so some interesting stuff you get the DNS values which are set on Osmo GSM we are still not checking those values are correct in Osmo GSM tester but it's it's quite straightforward it's just about having some functions actually we fixed back in Ophono a few weeks ago because basically the first one was duplicated actually this this probably is the case and I'm running an old version of Ophono here something also really important for us Ophono provides us with the interface name here from assigned to that modem and actually in here it says method static I think well links is will know better but that means you can configure modems in two ways like dynamic which is using the HCP and actually that's what we are using and static is you need to set up the address I think oh by the way I think addressing here it's actually the address that is assigned to you by GSM so if you remember the test I was talking about explaining you later the setup context data plane function so that's what basically does as I said we have to move so if at this point we know the interface name from that modem so we know which interface we have to move to our network namespace I also explain already how to get the name of the network namespace we want to use so we have all the data there so we just move it I'll explain you later how we do that but just call this function and it moves the network interface to yeah the network namespace we select it and then we have to set up this new network namespace and the modem e-phase so basically we have to bring up the network interface and run the HCP on it yeah and so why do we want actually to move the network namespace the network interface to the network namespace every time so originally we actually only had one Python script which just set up all network namespaces and moved all the interfaces there but that that could be perfectly fine if the interfaces would always stay in that network namespace but that's not the case because modems crash and basically when a modem crash well when modem crash then the kernel actually drops the network interface and then when the modem is back again the modem re-registers the network namespace but the kernel may decide to use a different network interface name that the one that it used to use before so basically it will try to get so the way that the kernel assigns the network namespace sorry the network names is basically it looks for the first counter available in the default network namespace as far as I could understand from looking at several traces and yeah that's dangerous because actually if you may end up with I mean in this scenario you can even end up in the situation where you have two modems and then modem a has a ww10 and modem b has ww1 and you can end up actually at some point having the two modems actually with the two network interface names swapped so and that's dangerous because actually a funnel doesn't detect this kind of stuff especially because we added this patch to avoid removing the devices so it could happen that we were since we are moving network interfaces to network namespaces it could happen that we were moving actually the wrong network interface because we still thought that that name was assigned to that modem but it was swapped so it was a bit of a mess yeah I'm not going to go into details of the series of events in case you want to look at it but it can happen so what's the conclusion from all these mess of honor should require catching interface renaming at some point so you can just issue a interface renaming by calling IP link set that way of course we don't do that but I mean in theory in theory it should support it in case somebody just wants to rename an interface for the modem and they don't do it and yeah since all these naming stuff is a mess let's just go saying and use persistent naming also for network interfaces so we are we don't end up having this kind of renaming mess so we decided we want to have persistent naming for interfaces we use so let's add some rules to you dev to actually have persistent naming right so here's a sample of the rule we initially used to do that so it will basically use this import built-in net ID so this actually this fits into the into you dev some variables which you can use to set up persistent naming so as I said before you nowadays we have this persistent naming of interfaces in Linux like you have well that like that one wwp blah blah blah so this actually is calculated let's say or generated by this net ID module and it's fit into that kind of variable ID net name path so that looks fine right I mean for the first case we initially get a ww1 to interface by the kernel and we just rename it to something persistent so in case you don't know also the last lines of the new name it's also based on the path on the USB path well it's persistent as long as you don't move the path but it has some limitations we'll see in this case for instance you may see like that that was real so after applying this rule some of the interfaces were moved so we're renamed but not some others they were not so there's small difference between one cis path and the other one one is a bit larger than the other one so what's the problem here so linux defines this define if num sis which is 16 and that's the maximum name an interface can have in Linux and what's the problem here so as you start stacking up stuff in USB your tree becomes larger your path becomes larger and eventually you cannot fit that into this persistent naming scheme right so if you count there it's probably I mean I made it probably by hand but should be like 15 but then if you try to add a zero like change that to by a 10 it's going to be 16 so the kernel basically in this case when net ID is run it detects the path is too large in UDF so it basically won't allow this and ID net names path won't be fit so it will be empty so that means that the rule is not applied because yeah system the UDF detects that the path is gonna be too long so it's not gonna work so how do we fix it well we have other ways of let's create our own persistent naming scheme for those cases in which ID net name path is not set by ID net let's do it our own way so we take the dev path so yeah you remember how we got persistent name persistent network namespace unique names we just got the last part of the system ID right it's actually the same as in here just it just builds the path right but it's too long okay so just let's hash that that's the way I found out I mean we take the dear name of the dev path so in this case it would be like 15410 we hash it I have to use your name by the way because last the last stuff on the path it's actually the interface name which of course changes over if the modem crash because it increases so that's not unique so you need to take dear name and yeah you hash it and you take 14 bytes which is like yeah probably could be 15 but as you yeah oh yeah but there's an error there so so yeah we end up with this kind of naming but it's unique it's a bit strange but it's unique so it doesn't change and we are not affected by these issues anymore no collisions yet and if we have some I guess we can just rearrange some usb paths to not get them I mean it's too like too yeah elevated to 14 so I mean we are not like running I don't know one million modems there so should be quite fine and still I mean for some parts we still use the other one so it's still less possibilities so yeah okay so up to here everything looks fine we are solved but actually I still so sometimes some interfaces were not being renamed so debugging it a bit more yeah and actually this renaming look a bit like random actually so it was not happening always on the same interfaces what happens sometimes so debugging it with you def admin test so with debug enabled I could see that actually you def was sending some failure messages saying that the interface was actually busy and looking up on the internet that means that actually so in order to change the name of an interface it must be down it cannot be up so you have to turn it down before before changing the name but actually that was a bit strange because I mean the interface was just bring that brought that by the by the kernel I mean just the device was started so I didn't understand how could it be that it was up if nobody was using it but actually it seems that somebody was using it even before you def so it's a known issue that DHCP CD it's also using a netling socket against the socket so against the kernel so it can happen that DHCP CD gets notification before you def does about this interface so it tries to use it to use DHCP on it that means it brings it up and it tries to use it so it can happen that when you def tries to rename this interface this interface already up and being used by the HCP so the solution for that is actually to tell the HCP CD to not not use these kind of interfaces by adding this configuration entry so we say just don't handle interfaces starting with Www or R which match our prefixes by the way if somebody wants to read more about the issue in there so up to here we have persistent interfaces and we are fixed with that some other stuff so one limitation of network name spaces I could not solve in a clean way is basically you need root access to actually change network name spaces so from one process to another you must be root and Osmo GSM tester is run under Jenkins which is not root so and we said that we are creating network name spaces we are forking what we are creating processes in other network name spaces so we need to be root right so okay just quick answer for that is okay let's just add some pseudo grabbers I know it's super ugly don't hit me please but yeah it kind of works right so we have some EDC pseudoers lines there which allow for Osmo GSM tester group to run some scripts right like these Osmo GSM tester net and s execs which basically calls IP net and exec the network name space and and the problem we want to run you you may remember these two programs here IP link set dev blah blah blah use the HCP that's run on the network name space to set it up so that must be run under root and I'm back here but then that actually made some other problems so Osmo GSM tester you can see the process tree there Osmo GSM tester runs pseudo which then runs Osmo so the script and the script runs hyperf right because we are running that on a different network name space so you have a process which is not root running something as root because pseudo is sui so that's root that means that you cannot you can only kill that if you're root so we have to we had to add some code in Osmo GSM tester to actually do pseudo kill in this case because otherwise we could not kill that process that was one of the issues next issue was that actually while running tests we noticed that these processes run under pseudo for some reason were not being killed correctly even if we were using pseudo kill and so reading a bit more on the pseudo documentation it says that pseudo prevents being killed by its children so it's to prevent I don't remember why exactly but it's to prevent some kind of yeah situation but actually the documentation is wrong it's not that pseudo prevents its children to kill it it's actually it prevents the whole process group to kill it so that means that that means that the parent itself the parent process which forked to pseudo is not allowed to kill it no sorry no I'm wrong here it's not that you cannot kill it it's that it would want forward the signals you send to it so usually if you send a signal to pseudo it will forward it to you to the child to the child right it created but the documentation states that it's not gonna forward these signals for signal sent by the children it created so that's to avoid yeah that's to avoid the children the children to kill themselves by killing pseudo something like that yeah but the problem here is that it's not only kind of children it's it's more like it extends to relatives you know and yeah documentation doesn't state so but actually the the process which forks to pseudo cannot have its own signals forwarded so if you run something under pseudo and and then try to send some signal to pseudo it won't forward it to its children actually if you the on the terminal I think it works I'm not sure but not if if the process is just for directly so in order to fix that we have to in Osmojism tester when we've create when we call pseudo process we have to add this this flag here which actually changes the process group of the of the pseudo process so we can kill it later so that's the end of the drama so far that's are quite stable GPS wise nowadays of course modems keep crashing but it's not like we can do a lot of stuff about that but they just crash from time to time and we recover from that so only few tests fail from time to time to do that and we still have to contact yeah vendors or whatever to get some updates so related topics on the future I'm almost done by the way yeah we need to add 3G network support which also implies a date plane there then some of our implementation stuff which may be useful I won't go into detail now but Ophono in QMI we come to specify IPv4 v6 EUA's or IPv4 IPv6 it just gets whatever it wants and use whatever they want because it's not implemented correctly and so right now we are running hyper test fine but we're actually not checking the results so I mean we get if you look at the locks you see uplink and downlink throughput for each modem but we are not checking those and comparing them with some thresholds or KPIs or whatever so we need to do that and we need to find out what do we want exactly there and yeah there's plenty more tasks in red mine so it's final rematch that's my last slide what we found out network namespaces and real hardware don't play well together so that's a mess I mean my intention yeah my feeling is that that's super nice when you are running virtual machines and whatever but software which is prepared to run with real hardware doesn't play really well with that like Ophono or you that yes always I think I copied this line in all my osmosis and tester talks controlling the modems correctly is hard stability issues they crash all of them all vendors no matter which one and yes this if is name limitation of 16 byte really sucks and that's it yep did you try to investigate what actually makes the modem crash no as you can see I was busy enough trying to get test working so what do you mean with some something simpler well probably I mean how I'll probably knows more about that but as far as you know 80 yeah it's kind of a rainforest you know like you don't want to go there well the problem is 80 commands have many different dialects and if you've ever tried to write a parser for any kind of modem 80 commands it's a nightmare because every modem or even firmware updates then they speak slightly different dialects and especially if you then want to deal with unsolicited results codes and so on I mean we don't want to only set up a data connection if that was all we wanted that would be nice and you could I mean you could use it's not simpler but you could use modem manager for example but we want to do all these voice features right and that's what most part of Osmo GSM test is about also is to really register unregistered issue of voice call acceptor call you know whatever maybe put the call on hold in the future or send SMS and so on and if you want to do all that by hand using 80 commands that's going to be a rather complex task with all the things running asynchronous and processing unsolicited result codes so it's also not simpler in the end also interesting from QMI from using QMI we get some nice features we actually have in our branch of Ophono like we print all the QMI messages we get to standard out so I mean we get quite a lot of logging actually we get we run out of journal CDL or journal the buffer lock for Ophono because we print so much information during the 16 hours it runs the whole test that actually I think I tried increasing the buffer of it but I'm not sure if it's actually applied correctly yet I have to check so just out of curiosity Ophono uses to talk to the modding 80 commands or there's it depends on the modem so some modems so when it runs through you def it actually finds which kind of modem it is and then for each modem it has yeah it decides like no I'm gonna use 80 for this one and then it has some specific configuration for that one because of course as we said like each modem uses a slightly different 80 commands and so there's lots of e-files in 80 I well I see some patches about that like really trying to oh it this works for me but that doesn't work for me so it's kind of so at least with QMI it's kind of standardized interface though some modems we saw that gov 2000 they implement older versions of QMI then some stuff is not supported so yeah how possible or difficult would it to run the whole Osmo GSM tester at home like does it need to be within a Jenkins environment or anything like that well the virtual one you can you can run it in Docker the other one it's it's a good question I mean it it depends on really if you want it or not like we have some debug test which is nice because it sets up everything and then you can use a phone to do some tests it's it's a bit I mean there's a lot of configuration involved but once you have that and you copy the one from production it's kind of feasible let's say the problem is that it's thing it's thought to actually be on a static system so of course if you you change your modem to another place then you'll change you need to change the configuration because the configuration contains the system path of the modem so I mean it can be done okay the other thing I was just thinking there is remember a few years ago looking at what you could do with Android devices and from like a automated standpoint if you looked at that at all I seem to remember well thinking that you should be able to like to airplane mode toggles and well I mean you can do that because a phone actually so I think the guys from Yola they have their own fork of a phone with lots of patches and actually they support the backend for real which as far as I know is kind of the backend that Android uses to control the modems so that means they support lots of modems that are actually Android but of course you need to run some probably some Linux based operating system on those Android phones but I mean that there's some phones which do that and you don't need like fancy drivers you just need the modem there to test stuff of course it would be possible to write some Android app or whatever but if it's already sucks that the baseband processor crashes and you have to recover and wait and so on and so on just imagine how often your phones will need to reboot during any kind of automated test setup and how many hours every day you're just waiting for entire Android phones to reboot completely or get stuck somewhere or whatever so I'm not entirely optimistic that this would increase the reliability or anything on that regard I mean what many people don't know is if you actually look under the hood on many phones the like the baseband processor crashes several times a day and you know it just reboots so not saying I'm not expecting it to be any more stable or any more reliable than using modems I mean in our case we also we had lots of phono crashes because we stress it quite a lot because we are really like okay now we pour it on now we pour it off and we do it really quickly because and then some stuff is still ongoing so I mean yeah we reported several stuff because so out of curiosity why do you need DHCP CD in the first place I thought to use you DHCP in the slides well that's that's I mean Osmojism tester is just running on a DB and on a DB and computer and it happened to have DHCP CD running actually it only happened on the R&D I don't know why it was only running on the on the on the R&D setup not on the prod setup so that was even more messier because I had kind of it working well I I was seeing that kind of things in R&D but not in prod and then the system paths also were longer in prod so when I had it working in R&D then I started running it in in prod and suddenly nothing worked so yeah one thing that I'd like to add about the setting up your own Osmojism tester as far as I know the setup of the actual like the software and all the stuff that you need there are ansible playbooks for that so it's only in quotes all the configuration files which is complex in itself but at least all the software installation and all that is automatic yep so does the modem crash actually and how do you deal with it within one run of the whole test? That's a good question I don't know to be honest like the problem is it was so the problem until now is before these fixes when it crashed then yeah everything got into a messy state because then interface were not moved so tests were failing because it was trying to send stuff through the interface which is not correct so it was really difficult to figure out right now stability is quite better in the sense that if it crashes it will only crash for that test so that test will fail only and not the next ones but yeah I didn't really had time to look into that because it's kind of stuff that I do daily because it takes like 16 hours to run or 18 hours so I just look at it at the results every day but anyway I don't have time to like fix all the issues like there's some stability issues also still in nano bts and some stuff but it's quite better now I mean if you look at the wave yeah so just only one thing we support the scenarios like yeah like I'm in a call and then well I'm using data packets and then I get a call in the middle so it switches to the call scenarios in which we actually change PDCH dynamically so we test all these kind of scenarios so I mean if at some point you want to test any of these scenarios or you want to add it should be kind of easy yeah and I mean easy because sometimes you think adding a small feature but then everything crashes and you spend like one month trying to fix stuff and yeah that's it so far the best one actually we are using so we're using Sierra wireless models for call so for voice related voice signaling related tests and EC20 for gprs ones because Sierra wireless was crashing more during gprs tests I checked and EC20 I think I had issues with them with audio so well with audio with signaling of voice call so actually Osmojis and tester allows you to select it kind of modems through a configuration for each test and we do that so we say run with this BTS and this modem and or this modem type and then it allocates it from the pool and whatever yep