 Welcome to this talk about IU3G testing of Osmo MSC, SGS, and H&B GW using TTCN3. So I somehow couldn't find more acronyms to fit into the title of the presentation. My apologies for that. Yeah, I only have 29 slides, but I also only have 30 minutes and not 45. Yeah, so the Osmo Com TTCN3 test suits, some of you know we developed quite a number of test suits in the 2017-2018 time frame. We compiled them to executable code using Eclipse Titan where we just use the command line compiler and make files to build this and you don't need any IDE or Eclipse or whatever for that. Those tests, they run containerized in Docker and we execute them by our Jenkins continuous integration. As if you follow this a bit, also know the tests that we run normally or at least if they run how we intend to run them. They test only one particular network element, so we only test the BTS or the BEC or the MSC or the SJSN or whatnot. And we only test the external behavior on both 3GPP and non-3GPP interfaces. So that means A, GB, IU in this case now, but also VTY and other interfaces. We also don't use the Osmo Com C code as the protocol implementation, so we test against another implementation. I'm going to skip some of the slides here, why we did that. Now if you look at SGSN tests so far as they run today or until today, we have external interfaces, only the GB interface which is the protocol to the PCU, the GP interface which is the GTP protocol-based interface to GGSN. We have the GSAP which emulates the HLR in our tester and we have VTY-related interfacing in the tests, but we don't have any 3G-related testing, so the typical test case looks like this. We have the SGSN on the one hand side and we have these three interfaces that we attached to, but actually GB, GB just, yeah GSAP is not just a missing arrow here, and we have the abstract test COD, SGSN tests, CDC and 3. Now if we look at the A versus IU interface, what's so different about testing IU? In theory it should be almost identical, since there's a very large similarity between IU and A. Both use SCCP connections per subscriber radio channel, both encapsulate the same layers. The same layer 3, yeah, somebody has rang. This is probably about our dinner that gets delivered, so somebody has to... Oh, okay, okay. Yeah, so both encapsulate the same layer 3 D-tab messages between the UE or MS and the MSC or SGSN, and there's more or less a one-to-one mapping between BSSMAP and RANAP. So in BSSMAP we have a complete layer 3 info which becomes the initial UE message on RANAP and assignment becomes a RAP assignment, the clear becomes IU release, also note the enormous difference of using entirely uppercase statements and camel case on the other hand side, but in practice things are a bit more complicated than that, and that's the A interface and IU, so that's the circuit switch side to begin with. To just compare, let's revisit the protocol stack, so this is our typical control plane stack for a circuit switch, right? We have our LAPDM here, RSL and RR terminating the BSC, and mobility management and call control go through all the way to the MSC where we have that nice protocol stack here. I mean, we of course pick it over M3OA rather than the classic E1 interface, but it's the same stacking nevertheless. If we compare that with IUCS and we only look at, we colorize whatever has changed, these are the bits that have changed in the protocol stack from 2G to 3G, which means basically on the MSC, which we want to test, only the BSS part has become ran up, the underlying stack is the same, the protocols on top are the same, so normally it would expect it's super easy to also test the IUCS interface on the MSC. What's now special about it, it's specified in ASN1, not a human readable text tables format of the traditional GSM specs, and it uses rather advanced features of ASN1, which is information object classes and it makes very heavy use of them, so it doesn't only specify a type or let's say a message and a message with information elements, but no, each information element gets wrapped in an information element container and then you have a message type specific list of permitted information element containers and then the entire message gets wrapped in another container and then you have some containment there. And it uses also an encoding that I think nobody really outside of 3GPP uses, which is aligned packet encoding, not packet but packed encoding rules and Ericsson didn't release any of the related protocol module for IU like they did for A which made our life rather easy on the interface side in TTCN3. So there's a very nice relationship between TTCN3 and ASN1, is that both are specified or originate at ITU and they have some similarity, which means that TTCN3 spec actually says you can natively use ASN1 files. So in order to speak or to interpret information objects or structures or whatever you might call them, data structures in TTCN3, you don't even need to do any translation or whatever of the ASN1 file, you don't even need to translate it into TTCN3. Like normally if you want to use ASN1 data structures, let's say in Erlang you compile them to Erlang or if you want to use them in C++ you compile them to C++ and so on, but in TTCN3 you don't even do that, but basically there's a one-to-one mapping between an ASN1 language construct and the TTCN language construct, so you can natively use ASN1 data types and so on in TTCN3, which is really nice. So you can just blindly use all the data types defined in ASN1 files and use them. There's some exceptions though, there's some name mangling because a minus or hyphen or dash is not permitted inside a type name in TTCN3, so all dashes are replaced with underscores, so you look at the ASN1 file and the data type is called ranap-pdu, then you have to, in your code, you have to write ranap-underscore-pdu and there's some other rules, but they mostly don't matter at that point. So now eclipse-titan, which is a specific implementation of TTCN3, but it doesn't implement all of the spec, so eclipse-titan can parse the ranap-ASN1, which is nice, so I just copied the ASN1 files from Wireshark because I had Wireshark Git tree cloned anyway, and you can just add the ASN1 files next to the TTCN files when you call the compiler. It's just like source code to it, and it just uses it and it generates C++ code and there's no intermediate TTCN file generated from this, which is elegant if you think of it conceptually, but actually it makes your life quite a bit harder because you now have to sort of use data types in TTCN3 for which you do not have a TTCN3 header file or a record or whatever definition of how that data type looks like. We have to mentally do that translation between how does it look and how does it specify in the ASN1 side and how do I actually need to name my variables but the field names and that kind of stuff, which would be simple if the ASN1 file is simple, but with all this nesting and the nested information object classes in the containers, it becomes rather hairy. So what I actually do sometimes is I read the generated C++ code to understand how it's going to be called in the TTCN side, but there's also another nice hack that I found how to actually do this. The one big disadvantage is that the Titan implementation only supports basic encoding rules and no aligned packed encoding rules. So now we have the problem that the ASN1 can be parsed and we can write and use all those data types, but how do we deal with this packed encoding rules? So what we need to do is we need to do transcoding, not in the media gateway, but in our test and not between voice codecs, but between ASN1 encoding rules. And what we need is a transcoder that parses, ran up encoded in aligned packed encoding rules and outputs it in basic encoding rules and vice versa. And what kind of tool do we use for that job? So the ASN1C that we use for our LibOsmo, no, LibIU client or whatever it's called, LibOsmoIU, I think, no. Anyway, for our CE code, it doesn't really fully support the version that we use. It doesn't really fully support all these ASN1 constructs. So we do not end up having one root data type where we can just say, well, here's a blob, transcoded to an abstract representation and then reencoded in another format. Also, the mainline ASN1C doesn't support all the features required for that because it still doesn't have APR support. And also, of course, it's always good if your tests don't use the exact same code than your implementation under test because otherwise you have the same mistake on both sides and you run into problems. You could use the Erlang, but then, yeah, you would need to run an Erlang VM next to all our TTC and three tests. Yeah, you can do it, but somehow another level of complexity that I didn't like and one option that also exists is that you simply use, if you have access to it, of course, a proprietary ASN1 compiler and we link to a library, so the proprietary ASN1 compiler generates some code and then you link that code to just do the transcoding between AVR and PR. And that's what I did. It's called LibFFTranscode. I think it's sort of funny because FFASN1C is from Fabrice Bellard, like FFmpeg, and FFmpeg is doing all kinds of transcoding of video and now we have a LibFFTranscode that does nothing to do with video. So it's a binary-only shared library that we compile at Sysmocom because we do have a license for that compiler. We cannot share the source code to it, but since it's only for our tests and it's a very well-defined functionality with a clean API, there's no question of, you know, linking against other software or license-related questions because you're literally only translating data between two different formats using a very clean API. I think it has exactly three symbols exposed, this library. And we then implement their RANAP, RUA and HNBAP, which are the three protocols that we deal with in 3G, but this can be extended actually yesterday during one of the talks I did that extension already, so we already have S1AP or this MyVersion that has S1AP inside, so we can use the same strategy for LTE-related protocols where the same problem exists in the same library. And it's actually rather small. So for these three ASN1 standards, these three GPP specifications in ASN1, RANAP, RUA and HNBAP, which if you've ever looked at them, this is like tons of ASN1 code and we generate, I don't know how many megabytes of C code are using ASN1C. The entire library is, I think, about 300 kilobytes or so in object size. And the Debian package compressed down is 50 kilobytes, the binary that you have to install. So just so much about the, how can I say, the ingenuity of how FFASN1C deals with these problems without generating tons of code or using tons of resources. So now that we have this, or we think that we have this transcoding library, how do we write the templates? As I said, we don't have the TTCN file to actually see the data types and then build them. Well, of course, we can just use a pcap file and run the payload of those pcap files through the decoder, and then in the log file of TTCN3 we will see the decoded abstract representation and then from that we can build templates. Literally copy and paste them and build templates from that. Okay. Well, this is what you get when you try to do that, so it tells you expecting to tag mismatch received universal 16. So somehow Eclipse Titan cannot parse the BER that is generated after the transcoding process. So the proprietor is in one compiler, parses the APER, translates it to BER, and then you feed the BER into Titan and Titan says, this does not reflect my understanding of basic encoding rules for this ASN1 syntax. I gave up at that point about six months ago when I said, okay, I look at some other problem now. I don't feel like investigating that. I did this again in April this year and debugged further and I could actually make it work somehow with a hack in the runtime library source that's proprietary again. But the other direction requires a fix in the compiler and actually it was a bug in FFASN1c. Luckily, even though there is no support contract and the product is no longer sold publicly FFASN1c and we bought it seven years ago. Still we got an update from Fabrice. So thanks for that. So now we have working transcoding library and the strategy of running the APR through the transcoder into TTCN3 to get the abstract representation and using that to generate or to write templates has worked. Now we need to integrate that into the existing test suits. A lot of what they test is Layer 3 DTAP anyway. So like an authentication test or a location update test of course is more or less the same. So how do we reuse them without changing or copy pasting all the existing test cases. So one thing I did is I generalized what we call the BSS app emulation into RAN emulation so the changes look like this if you look at the actual DIF rather simple. So now it's a RAN adapter and not a BSS function and the RAN adapter is generalized for those both sites and we also have a rename from the codec port so there is now actually it's a copy so we have a BSS codec port and a RAN app codec port and the RAN adapter on top abstracts these things away and there's a new configuration option by which you can choose for a given RAN adapter whether it should run IUCS like we already could choose whether it should be A over IP or SCCP Lite so far and then there's some magic that connects it to the right ports and in the end how does this look in the test case well for all this layer 3 PDUs you don't have any change you just send your DTAP or receive your DTAP from the ports but all the things that are BSS map or RAN app specific you need to differentiate so here is a function that expects a clear on the A interface and originally we just invoked this alt step about the clear command and now basically we say well we have a guard condition if the RAN is JRAN then we call the old one and if the RAN is not JRAN then we call a new alt step that basically handles the respective IU translation and since we have quite a number of these functions it's actually rather nice to abstract that so it's the receive side we could use some function like this like a function that sends a complete layer 3 or an initial UE and depending on the RAN type we either wrap the layer 3 in a BSS AP or in a RAN app message and then in the actual test case we just do a bulk replace wherever we used to call complete layer 3 we call this new function that then dispatches depending on the respective RAN layer and so how do we extend an existing test case to 3G this is now an example of a real test case so we had this location update IMZ reject test case where we basically say well we first create a G-sub-expect and we have this new function complete layer 3 or initial UE about this location update message which we built up here then we reject that the simulated HRR receives the G-sub message we say well no this is not permitted and then we handle it and so on and then this function is what we run inside the connection handler and we have this actual test case and what we have to copy is the test case itself to a new test case where we have some additional parameters that we add basically at the end of the f-start handler function so if you compare here the 3 is the suffix used for the IMZ that okay all the IU test cases get 1000 added so if you ever have an IMZ that ends with 1000 plus something at the end it was a 3G test case it's pure convention RAN IDX2 means basically I'm using the RAN connection number 3 so there's 0 to 2 0 and 1 are likes so far are A interface connections and index 2 is the new IU connection so basically from the test suite we simulate two BSCs and one RNC connected to the same MSC and we can basically send messages here and there and I can also say well first you do something on this BSC and then it moves to another BSC and then you move to IU and you can basically simulate those in parallel it's a bit ugly that you need to basically copy that code but then on the other hand actually I have a slide about this no I don't then we actually want the IU test to show up a separate test case in our JUnitXML and the only way to do this is by having another actual test case object let's say in TTCN3 and you cannot create them dynamically so you have some level of copy pasting and I think copy pasting 5 lines for each test case is acceptable and I put them always next to the A interface test case so they are all grouped together and it's basically you do any changes you see that basically this function is used by that and by that test case microphone would it make sense maybe to separate them into different files that's what I thought but then the point is if you do any change then you have to touch multiple files and here you say basically well this is the test case doing a location after a reject case and anything that relates to the test case is grouped together and I thought if you do multiple files then you don't really you do a change here and you forget that it's also used there but yeah maybe it even makes sense to run them in separate jobs even just to have you could still do that by having a config file that lists only the IU test cases if you want to I mean I'm not saying this is the device let's say the whatever in English this is the I'm not saying it's the best thing since live spread but that's what about the naming of BSC conhandler actually it is this is just something inside the inside the MSC tests so I changed all the infrastructure but I didn't do any renames inside the actual tests itself so all the infrastructure names have been generalized but not the actual test cases so the BSC conhandler is actually a component type that's defined in MSC tests so TTCN and yes one would also could also do a rename there but it cannot be called a ran conhandler because the ran conhandler is already used by the generic infrastructure so you would have to say it's a maybe an access stratum conhandler or a a BSC or RNC conhandler but anyway let's continue about the naming at some other time so that was the easy part and this is all in a branch of mine and I'm merging step by step the stuff into master so we get a new test runs and see if all the modifications broke anything on A and then soon we will have all those additional IEU test cases running at least in my manual execution it looked quite promising and I was quite happy with it so far now for the SGSN that's unfortunately rather different because the A and IUCS are rather similar but GB and IUPs are rather different in the GB interface we don't have SCCP as a transport layer and we don't have a connection abstraction per phone there is no connection notion over the GB interface it's a purely packet based transport there is no connection abstraction so to visualize this this is again the 2G packet switch control plane protocol stacking which by the way the tool for doing this is numeric so I'm drawing these diagrams as a spreadsheet and I forgot to make a box here so if we switch over to the 3G side and we look at the SGSN we see many more pieces have this color background and hence are modified in the transition from 2G to 3G then it was the case for the circuit switch side so suddenly we have to deal with protocols that we never had before so previously the SGSN was all IP UDP and now we have to deal with SCTP with M3 UA with SCCP and then additionally also ran up in there so we have this concept of we didn't have a concept of subscriber connection in 2G and now we have it with 3G IUPS and there's not so much common infrastructure that can be shared in the SGSN tests between the 2G and 3G tests and that's sort of where it stops because it's still working progress I do have some IU tests on the SGSN side already I do have all the the ran emulation which includes the ran up emulation which includes the SCCP M3 UA so basically all the infrastructure is there and the tests now and I can send basically ran up stuff from the test to SGSN and it parses and vice versa but the actual let's say the meat of the test cases is still to be done there but I expect less sharing between the existing code and that in 3G or in packet switched okay that was the state of affairs yeah I think there is not much other than some random links so does Erlang support this aligned packed encoding yes Erlang supports it so somebody could write a libFF transcode that would for example connect to an ASN1 file that runs and you just send all the messages there and you respond back so you have a very small C library that you link into the test suite and you just have this demon doing the transcoding using Erlang interestingly Erlang couldn't immediately parse the ASN1 files that Wireshark ships but I think that's rather a problem with the ASN1 files than Erlang it's such a big dependency so this libFF transcode is available as a Debian package a binary Debian package for Debian 9 and our build slaves and the docker images pull it it's not nice that we have a proprietary component but as I said it's just for the test suite and all the alternatives seem a lot of effort why now is it to attract more users of just for completeness sake do you have too much time on your hands well why now the point is why do we still not have it yet and as I said I already put it aside once after being too frustrated with the problems and it needed to be revisited and why now well we have also some rather large changes happening both in the MSC right now and it looks like we have rather large changes in the STS and in the near future and the only way to ensure that we don't break 3G completely is to have some tests there and I'm not saying that the coverage will be great but at least for all the if we do all the tests in CS that we do on 2G on 3G that already is quite a big gain so yes ideally we would have had it a long time ago and I don't expect that will attract more users but I just expect that it will help us to maintain a a level of quality at all with a break it every day maintain an IU level actually 2 more things I wanted to put this on a slide maybe I put it I forgot it I already solved a couple of problems in our IU code not actually by executing tests or not actually that the new tests have found something but just by creating all this infrastructure I ran into it I ran into it and one of the problems was that the criticality was wrong in some of our information elements and quite a number of messages so if you've ever looked at IU container whether it's the message container the PDU container or the IE container contains a criticality and the spec very clearly specifies which information element in which message type has which criticality and we probably did a lot of copy and pasting when we wrote the message generation routines and they had the wrong criticality at locations and of course since the templates I'm using in TTCN3 are derived from the ASN1 and the template would only match if the criticality is exactly what is specified in the spec and a lot of messages wouldn't match because we just said the criticality wrong and there was one other topic which I forgot but there was some fallout already happened in a positive way okay any more questions no do I say it now or tomorrow tomorrow well that's something that okay then since I think I still have three minutes so as I said I added S1AP the LTE protocol between the E0B and the MME to the Transcode Library and interestingly as opposed to all these previous interfaces that we have here Etsy actually developed a TTCN3 test suite with all kinds of tests like hundreds of test cases for conformity testing of the S1 interface in TTCN3 so I have the hope that with limited effort and I'll see how true that is at some point in I don't know six months to basically glue together this Etsy S1 test suite with the Transcode Library and make it all compile on Titan and then one would have a rather comprehensive set of tests that one could run for example against next APC's MME and I'm sure that will also lead to interesting results and it's really nice that finally for LTE and onwards the test suits are published whereas in all the previous technologies if at all there are some human language documents to say oh you should test like this and you should test like that but not like an actual test suit that you can execute