 Okay, good. Anyway, let's get started. Otherwise, we have schedule issues. So, TTCN3. This is a generic introduction into TTCN3 and Eclipse Titan, which is hard to do in 30 minutes, but still I'll try my best. Afterwards, we will talk a bit about what we do in Osmocom and what are our test suits and what's the state of those test suits. So, why are we doing protocol testing? Well, of course, we're doing this because we want to ensure conformance to some specification. We want to ensure interoperability with other equipment. We want to test for security-related aspects. We want to test for regressions and last but not least also for performance of implementations. And there's no standard methodology, at least not that I'm aware of or language or approach or tool. You can, of course, if it's a symmetric protocol, you can test the protocol against itself. Like let's say with TCP, it's easy to test TCP against itself. Not so much for a lot of the telecom protocols, which are asymmetric. But, well, the problem is, of course, you don't see a lot of the problems because you, let's say, you interpret your NDNs wrong in your implementation and it still works against your own, but it doesn't work and interoperate with anyone else. Testing against Wireshark is also not a good idea because Wireshark more often than not is buggy, particularly when it comes to the kind of protocols that we deal with. So if you think, oh, it's correct because Wireshark decoded it or shows it correctly, always verify that this is actually true, otherwise you can spend hours or days believing that something is correct, but maybe it isn't. Yeah, of course, people do custom implementations of test suits and test protocols. Python is a very popular language in that area. Scapey, for example, is a tool that's used quite a bit originally from like security testing, fuzzing kind of background, but you can implement basically any kind of protocol and protocol stack and that. Also, I think Erlang is a nice fit because you have a good binary encoder decoder and a nice syntax, how to describe binary protocols in there. Then there are tools like PacketRill, I'm not sure who has heard of PacketRill ever here. Nobody, it's I think originally developed by Google and it's used a lot to test Linux kernel. And it works in a way that you can basically, there's a specific output of TCP dump, so it's very much TCP IP oriented, of course, but there's a specific output format of TCP dump and basically you can specify scripts of packets in the same syntax thing. You can literally copy and paste some parts from specific wire, TCP dump output and you can then save that in ASCII form and it will basically regenerate flows based on this description. Yeah, but it's TCP IP, so not very much oriented to other protocols like the ones that we are dealing with. So, well, I don't need to tell you about this, I use these lights in other contexts. I mean, you know that I've been doing telecom related stuff and TTCN3, every so often I read about TTCN, not specific three in some specs and I did some research and basically always discovered, well, it's not interesting because there's only proprietary tools and those are not accessible, so it wasn't possible to do anything with it in free software world. So, yeah, the TTCN3 language is a domain specific programming language and it was just specifically made or created for doing protocol conformance tests. History goes back to the 80s, so it shares something with GSM there and a lot of the other telecom protocols. TTCN3, the third incarnation, exists since about 2000 and it's used apparently quite extensively in the classic telecom sector. So, it's used in, for example, you can find papers where Nokia publishes how they test their SGSN with it or you can find other papers from Ericsson how they test all their core network in elements using TTCN3 tests used that they have created. So, based on conference papers and publications you can see that this is used extensively but not in a very public way. Etsy has specified or has published a couple of so-called abstract test suits, that's a set of tests written in TTCN3 for a number of protocols which many people are not aware. There's a quite comprehensive IPv6 test suit, for example, which you can use to test an IPv6 implementation. The same goes for SIP diameter electronic passports, so the protocols you use with machine readable travel documents, digital mobile radio, of course with LE, not EL. So, DMR has an abstract test suit published in TTCN3 also for SIXLOPAN, for example. So, there are many, as you can see, in different areas, not just telecom electronic passports are clearly not a telecom protocol but there are also some other specification bodies that have tested, sorry, have published test suits so that some auto SAR stuff, some most MQ, this automotive sector or MQTT and co-app and so on. Until 2015, though, only proprietary compilers existed which reflects my previous experience and it was basically out of reach for us. In 2000 already, Ericsson internally developed TTCN3 toolchain, that's compiler and runtime and executer and so on, which they have apparently used quite a bit internally. Of course, not a lot of details are known but some papers have been published. These used to be commercially available proprietary software. They also licensed this to third party, so it was an Ericsson product that you could license as an external entity. It's about 300,000 lines of Java and 1.6 million lines of C++, so it's a quite comprehensive product. In 2015, they released this as open source software under the Eclipse Foundation and the umbrella of the Eclipse Foundation doesn't mean that you need to use an Eclipse IDE to develop with it. It's not just a compiler but this includes lots of documentation and lots of protocol modules which we'll come to in the OsmoCom context as well as so-called test ports. There's also a module for the Eclipse IDE but I think I used it one day or so just to use some of the examples they published. There's the log file viewer and filter and analyzer and a parallel test executor so you can even run distributed tests across many nodes in a network and the Eclipse test executor will take care of starting all the different processes on the different machines and running them and aggregating the results back and so on and so on. Finally, we can use TTCN3 in open source software. How does the workflow look like? First, we have a human developer that writes some code in TTCN3 in this programming language which results in the creation of what they call an ATS, an abstract test suite. The abstract test suite is compiled by the TTCN3 compiler into C++ so you generate from TTCN code you generate C++ code. You can then of course link in some other C or C++ stuff that you have around so if you have maybe some existing implementation for implementing a checksum algorithm or something like that you can just link those functions in and use them and then you have those C++ code that you just use normal GCC or G++ and I'm quite sure you could also do LLVM Clang stuff if you wanted to. It's just C++. And then you get a binary executable in the end that you can execute. So this means it's not an interpreted language. It's not a scripted language. It's a strongly typed compiled language and strongly typed is extremely useful in testing because you don't want a lot of errors only to show up during execution but you want to make sure even before you compile it or while you compile it that it will work and it's so very different from a Python experience in that sense. The terms they use is ATS, the abstract test suite that's the source code and the ETS, the executable test suite but I mean acronyms everywhere but we know that. So what's interesting language features we have a comprehensive type system. What does that mean? We will look into it. We have parametric templates because what do you do in protocol testing? You do encoding of messages and decoding of messages and matching received messages against how those messages should look like. So you have some expectation about what you receive and you can match against that. You have an automatic and comprehensive logging framework that will basically you don't need to explicitly log some stuff. That's very rare actually because at lots of events the test suite will automatically log everything interesting so you don't need to do much there. It has a built-in notion of test cases, test suites, of verdicts as a result of tests and as I said you have this runtime executor that takes care of starting that. So what kind of types do we have? Of course the usual suspects such as integer, float, boolean with an A of course. We have then some more interesting types such as bit strings, octet strings, hex strings, character strings, universal character strings. From those elementary types we then can build constructed types or structured types such as a record or a set. A record is like a C struct basically. A set is any arbitrary order so you can permutate the order of the elements and they will still be equal. You have a record of which is an array and a set of is again the same for the set. The verdict type is quite interesting because it's basically a built-in type that can only get worse and never get better. So you run a test and you say well the verdict is pass or fail or error or whatever and if any part of the code ever set it to fail even if it didn't terminate the test at that point any later code that would say oh verdict is pass is just completely ignored because somebody else has already said fail and each component and they speak of parallel test components which you can think of processes or threads. Each component has their own verdict and all the verdicts get aggregated and of course the worst verdict wins so you always get the result of what was the worst basically from those. Structured types well it's just like structured types in any other language. You can basically have fields what's in so you have a record constructed of three fields here an integer field and character string and a Boolean field. What's interesting here is that you have an optional notation so you can say this can be there or it cannot be there and you can even further specify conditions and so on in which this might be needed in encoders and decoders. You have unions of course so here we have a union from an integer and a character string and the nice part compared to C or C like languages is that you don't have to have an explicit information which of the union members is chosen. So in C you would normally have a union and then a struct encapsulating the union with a field an enum or an integer that tells you which of the union elements have been chosen and here in Titan this is built in so you have a built in function called is chosen so you can match on if is chosen union dot field one and then you have some code handling that so that's built in. Also we have a way to prevent ever using uninitialized fields or uninitialized memory because all the variables or fields in structure types are always assigned unbound values it's a magic value called unbound and if you ever try to send something that has any part of it unbound so you have a structure or record it to some remote side and if there's any field that's unbound then it will crash basically at that point and tell you that you try to send something that is not bound so you can never use uninitialized variables or structure members. If you want in these optional cases if you want to say well this element is not to be sent then you say omit you basically assign the magic value omit to that variable or to that field and that will tell the Titan runtime that it's intentional that you do not want to send this particular element or variable so that's the case. You can subtype so you can say well I have an integer that goes from 1 to 100 and if ever somebody tries to assign a value that's different it will crash. You can have the same for characters and you can combine lists or ranges with discrete values so you can say there's an integer that can be 1 to 5 or 7 or 9 and you can have length of so this is basically you define a record of 0 to 10 numbers of integers like an array, a variable length array which can have 0 to 10 elements and you call this new type reg of int you can basically say I define a character string that must have a carriage return line feed at the end and things like that which is very useful again because any violation of these type definitions would basically create failures either at compile time if it's already known or then at runtime when you go to that come to that point. In templates that's basically the one really strong concept of the language. In templates allow you to in both ways to either send templates and receive templates the receive templates are I think the more obvious of the two so sending a message typically it's quite easy you know all the field values that you put in but when you receive messages then it's sometimes some fields are wild carded because the remote end that sends you the packet is choosing some transaction identifier or some other value that you as the test case you don't know which exact value will be chosen but some other fields you know so let's say in TCP or whatever yeah you know what kind of TCP flags you expect from a packet but you don't know what's the time stamp of the packet maybe for the time stamp you only know a range or you have a wild card or something and using those templates you can basically define such rules what exactly how a packet should look like and then you have a match operation and you say if this packet matches the template then do that or whatever so you can further more extend the templates by arguments so called parametric templates so we can say here for example we have a TR is just a convention for receive templates so template receive that's complete convention only so we can say well it has to be A, B or C and then you can match any character string against that and it will only match in that case you can have something like oh a value that's near Pi but not exactly or something that would fit in one byte because it's 0 to 255 as an integer or you can have like positive integers, negative integers with or without 0 or whatever you may want and here again you have something like you say 0 to 127 or 200 or 255 so you can do that of course also with much more complex types for binary matching or for bit matching we have the bit string bit strings can have any arbitrary number of bits so you can have a 3-bit long bit string or something like that doesn't have to be octet aligned and you can say well I want to match on something that starts with a known pattern but the last 2 bits we don't care and you can express this like that with the character strings you can use some matching even regular expressions at that point and then you have some more interesting capabilities where you can match on permutations of something or a subset or a super set of things or you can have conditions in there and so on which I'm not covering here in this simple example so how does a parametric template look like let's assume we have this record again we had this before on a slide we had this integer character string and boolean in a record and now we define a template we say the template relates to the my message type which we have defined up here and we call it and we have a boolean parameter that we put in here this is the parametric part of the template and we say field 1 it must be present but can have any value that's question marks and as the risk would be be not present or present and have any value and field 2 can be bo or q and field 3 must be the parameter that I pass here into the parametric template and you can of course from that simple example you can create quite complex constructs and and you use this either by explicitly calling the match function that I expressed as you can see soon there's a concept of so called test parts and you can send and receive on such test parts and a receive statement you can directly give a template so the receive statement would ever only return if what it has received matches the template that you have provided so the explicit call to the match function you almost never need it's I think 95 to 99% of the cases you don't need it so yeah you can have hierarchical templates so you can say well I have a template that matches any message type here and then I can say if I have a template for message type 23 that modifies the other template and it inherits all the fields that you have specified here but you can overwrite some fields in your specific template so you can start from a very generic one you can create more specific and more specific templates that you can then use in matching now the next part is of course how do we encode and decode data because all the protocols have whatever kind of encoding decoding some of them are in ASN1 some of them are just human readable specified maybe they are ASCII based they are text based or they are binary protocols but TTCN3 specifies how you can import some formal schema definitions so if you have an ASN1 protocol it's very easy you can just the TTCN3 compiler can just compile the ASN1 files and we generate C++ for that so you don't even need to do anything it's just you have your TTCN3 files and your ASN files and you throw them at the compiler and it will generate code for it nothing to worry about for other formats such or like formal descriptions such as XML schema definition or JSON schema definitions also is the same you can import it rather easily but for all the other protocols like the legacy GSM stuff that we deal with which are all not specified in a formal syntax there is the raw codec that you can use for binary protocols and a text codec that's for text based protocols those two are Titan specific extensions of TTCN3 you will not be able to use them with other tools but since there are no other open source tools we are not really worried about that and the codecs now express how do you get from a concrete binary or on the wire encoding to an abstract representation of this that Titan can deal with and vice versa and the nice part about this is that you do this declaratively so you never like in C or in other languages you would normally write if the first byte is that or that then do whatever blah blah blah so you have an imperative description of how the parser encoder works but with Titan it's a declarative description so what does that mean well if you look at the UDP header definition this may look a bit weird if you think okay well in C I just have one struct that has a couple of unsigned integers so this is a bit more complex but of course this works for even the most complex protocols and not just for something that maps very nicely to a C struct so we first define an integer that is basically 0 to 64k and we define it has a 16 bit field length it has no sign bit and it has a last byte first a byte order in there you can also specify a bit order and all kinds of things so we first define us basically what's a u in 16t in network byte order and then we say well the UDP header has source destination port a length and a sum and we define the field order is MSP which basically means it starts with this field in the struct not from the button up and then we define a UDP packet which has a UDP header in front and an octet string as payload and then we say in the header actually we have the length of header and payload together we store in the length field which is a member of the UDP header so basically you can declaratively express that there is a length field and it covers this part of the packet and both the encoder and decoder will use this so if you ever send a packet you never have to specify the length the encoder will do it and if you receive a packet the decoder will use the length value from there and it will truncate so if let's say the packet is longer than the length value it will truncate and will give you a warning that there's data remaining while you decoding the binary and if you look at the GTP header that's now a more telecom example you can say well okay we have so this bit one and oct two and so on all these types you first have to define but this some normal header files that you just include so you have let's say a checksum present bit RT present I don't remember what this is or a key present bit these are basically bits that indicate the presence of other optional fields here and then you basically say well the checksum field which is defined here has a presence condition of either see some present is one or RT present is one and something like that so again you describe and you declare how your message is formatted and then it will basically create the code for parsing and encoding the data according to the description. I think this is extremely nice I'm surprised that we don't have such features in other languages or tools at least I've never seen anything like that and it's extremely comprehensive because you can even express something like pointers if anyone has ever looked at how SCCP is implemented you actually have relative byte offset pointers to other optional fields in the message you can express that in there and all kinds of other craziness that you can do the same exists for text-based protocols so this is MGCP which of course we have in OsmoCom as we heard yesterday and I can define the different parts of the protocol here this way so I say well transaction ID is a number with decimal digits with lengths of one to nine bytes and I say MGCP endpoint is something that has to have an add character somewhere the version number has two decimals interspersed with a dot in the middle and it has to begin with MGCP at the string so and then I use these elements and I compose basically the MGCP command line from that to say well the field separator between the individual fields can be a space or it can be a tab multiple tabs or multiple spaces or a combination of multiple tabs and spaces and the entire MGCP command line must end with a CRLF or well an inverted CRLF with in opposite order and so on so you express all these things and then you can just throw an MGCP message at it and it will completely decode it in abstract representation program control statements are just like any other language so if you're coming from C or C++ or Java or Python it's not really going to be a surprise you have just the switch statement becomes select for the reason but otherwise you have if, else, for, while, do, while go to labels even go to break and continue now we have these abstract communication operations which mean well we need to exchange data somehow with our implementation under test and TTCN3 introduces a concept of test ports this is not a TCP port or UDP port or a socket or something it can map to one but it's a much more abstract concept you can think of it like pipes or some method by which one part of a system can exchange data with another and you have a send and receive operation the send operation is always non-blocking so there are queues inside that will basically enqueue the message and you can send any literal value constant any variable any specific value template and so on the receive operation is normally a blocking receive operation so it will block until it has received something that matches the template that you have specified and again you can specify of course literal values that's not so but the template is the real really nice part that you can do the problem now is of course well if the receive blocks how can we wait for any end of different event and that's where some interesting program control statements and behavior they call it behavioral statements come in which can do all kinds of interesting things and look at an example here well you say basically if you have a code like this p dot receive x and p dot receive y these are sequential and blocking statements so basically the first statement waits until the whatever matches x appears on the top of the queue of p and then you do receive y but what if the two receive in opposite order then you block indefinitely so that's not very useful so that's why the old statement is introduced so you can basically say this is an example well I'm sending some request to some implementation I'm starting a timer and then I have this old statement which means well either we receive something that matches our response template or we receive anything else or the timeout happens and then I have clauses that specify what happens in these respective cases and this square brackets at the beginning is a guard condition people coming from airline will know what that is basically it enables or disables this particular alternative based on some value which is really nice for state machines you can say well this line only applies if the state is such and such in your state machine and so on now that's nice but still it's rather explicit so you have to write a lot and you always have to start a timer timeout and so on so that's okay first before we go there is one more slide about the repeat statement so basically in the normal else statement we will execute that section and then continue at the bottom so basically the first alternative that has arrived will be executed and then the program flow continues after the entire old block the repeat statement is basically something to prevent that so let's say you have a protocol where you have some requests and responses and at random intervals the other side also sends you a keeper life request and you need to respond to that so we send a request we start the timer we want to receive a certain response but if we receive a keeper life then we send some response or whatever and we repeat which means we start again at the old statement so that happens in between but it doesn't affect our actual test logic that is being performed so you can use repeat for that interleave is now an alternative to different behavioral statement which basically means I have these three events that all should happen but I don't care about what the order is so each of those must happen exactly once but they can happen in any order we don't care and that's what you can express here in asynchronous telecom stuff that's actually fairly common behavior because you know some things need to happen let's say authentication must happen at some point but the spec doesn't say when exactly the authentication has to happen but somewhere in the overall transaction authentication at some point must happen and then you can use the interleave statement for such situations which will guarantee that each of the events has happened once but all of them have happened now as I said it's a bit verbose because if in every small part of the code you have to start a timer and explicitly handle the timer it takes quite a lot of lines of code this is where all steps come in so an old step is sort of part of an old statement that you can abstract out and share and reuse so we define here an old steps like a function definition but it's not a function it's an old step which says well if we receive a ping from the side we send a pong or if the guard timer times out then we set the verdict of the test case to fail with this description so the test fails if the time out is happening so we define this my old step and then the my old step can be used in any other old statement so we say well we send something and if we receive whatever we do whatever handling but we can also execute the my old step and then we don't need to repeat all the lines that we have in this old step in every of our actions in the code so we can abstract that away and it even goes further we have so called default old step activation which means we can basically say at some point in our code we can say well activate this old step ping pong and then this will basically become a background action that will happen all the times in any in any receive statement or any other old statement so if I do something like this then whenever my code at any point receives something that matches TR ping we will send a TR pong in response but we don't have to handle this in all of our code anymore we can focus on the actual logic of the test and all the default stuff is default activation code in TTC and 3 is written in so called modules a bit like python modules they also have import statements the syntax is slightly different and a module consists of definitions that contain a so called control part or other elements such as module parameters and whenever you declare something not as a normal variable but as a module parameter then it automatically sort of appears in your config file for the test so if you want to have a certain value configurable let's say an IP address of your device that you want to test you don't have to manually somehow make that configurable but you just move your variable that holds the IP address in this special module parameter section and then if you execute that sorry if you run the module you can specify a config file in the config file you can overwrite all the variables that have this module parameter part you define your data types, your constants, your templates, you define your communication ports and your actual test components, your functions your all steps, your test cases and all of that together and builds modules and you can import definitions from other modules you can also have private all steps or private functions that cannot be imported into other modules and you have something that's a little bit like inheritance but it's not so it's not an object oriented language but you have something that's a little bit like inheritance so you can basically say some component extends another component and it will basically inherit all the fields and definitions and so on of another module but you can extend them so it's a bit like poor man's inheritance okay real world examples let's quickly look at that I think we are running out of time by far so let's quickly try to look at some code and we talk about the osmo-com side anyway in a short time so let's look at I don't know the bts test suit for example so if we look at the bts test suit we will, well this color is not matching well with the daylight here so I'm switching off syntax highlighting so yeah you see lots of import statements all the different things we import and so on and so on we see this module parameter section that's what that's what I describe so basically we define we have a character string module parameter rslip and the default value is this but you can basically just in the config file specify that and it will overwrite the compile in default so this is how you make parts configurable and so on and so on let's look at an explicit test case so here we have a function that tests unit data indication and we basically have some helper functions that the tune the layer one control in this case we clear the rsl receive queue we transmit some lapdm frame and then we have an old statement which says well if we receive on the rsl side something that matches the rsl receive the unit data indication template with those specific parameters such as channel number link identifier and layer 3 message then we set the verdict to pass because that's actually what we are testing for so this test is basically we have osmo bts in the middle and we attach to the rsl side of osmo bts and we attach to the layer one side of osmo bts by means of using trxcon fake trx on the other side so we basically can behave like a phone towards the bts and we can behave like a bsc towards the bts and we send something on the um side and we expect this to be translated to an abys rsl message on the abys side that's what we do if you receive anything else we repeat so basically we ignore any crap that the pts is sending us but we make sure that we receive that one particular message that we send a message here and we expect to receive it on the other side of the bts and then we call some helper functions to deactivate at the end and that's it so um that's sort of how test cases look like i don't want to spend more time here right now because we have actually yeah um that's i have implemented timeouts yes on a global level so if we we have i have a timer here a guard timer the gt guard as a g is the convention again that i use for global variables um and if we look at the gt guard um there's an alt step for the t guard timer that's one of these well whenever this happens in the background and we activate that in the initialization code somewhere um yeah there's here the handler initialization function that says we'll start the t guard timer and activate that default alt step so basically whenever this guard timer in the background will expire it will call that alt step and that alt step will go back to that it will set the verdict fail and it will say self stop to terminate the component basically so that's what i'm saying i mean you don't need to rewrite that all the time but you just do that once and it will run in the background and you will fail when it when it fails okay now um yeah logging um as i said lots of things get automatically locked you don't need to um write that yourself um so for example any encode and decode operation can log the let's say you're decoding some binary stuff it will log your hex thumb of the binary and will log the abstract decoded value that after the decoding so you can really trace the code and any component sending or receiving something that will be locked automatically um and you even have a log format tool for formatting the data nicely we will look at that in the next example so this is an example log output which is very hard to read but you can see basically here it says send on gtpc to system blah blah blah and gtp1 control unit data so this is from a gtsn test or something like that it will it basically says we have sent this abstract message um and then it will actually say this message was mapped to and this is the binary encoded part of the message that it has generated but this is not nice to read so that's why there is TTCN3 log format which will log it to you like this which is really nice so you get a chasen-like or chasen-similar syntax of the message it's exactly the same log files just post-processing that will show you the message like this and it's much more readable and this is the binary outgoing message that it was mapped to so that's what the log format is doing to make it more human readable now and then the next interesting thing is that we have all the existing protocol encoders so this is a list of I'm not sure if it's complete but it's what I could find all the protocol encoding and decoding and you see lots of telecom related stuff here so bssap plus is actually the gs interface bssgp is ns interface this is a interface we have gtp as used in sgs and gtsn for call control we have llc that's gpsllc we have all the siktran stuff m2pa m2ua and so on and so on m3ua of course we even have stuff like smtp mime google protocol buffers frame relay rtpsrtp ipsec whatever you think of even smpp is there so these are just the encoders and decoders that will basically translate the binary to the abstract encoding and decoding then we also have protocol navigation modules many more but those three are particularly used in the osmo-com context so m3ua sccp and sua and then we have test ports which you can use for example a telnet test port that are always used for vty so all of our test cases they connect to the vty interface of the program that we run and then I have some helper functions to basically say well go to the configure node and change the time slot configuration or something like that so from my test case um yes and no no so it only so s1ap is defined in asn1 so it can compile that yes but it only supports basic encoding rule yes yes yes it's possible just for the s1ap s1ap I think is a packed encoding rule um but Titan only supports BER basic encoding rules but what you do is basically you use Titan to generate the BER and then you use some like asn1c code to decode BER and re-encode as PER but that's it's not it's ugly but it's very simple to do you don't need to write the code for that you have one function called for decode the BER another function called to encode the PER I mean and for most test cases you're not so worried about the performance loss at that point so it can be done and the they have what they call mobile layer 3 is what Ericsson calls it layer 3 includes gsm, umts and lte layer 3 messages encoding and decoding yeah okay further reading so if you want to read some more stuff I really recommend this tutorial it has only how many slides so it's only 250 slides but it's an excellent tutorial I can strongly recommend it and it's even I asked and they now released it under a I think even under eclipse public license I think it's the first presentation slide that I see under this license anyway it's under license now that we can reuse it so the next tutorial or whatever I give I can reuse all the diagrams that they have in there okay good