 Now, we will have Daniel again, whom you still remember from earlier today. Daniel is going to talk about the reporting and investigation issues in OsmoCom, which for us as the developers it always seems so clear how people should report issues, but then in reality the issue reports always or mostly lack a lot of important information. And let's try to basically be proactive and try to explain what we as developers would like to see in terms of issue reports. Thanks. Yes, hi. I'm still Daniel. I still have a cold, so bear with me. In general, before we get into specifics, issue report or bug report always should be as complete as possible with all the different environmental factors that may affect it. So in general, you probably all know that. Try to be as complete as possible. This talk will focus on how you can achieve being as complete as possible inside the OsmoCom stack. So first of all, we have a logging infrastructure in all OsmoCom projects. It's in LibOsmoCore and it can be configured in most every project. The exception is OpenGGSN, which is not OsmoGGSN, so it doesn't support that. For logging you have different destinations, so you can log to standard error if you're just running your service and want to see what's happening. You can log into a file, so it stays there a little bit longer. You can also log to SysLog, so you can have some central logging mechanism and pick the logs up from there. Maybe a very short interruption. I don't like to do that. The standard error is also very useful in the context of SystemD, for example. SystemD starts the process and you get the standard error into the regular logging framework of SystemD, and that's actually quite handy. For logging, not general logging, but if you want to debug specific issues, you can also filter by IMSI for OsmoNetB. So where it makes sense, we have filters, so you can filter everything, which means pass everything through. That's sort of misconception that many newcomers fall into. Logging filter all one means don't filter everything out, but filter everything in. I mean, it would be too easy otherwise. You don't have to filter all, but you can also filter by IMSI. So filter IMSI and then the IMSI name will give you only the messages that are associated with a certain subscriber. Of course, on some layers, that doesn't make sense and won't yield you the correct logging messages. There's colorization, so we have different logging categories, which are colored differently. You can turn on colorization, turn it on and off, also when logging to a file, which is quite handy. If you viewed with a text viewer that supports highlighting, you can always just make out the important bits. What do we have? Yellow, I think, is mobility management, or white is SMS level, so that's quite nice to have. There's timestamps, so we support two different timestamps, just second granularity, and we also support extended timestamps, which have, I don't remember if it's micro or nano or femtosecond, but it basically has enough resolution, so you can tie that to some issues that you're seeing that are happening quite fast. For example, if you're looking into wire shark trace, I mean, if the clocks are synchronized between the PC doing the tracing and the one doing the logging, then that becomes quite good. We have different log levels for different logging categories, so you can log into different categories, I already mentioned some earlier, and you have different log levels, so in order of verbosity, the most verbose is debug, then you have info, notice, error, and fatal. Fatal will usually, as it says, then also end the process. Debug can be used for debugging, then the ones in between. On the lower layer, you can see how the config would look like, so Harold already showed and went through it in his first talk. I already said everything. There's one thing, logging print category one, which is a more or less recent addition, not too recent, but it also hasn't been there from the start. It will show you, if you look at some logging lines here, we have the category in easily parsable number format, and if you say logging print category one, then in addition to this rememberable number, you will also have the actual category here. It's probably easier. If you don't have colors, if you have colors, you will probably know after a while which is which anyway, but you can see that's radio signaling layer, RLL, mobility management, and so on. It will print the different categories, followed by the file that the log statement was introduced and followed by the line number. If you want to dig into the code, enable logging, then that will allow you to see where the line came and investigate further. After that, here we have certain primitives to... So, if a log message is associated with a TRX or an L-CHAN logical channel, then we have certain primitives to print info for that as well. So, you can easily see that all these messages actually belong to the same conversation. So, time slot zero, sub-slot zero, which is the first SD-CCH on there. And yes, then the message, activated channel of the AC is returned, established indication, and then from the mobility management at some point after the channel was activated, subscriber is attached. So, that was logging. Now, just a brief introduction into the VTY interface. Which one or who of you has used the VTY interface before? Okay. Has? Has. Has, yes. Okay, it's not all. Okay. So, yes, it's a human interface via Telnet, so you connect through Telnet and then you can do stuff. So, you can configure the service on the fly and you can look at status information, also manipulate some things. It's basically up to whichever program implements the VTY interface, what commands it wants to provide. The nice thing with VTY, as has been said before, is it has auto-completion via tabs. So, if you don't know anything, what you should do, you just hit tab and it'll show you what commands can come next or the question mark will have an annotated help, so it will also show you a brief summary of the commands. Which, basically if you've seen the documentation of the VTY, which is generated automatically, then that's just an interactive way of doing that. And that's where it's generated from. If you think you should, or a comment should be available to you, but it isn't for some reason, then you might need to enter the enable command. So, by default, you're just in user mode and if you enter enable, then you're in privileged mode. The prompt changes and you have access to some more commands that are considered potentially dangerous. And I'll also show a little bit more or show the VTY interface a little bit more in a bit. So, this is basically how it looks if you want for information. So, say show network and it'll show you what type of network it is, encryption on or off the channel load. So, that's useful to see whether all traffic channels are actually in use. If you have problems and if all are in use, then it's quite obvious why you can't open another call. There's also show LCHAN or other commands after show, which obviously show you stuff. So, give you information about the system that you can use if channels are in a broken state or what their state is. And also open channels, if we get measurement reports from phones, then the latest measurement report will show up there as well. So, quite useful to debug a running system. It's just minimal logging example. You log into the VTY. This is Osmonit-based, so it's the port 4242. And then you say logging enable, logging filter all, logging level all debug. And it'll just print lots of messages. And, of course, if you have a specific problem and a specific area, you might not want all categories, but the categories vary also with Osmonit-bay has others than Osmo SGSN. So, you can hit tab as well after logging level and then see what kind of categories you actually want to enable. GSMTAP is a neat way for the BTS or for Osmo BTS to send all the communication that is ongoing on the UM, so on the radio interface, to also send that to the network so we can capture it. So, there's GSMTAP header, which has the information that the actual message is missing. So, what RFCN are we sending on? Up link, down link. What is the GSM frame number? What is the logical channel? The LCHAN and so on and so forth. And then the actual message that was either received by the BTS on the radio interface or sent out. So, enabling GSMTAP, you can basically trace all the RF communication that is ongoing. And in Osmo BTS, you can use GSMTAP minus SAPI 2 or no GSMTAP SAPI to disable it. So, you can enable certain channels selectively. Here it's the broadcast control channel, the access grant channel where the BTS grants access to an MS. The slow associated control channel and the other wasn't slow. It was standalone dedicated control channel, something like that. And, yeah, there again you have the packet channel or traffic channel that you can enable as well, depending on what you want to look at. So, when you have that, you can capture it with TCP dump or just look at it in Wireshark directly for Wireshark. So, a stock Wireshark should work and decode most if not all of the stuff that you're going to encounter. So, you don't need any patches, but you have to have some configuration for it to pick up all the dialects. So, first is GPRSNS. You have to add the port 23,000 there. You can also do that in the graphical user interface, by going through the protocols and adding the things, but that's just... So, this is the easy part. And then for the GSM, ABIS, OML and RSL, so, organization maintenance layer, which configures stuff, and the radio signaling messages, we want to use the IPAXS dialect. Otherwise, some messages are not decoded correctly. Yes, and then with Wireshark, we'll look at it in a second, but with Wireshark, you'll have lots of messages that probably don't interest you. And so, do you use filters to filter out the noise and colorize interesting packets? And I'll show you how that works in a second. The command line is quite nice, quite useful. I don't know where it originally came from if Holger made it up or... I know he showed me. So, you use Wireshark that captures on the interface standard input, and then you spawn a shell and SSH to the actual BTS where you run TCP dump that uses unbuffered mode for standard out, captures on the local interface and writes it to standard out so you can have a remote Wireshark, which is quite nice for live debugging. And then, the last two parts, I forgot to say, GSMTAP sends its messages to UDP port 4729. So, with this command, you would just filter out the GSMTAP traffic. You can, of course, also capture just everything or just capture the abyss part of the communication. Okay, let's see. Basically, the problem you're working around here is if you run all the software inside one embedded device where you don't have a screen and all the communication happens over loopback, how do you get that communication out of the box? And that's basically one possible solution to that problem. So, yeah, just from earlier to show you, this is the colored logs, so I don't have the category set, so you have to just use... Yeah, okay. So you can see here, you have the database logs that fail to find... Oops, okay. Maybe we don't want to show that. So, it's, again, some SIM card that tried to register it wasn't... It's not known to my HLR, so we rejected it. Okay. Next. Okay, so this is... Yes, so fun fact about Wireshark. You can use Ctrl plus and Ctrl minus to change the size, which is quite nice. So, this was the trace I recorded two days ago. Basically, this is just mTab, just one packet with just mTab header where we can see that in time slot 7. That's where I configured the packet channel. It's not an uplink packet, it's a downlink packet. Uplink is zero, which means it's downlink. In that RFCN, we don't have any signal level or signal to noise ratio recorded here because it's downlink and it doesn't know. It's the GSM frame number and I'm just going to resize it so I actually see everything here. So, basically, you can take a UM protocol trace without an actual air interface sniffer. You could, of course, use something like GPS decode or Air Probe or something like that, but if you run the BTS yourself, why would you intercept the radio interface again if you can just have the BTS write you all those frames that it receives and transmits? Yes. So, yeah, with different filters, you can filter out different stuff. So, I just want to look at the GSM tab, which is all I'm seeing right now anyway, but I don't want to have any of the packet channels. So, you can also just right-click and say prepare or apply as filter and not select it. So, I want GSM or GSM packets and not no packet of type PACCH, which then leads me to some... Let's close the GSM tab here. This is a system information type, so this is just a regular system information type that the BTS broadcasts about its capabilities. And here we have LAPDM from a phone which sends a detach indication and class mark change, GPS, suspension request. What we can now also do is if we want to follow this Timsy... Let's take this one. This one might be alive a bit longer. So, we have the location updating except. We have this Timsy here, and we want to basically follow this around or just highlight it so you can colorize it with a filter. So, now this packet is green and the next time it detaches just down here, you can directly see that those are related to the same MC. Of course, once you have a periodic location update, the Timsy will change and it's not useful anymore, but for some things, that's quite nice, quite useful. Also, to follow around, you can filter on the channel type and the sub-slot in order to just have one... or follow the messages from one L-chan around. There's also this colorization... coloring rules in Wireshark that you can use. You can see up here for 3G, where you have some rules that differentiate the circuit switched and the packet switched traffic. And while the other colorized with filter don't stay permanently, if you save this here, then you will have it after restart as well. Just for completeness... Wireshark also has auto-completion, so if you're not sure, if you don't know where the protocol is, try to prefix GSM and then see if it matches, but then it will also tell you what kind of protocols are available. So on here is basically the trace of the ABIS interface. So we had the air interface and this is the ABIS interface between the BTS and the BSC, where interestingly the coloring rules still matches. There's location update, except... and lots of load indications. Again, we can filter those out as well, but I think I'm running out of time. So apply as filter and not select it. So GSM ABIS RSL, but no load. So here's the detach indication, which basically is the ABIS part of the RF traffic that you have previously seen. So that was the short demo. Then I think Niels is going to talk much more about this in his infrastructure talk, but just for completeness, once you've found a bug, you want to report it. Try to be specific in your report, be detailed. If you know how to reproduce it, those are always the most valuable bugs because you can actually debug it all the time and don't have to wait for the issue to appear. Then again, it's not always easy to find the steps to reproduce it. Attach all of the logs that I showed you how to make, the Pcap traces, everything you have about the bug, attach it to the issue. For logs or other stuff, please provide screen dumps so all the information in text format, not in picture format, so don't do any screenshots. I mean, we can read it, yes, but it's much less pain to have it in text, and it's faster as well. Search in text. Yes, search. Then also the last thing is be specific with regard to the version. The VTY has a command, show version, which should show the current Git commit that it was built from. Then again, if you tested yourself and you built yourself, please provide the exact version that you tested with. Or for open embedded installs, you can also do OPKG list minus installed and then grab for all the Osmo packages or just attach the whole file so we can see what libraries were used, what exact versions. The open embedded images sometimes also have, it's not nice, but it's happening. They also have patches attached or patches directly applied onto a Git source, so OPKG list installed might even be the more accurate version for whatever. Okay, so yes, that's the URL you're probably all familiar with. You want to report bugs? Do you want to report any questions? Stupid questions are always good. There are no stupid questions. That's what I was talking about. Let's hope it's right. Can this really fun protocol tracing thing with Wireshark be done with any other Osmo-com projects other than Osmo-BTS? I think that was the... What do you mean other? The non-GSM based Osmo? Sorry, so you're getting this wonderful trace of what's happening on the radio interface. Can that be done on, say, the interface that you would see like the ABIS over IP or any of that? I'm just curious. I never use it anywhere else. ABIS over IP is what you're seeing right now. Isn't that the radio stuff? ABIS is communicating... Well, it's everything in a box. It's communicating from here to here. But it's basically the BTS talking to the BSC or in our part the Osmo-Nitby. And A would be the part from the BSC talking to the MSC. We have Wireshark as dissectors for that as well. So this is ABIS. The IP addresses are both local hosts because it's both running on the same thing. But you would see this trace... I took the question for a stupid. Basically, you can look at the same messages on the radio interface using GSM tab. That, of course, only works if you have Osmo-BTS. So if you use IP access or Ericsson, you don't have that. And then you have the traffic on the ABIS side. And then now with the upcoming A interface between Osmo-BSC and MSC, you can look at the traffic again on the A interface. And it's again going to be the same layer 3 message. So you see it basically appear at every step as it traverses. So right here, without all the filters, you have the... You can see GSM tab message for the detach indication, which then is being... So this is just a UDP port. 4729 with the GSM tab header. And here you can see it's... In TCP it has the IPA access header using RSL and you have the detach indication in here. So that is ABIS RSL. Yeah, and if you look at the timestamps, you might even see... Well, I'm not quite sure if you... You can see how long the processing of the packet took place between the UAM interface and the RSL site inside Osmo-BTS. Or is it? Yeah, actually, they are different, right? So you can see how long it took to process that. So I don't know if I do something wrong here, or if it's intended. It's within the enable logging, logging all one logging level, globally blog, blah, blah, blah. Type it a million times. It seems that if I want to, say, see mobile management logging levels at debug level, then I also need to type logging level all everything. Otherwise, if I set the global logging level to something below the specific logging level, then I don't see the logging level for the specific facility that I have asked to see debug for. So in a way, you have to turn off everything other than what you want to see if you want to use anything other than logging all, logging level all debug. And I don't know if... I think it's a good point because it's confusing even to me and others that system have come. It's one of the things we've just learned to live with. So logging level all, all is not really a category, but it's more or less, I think it functions as a filter. So if you say logging level all notice, then no matter what category you have set to anything, it will only print notice and above. And so what you want to do is you want to set logging level all debug, and then you probably want to set logging level MM, or DMM, or whatever the category is called, to debug as well, because the debug category might actually not be set, might just be set to notice or to info. And logging level all doesn't change that. It just changes that whatever is sent inside these different categories, whether it actually is forwarded to the log or not. Specific thing for me is that I... I'm always having these annoying messages from the SNPP ping. Are you there, message? So it's causing the terminal to scroll constantly, to turn them off. But if I want to turn that off, I actually need to turn everything off, because you can't... It would be nice if there was a command. You can set logging level SNPP fatal, and that would turn that off. Yes, but it doesn't unless I do logging level all everything. If I put logging... If you want to just log one facility, you have to turn everything else off. I don't. Yes, that's what that was. But then you have to... I use expect scripts at this point to just configure it every time if I have different... I want this level, I just do it. So you're basically saying we should have a script language in the VTY so you can have your customized logging configuration snippet that's stored on the server? Sorry. It is a bit confusing, but I think the logging level all... Probably you should have it on everything all the time. I think very few people would ever want to do anything else, but have logging level all and everything. And then you only have the individual categories, the individual subsystems, and the logging level all is basically... You just ignore it. So if you have a proposal how to change the naming or the functionality, I think... Yeah, indeed. And I think Max did quite some work on it back there. And in the end, I think everyone was confused. Not because of Max's work. Anyway, so we're running out of time so we need to switch over to Nils now. On the recording.