 So I want to talk to you about the best practices from billions of calls some lessons. I learned from Basically reverse engineering some major VoIP services. I work for and yet I block occasionally at WebRTC hacks So you'll find a lot of country Then it's there for me. So I'm a mechanical engineer. I like to build things But actually I prefer deconstructing them reverse engineering them a lot more so back in May 2014 chat did a Article on Amazon Mayday on a web RTC hacks You can read that there. It basically looked at how Amazon Mayday worked and it was pretty important at that point that we saw that WebRTC was ready for production and it used the WebRTC alt library and Following that I did a deconstruction of hangouts when hangouts started using WebRTC and I looked at Firefox. Hello And at some point I got an email from Serge saying oh, you've done great teardowns and he wanted to look at this Shift to mobile we've seen with WebRTC back late 2014 early 2015 and we wanted to identify some best practices and So I spent a couple of weeks Looking at various services. I wrote 80 pages of reports on that a couple of blog posts lots of notes and some tooling for that and Basically looking at services that have billions of calls WhatsApp FaceTime Facebook Messenger wire Viber and Skype and We learned some lessons about the adaption to the mobile platform about codec tuning About encryption on those heavy loss 2g and 3g networks and about ice So I want to talk about that So chat used wire shark in his deconstruction of Mayday and wire shark is one of the best tools out there for decoding network packets for analyzing stuff and If you know how to use it, it's very powerful. However Sometimes you don't notice things so I was on Mayday used a certain Stun server and that information has been in this dump we had since then and nobody noticed that it uses this net net OS a thing which turned out to be an acne session border controller and Also wire shark as good as it is it is not very good at visualizing things So I'll come back to this graph later and sometimes you need better tools that can generate nice graphs that people love So I looked at things like FaceTime, which is from Apple. It does voice chat and video chat it works on iOS and OS X and No surprise. It's not using we're pretty see However, it is rumored to be big at least that's what people tell me and they have a couple of years of deployment experience so it's been around since 2010 and Before that it was it had AV So lots of experience so they should know what they're doing Well, we found that they're using h.264. They support h.265 However, they don't use h.265 when being on Wi-Fi That's something where you can see that h.265 it is better in terms of bandwidth However, it consumes more battery. So if you have enough bandwidth, don't you use h.265 and And they can negotiate the image size between devices so the sense of optimal resolution doing that with your STP and Of course, they use turn service run by Akamai distributed about around the globe for low latency and We found that they're using a very fast mechanism for the candidate allocation which skips some Roundtrip times and that reduces the call setup time and as we've heard earlier this call setup time the time it takes to Establish the call is very important for user experience So I also looked at what's app which does voice calls and it's pretty big as well So they're not using WebRTC. They're using an old library called PJ zip. They don't do this modern ice stuff They don't use detail us instead. They use the older security description encryption scheme and Looking at the binaries we found that they might use some WebRTC components like the echo canceler However, they don't tell us if they do use those license what requires them so The audio codec has been a mystery for quite some while so he said oh it's opus but I showed him this graph and basically shows how the packet lengths are Distributed and you can see on in black is chrome on app RTC sending opus and What's up? It's definitely looking different So recently someone commented on WebRTC hacks and said oh, yes I did a man in the middle attack on the signal link traffic and it's using opus at a different sampling rate which explains why this looks different and And One of the most interesting things I found was that the call is always relate for the first few seconds Emil told me about that those will be a great idea and I found it to be true for what's that and if They can't switch to peer-to-peer. They go to peer-to-peer to take the load of the turn of the service they have so they're using service. They call a conference bridge 50 something in Ireland and Again, this reduces the call setup time So I also looked at Viber which is doing voice and video chat It's not using WebRTC. It might be using the older global IP solution stuff and It's not using this RTP protocol. So it's completely encrypted. So when sash asked about it I asked him you really want me to look at that because I wasn't sure what I could find in that and It turns out that you can still see all those traffic patterns like this graph which shows that basically is a switch from relay servers to peer-to-peer Which we've seen with what's up before I Also look at Skype and Crypt it as well However, at least they show some debug information so we can see it's using silk white band as an audio codec it uses hd64 video and It was summer in Sweden such had lots of time we did some calls and We were testing on simulated 2g 3g and 4g networks and we found that Skype really dynamically adapts things like the sampling rate The packet frequency and the bitrate to those network conditions. That's very impressive Well, we also found that despite those as us being in Europe. It used to relay service in Redmond Which doesn't make sense to optimize one thing and not the other We also looked at wire which does voice calls text chat and picture sharing Dev an Android app an iOS app and they work in Chrome and Firefox and They use opus and we were interested in how do they adapt opus to the mobile platform So the browser interop is using weapon c. They're using detail as for encryption with very strong encryption mechanisms and they're Partially using weapon c.org But not for networking like Firefox doesn't use it for networking easy. They're using the media engine and their own fixed version of lip opus And you can see that they have tuned the audio codec massively So the bitrate in Chrome black and the app in blue so you can see the bitrate is a lot lower in the app Also, you see that in this packet distribution thing. So it's shifted to the left and When testing on a simulated 3g network, we saw a drop from the usual 50 kilobit per second traffic to 25 and Going to 2g it dropped again to like 18 kilobits per second So this is heavily tuned for a good audio experience and This is very hard to do. However, they know what they're doing. They were involved with the iSEC codec with opus Oh, these are people who know what they're doing really So I also looked at Facebook messenger which is audience video chat 10% of mobile voice over IP and It is pretty apparent that it uses with WebRTC Arc Libraries They have a nice notice file showing that they use it works on Android iOS Chrome Firefox Opera and In the browser chat asked me to look at that back in January and I told him it's boring What do you want me to look at that? It's pretty common. However on mobile. It was completely different They've heavily tuned their app for mobile usage And you can see that in the distribution can see that they send lots of small packets and lots of large packets So adapting to those network conditions they see and They use a lot of different audio codecs and negotiate that through STP For example, they use the iSEC low complexity codec, which is virtually unknown outside Google the iSEC codec when talking to Chrome and Opus tuned for mono when talking to Firefox all handles through STP and For encryption, this is a complicated story. So when talking to browsers, which must implement detail s they use detail s But between their apps they used the older security description mechanism and that allows retractive Decryption, which is a bad thing. However, it is faster than detail s in some conditions So in summary we learned quite a lot of best practices from those services and also things that are not so good You're really interested in all the gory details. All those 60 80 pages go to wherever you see hacks So we saw a lot of adaption to the mobile platform, which is networks with high packet loss and delay devices with constraints on CPU and battery usage and We found that audio is way more important than video and that one-on-one sessions are more important than multi-party and Encryption on heavy loss networks is a problem because if there is packet loss the detail s handshake takes a lot of time However, while you can work around that using s desk, please don't do that We've also seen a lot of codec tuning that is using different codecs lowering the bit rate increase the frame size of the frame send using mono instead of stereo and We've seen some hardware acceleration for example with FaceTime. However, Facebook Messenger uses vp8 on iOS and software one of the interesting things we saw was that with ice a Lot of people go for the relay first So the turn server is not the last resort That's the first thing you go to that reduces the call setup time then they switch to peer-to-peer if they have successful That way you could also hide this detail s latency in the ring latency that is the time until the user picks up the call So, thank you. If you want to follow me on Twitter, I'm Hornsby cornflower Or you go to WebRTC hacks to read about stuff right there Thanks to both Do we have any questions? I think you mentioned the beginning that you need better tools than Wireshark for looking at the stuff. Did you write something on yourself or yeah? It was I was writing stuff based on note p-cap and then generating those graphs with the high charts library It is a small set of tools, but it is very useful other questions Search any suggestions on the next tool Fippo and I should test. Well, we can compare browsers That should be interesting I have a quick question of all the things that you found. What was the most surprising to you? Well, I think see That's the What's that went for this relay service first? I mean Emil had told me about that quite a while ago and when I found it I immediately thought oh, yes, he said that And it was good to see this use in practice Well, FaceTime is different. They use a very complicated architecture a combination of SIP and XMPP and they Establish a data channel and then do a SIP session over that and it must be some very Acquaint reasons to do that right very old architecture How do you know more about that? But you don't want to talk about it you