 Thank you So hello everyone Just before we start I'm really interested who of you came only because of the name of this talk Because I'm doing research. I'm interested whether the name sells or not But yeah, I'm I'm Jan I'm software engineer at NS1 and I would like to talk about Tool that we open source recently. It's called flamethrower and it's DNS performance testing tool This whole started about two years ago We started working on a new like custom DNS server for our company and we needed to test particular features and the existing existing tools like DNS perf like flamethrowers inspired by DNS perf Was didn't didn't get every or didn't contain everything we needed Particular thing that we are interested in was like source port distribution DNS perfuses one single port So when you actually want to test this in some real production setup, you realize that the traffic patterns are completely different then from real clients we also wanted something that has much better TCP support and TCP in general in the DNS port used to be a second-class citizen Like the support is not always perfect in DNS servers. So we needed a tool first To test it to make sure that our server worse worse like well or as we expect we also wanted something that can generate realistic traffic patterns and Since the beginning we wanted to integrate this tool into our CI CD pipeline Because as you are working on a server adding feature, we are interested how the performance changes The flamethrower lives at Github and our DNS work like organization We open source is like early this year at the beginning of January DNS work for you don't for you who don't know the organization is or the arc stands for operations analytic research center is a community of of people from like DNS for operators software vendors Top-level domain Like registrars and so on so we are happy that we can host this this project there and Shortly after we open sourced flamethrower DNS birth move to a DNS arc as well and yesterday also DNS was moved there. So I think flamethrower is in pretty good company So I would like to start with just simple quick start just to give you some idea what this tool does Because later I will talk about some internal architecture and I don't I want you have to some idea how this like works from the user perspective so On our side, there is some sample invocation of the of the utility you are on the binaries called flame and you give it Some parameters. So in this case We are sending some queries to server a NS example of the test and we are saying sending one some same query over and over The record name will be full full example the test and we query for the SOA type the dash queue configures a query rate. So this means 2,000 queries per second and Dash L like limits the execution time. So this will run for five seconds and So so the flying will like first output some some like basic information about what's going on And then it will generate like short stats every second and at the end it will like print out Like the result in like human readable form I run it At the airport before I before I left to this conference. So you can see for this query a lot of surf fails It was probably some DNS firewall in the way Like internally the tool is written in C++ You need relatively new tooling you need C++ 14 or it should work with C++ 13 But it's it's likely that we will soon use something that requires the plus or 17 We use LDNS as a DNS library to construct the packets and We also use UVW, which is really nice wrapper for lip UV The UV is like a sync IO framework It's used for instance in not not resolver. It's used in Node.js It's pretty nice library and UVW provides Like pretty nice interface for C++. So so it's much easier. It uses lambdas it uses It can handle the memory allocation for you So so it's pretty nice. And if you have some time at the end time I might get get into some details about UVW Like internally framework. It's it's not a rocket science. There's only only a couple of components There's some kind of payload generator that prepares the DNS queries that will be sent Then there is a traffic generator that handles the networking so it accrues TCO like UDP sockets TCP sessions sends the queries This closely interacts with with the rate control because the the payload generator can Or the traffic generator can send send the queries in some batches and sometimes you won't just like put limit on on top of that So that's that's what the rate control is there for and there's Yet another component to collect the metrics in format. It is useful for instance for for like machine machine processing So talk about about the individual components the payload generators These these are the ones that we We implemented The study one is the the simplest one that I showed at the beginning in a quick start You just sense the query for a single single name and type Like over File is another pocket generator this actually compatible a DNS pair so for DNS pair if you could like write the text file that has Like record name and record type on each line so we can process the same same format as well there is a generator that Generates just random garbage non valid DNS packets. So this is really really useful It probably will like won't crash your DNS server, but it's useful for instance if you want to test your your monitoring Because what would be realized when we are testing our service that for instance the tooling that we use to collect metrics Doesn't see this kind of queries at all because it's not a query It's it's malformed and the parsers in the metrics collectors are not expected. Just just random junk Then there are like three very similar similar generators that generate valid DNS packets, but with random Domain names so the number Q name just Uses like prep ends a random numeric label as a first Like label for the name Random Q name That's something very similar But it's like generates again random possibly binary query name like DNS specification Says that in the domain name you can have only like letters numbers underscore Whatever but in theory you can put even binary binary in the query names like binary values Like for instance binary zero like most of the DNS software can handle that It's not all about a specification, but it's it's interesting to play with this as well and then random label which Most of the people in DNS know like two years ago the mirror attacks It was essentially this like random random prefix attacks for some domains So we can simulate this with with this payload generator as well So you generate your traffic and this come goes into like the traffic generator which actually handles the networking At a moment the flamethrower can run only in a single track. So So all the like operation on on several sockets are like Driven by the by the IO it's is the leap UV style like asynchronous, but in a single track You are switching the context The max square rate we You should expect on a single core is like 100 K thousand queries per second Of course, it depends whether it's on your laptop or if it's on a high-end server But it should be roughly by a hundred third thousand. So it's not much, but it's enough for the testing and The main configuration options for the generator are like these three You can set up the number of concurrent Traffic generators this like depends whether you like later pick UDP or TCP, but essentially if you create 10 for instance, or I don't know hundred concurrent traffic generators. It means that for UDP Flamethrower will open 100 UDP sockets So you will see the traffic coming for from 100 random ports if you do this for TCP It will go open 100 connections and it will try to keep these connections alive So whenever the server closes the connection it will like re-open and a new one but there should be like 100 connections all the time open and Then you are sending like batches of queries on these either like UDP sockets or TCP sessions or TCP connections so this is configured by the next two arguments the dash Q which is number of queries to send in a batch and Then there is a delay to wait for the next batch So for instance, if you want to send 1,000 queries per second You can have I don't know 10 concurrent generators which gives 100 QPS per per Per the generator and now you can set up the generator to send 10 queries every 100 milliseconds and you get as a result 1,000 QPS There's also the the rate control that I mentioned so this is how you can like cut Cut the the rate of the queries in case the generator is too fast You can either set it constantly for a constant rate or you can write like a QP define like QPS flow So you can always set like The rate you you want to see on on the output and the duration of milliseconds. So This is actually what you see some see on the graph below. You can for instance Crazy crazy speaks. So there is like 2,000 queries per second for six for 60 seconds Then 100 queries per second for 60 seconds repeating so This is from one of our testing servers. There was some nominal traffic for of 600 QPS Like this is the first first first minute and the second minute is it's like back to the 100 and so on and so on Yeah, we don't sample we sample the the the rate at 30 seconds. So it looks Like this but yeah Then the metrics So I've already shown you at the beginning how the the output for for humans look like But if you run flame with dish O It will print out a Jason like the results in Jason into the file So you will have for each second you will have one line of Jason that essentially contains everything you see In the like human for at the end But you will get get the same information for for each second so you can see like the development during during the run of the flame and We for instance feed this into the elastic search and you can later do some visualizations based on based on that I didn't include it in the slides because I didn't find something really interesting But yeah, so just so we have some idea it supports Jason Here are some examples Simple one that uses the the DNS perfect input format I included some some description like how it will What it will end up doing so in the first case you're opening like it's on UDP you're opening 100 source ports or source source sockets and you will send 10 queries each 500 milliseconds Which will end up with 2000 QPS Here the second one is slightly more complicated. It uses The QPS flow and it's it's to simulate some some kind of peak So you start with 10,000 QPS for five seconds then decaying to 5,000 QPS 250 and so on and then there's some like 1000 QPS At event for 40 for 40 seconds So in the end will run one minute, but you will have some some peak of traffic You can use it to simulate cold cash or some failover and Yeah, the last one just shows Sending like random garbage on on TCP With some right like 1000 QPS Yeah, we have some ideas to improve this this to further We definitely want to improve the improve the use of the resources on on the the machine So we want to support multiple CPUs We want to be able to target the target multiple servers or IP range within a single run We wanted to spoof the source source address because this is useful for testing as well We'll probably add DNS over TLS DNS over GTP if someone has a need for that And Yeah, quite a rate is something that we control only with this QPS flow or like static configure statically at the beginning So we were looking into like extracting for instance the rate from from peak up and then just pretending Pretending that the rate is the same and of course like this is this this used to be or we started this To test particle of features. We didn't focus on performance. So maybe maybe that's maybe that's something something we might focus on and Overall overall it like the tool needs some cleanup It's not really a state of the art C++ because we were all learning C++ at a time and people from multiple teams contributed contributed to code So yeah, that's it Thank you and I'll be happy to answer questions and I will also happy to talk to you on a github if you send out send us some PR Any questions for young? Yes, Matthias both Yeah, sorry, sorry, I should repeat a question. I was talking about QPS and you are interested whether it's UDP or TCP Like it can it can really generate The same QPS on TCP as well It can I didn't mention this explicitly, but it can it effectively Does TCP pipelining it opens the connection and tries to keep that connection opened For the whole run. So so you're just then sending the burst of spuckets So that's essentially the pipelining and if the server closes the connection flame will reopen it a new one But the overhead of opening the connection is not that high So you can really generally like thousands of QPS on TCP if the server is able to respond Yeah, can I run flamethrower to find this peak where? Like the server stops being able to respond on on TCP or UDP. No, you can't you have to do it manually But it's it's actually nice nice feature request. Yeah, small feature request I would say do not focus so much on the high-performance stuff because everyone is always focusing on the high-performance stuff And no one here ever runs a name server that does 2 million QPS But we do want to run DNS over T in a TLS really well And we want to run DNS over HTTPS really well, and it turns out that those two protocols have a lot of modes So we can have many connections. We can have connections to keep a life for a long time We have can have connections that come with tickets and that resume We have all kinds of TLS modes with zero RTT and stuff What the world super duper needs right now is a testing engine that can exercise all these modes Because we don't yet know what the browsers are going to do and because we don't know we would love to have a tool that allows Us to explore that landscape. So here my feature request and if you do it, we would love to help. Yeah, that's cool Thank you one comment in one question are really waiting for the To import to pick up real traffic for the testing and the second the question is like in in DNS Perf there was like a maximum number of outstanding queries So where it's rottled until the names are responded again. Do you have like some such similar? No No, there's no way No, it will just Like it tracks the the number like internally we know the number, but there's no function of that Like stop sending queries if there's some amount of outstanding once in flight Any other questions Martin? Thanks for the call. So I'm not so much of an DNS expert, but no no a bit about low testing You may want to consider to print out the presentile instead of the average For your response times because the average usually doesn't tell you anything at all. Yeah. Yeah, just as an idea Yeah, like in this in this JSON format you will see the actual values So if you want to do some post aggregation on top of that you can Not for each request, but you will see current stats for each second and there is a There is there is always a current like query rate and current response rate and the number of for instance time Queries that I'm out during that second So I think you should be able to get a number you want But it's a good point. Thank you. I've lost question over here Well, maybe it's more or less a feature request. I'm sorry for that Did you ever consider not only checking the result code, but also some other expected patterns in the answers? Yeah, of course some bugs only pop up under high pressure Yeah, you're right No, we are checking only for the response codes. We are not checking any other content in the in a packet We could probably do that. I don't know if it's useful