 Mae hefydいうこと o gynnwys eu worfodd gyda eu gwaneddurol i Wales yma yma oherwydd ein bod gennymai ymwybr yng Ngyriadol, oedd dweud mae'r bwaith o'r arfer Rhysgfaeth Ynwys yma yn 2010. Roedd mor hefyd yn Ringhed Fyra, os yma'n ei gynnwys wedi gyda Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Gwyrdd Chynyddiad. Rwy'n doesi. Aral fan hyn ac sut yn y rai holl o'r cyffredd. Mae'r cyffredd Rych Sid, I'll start by telling you a little bit about me, I don't want to spend too long. I'm a Python programmer from the UK. As John says, I'm involved in the Python UK conference as an occasional speaker, and my most important job there is to organise the conference dinner table plan. I run that, I maintain the table plan software. I'm also an enthusiast, a real enthusiast for Twisted, which is a framework which hopefully you've all heard of. It's one of the oldest Python frameworks. I have become, over the last year, a contributor, a core contributor, I suppose I could call myself a core contributor, and I guess the de facto maintainer of Twisted Names, which is the component of Twisted, which I'm going to be talking to you today about. I'm currently working in Bristol in the UK for a company called Cluster HQ, where we're working on some software to manage the deployment and the state of Docker containers, and it's a really interesting open source project that we're working on called Flocker, so you should look that up if you're interested in any of those technologies. I'm working on that with some of the Twisted founders, which is super exciting for me. OK, well, I haven't got along with this talk, and when I did this in the UK last year, I over underestimated how long it was going to take, and so I've cut out a lot of stuff. This is going to be a much shorter version if anyone saw the talk in the UK. I'm going to talk less about myself, less about the history of Twisted and more about the technology in Twisted Names and also the project that I've been working on recently to implement EDNS and DNS stack support in Twisted. I'm going to start with an overview of DNS. Well, a very short overview. I'll explain why in a minute. I'll give you a tour of the components in Twisted Names. I'll hopefully give you some interesting examples, some quite interesting examples. I'll give you a status report on this project, and then I hope to have some time at the end to answer any questions. I hope there will be some questions. I had planned to give an overview of the domain name system, but I don't think I'm going to have time, and I don't think I need to anyway, because you've probably already been to a talk on Wednesday by Lynn Root, who explained really clearly about the domain name system and its structure, its operation and the terminology, and I think some of the software that you may be familiar with for serving and sending DNS requests. She did a much better job than I probably can at explaining it, so what I'm going to say is you should just go and watch her talk on YouTube. It's a great talk. I didn't make it to the talk. I wish I had been able to, but I watched it last night, and I'm glad she did it, because it means I can get to the interesting bits, and I want to talk about it. So I'm going to skip this, skip this, skip this, skip this, skip this, skip this, and somewhere... I might talk briefly about the software that you may be familiar with, just as a contrast to the twisted name system which I'll explain in a minute. So probably you're all familiar with BIND, which is the original DNS server. I think that's true to say. It's the original DNS server. It's an authoritative DNS server and a recursive DNS server and a forwarding DNS server and all sorts of other gubbins that are mixed in with it, and that's part of its problem. It tries to do too much. It's feature-packed, but it's over-complicated and it's full of vulnerabilities. This idea of one binary to satisfy all the different DNS requirements. DNS server requirements is a mistake, which has been learned and implemented better in another piece of software called Power DNS. So if you're using BIND, then I recommend you go and look at Power DNS, which is a much more modern, much better designed, a much more secure DNS server. And it's actually more powerful than BIND in a way because it has a much cleaner way of interfacing with a database back-end, for example. And it also splits the duty of authoritative server from the duty of recursive DNS server, which is important to avoid cache poisoning attacks. Other servers you may have come across are Unbound and NSD. I mention those because they are written by an organisation that I've been involved with with this project that I'm going to tell you about, NLNet Labs. Again, they are much more modern, much more secure DNS servers dedicated to... Unbound is dedicated to answering recursive requests. NSD is dedicated to answering authoritative requests. Okay, so let's now get to the subject of this talk, Twisted Names. So Twisted Names is kind of as old as Twisted itself. It's celebrating its 13th birthday this year. It probably started life as you may or may not be able to see from this check-in. This is the first change set where Twisted Names was first landed. It was probably introduced as a sort of demo of a new, what was then, new UDP transport facility in Twisted. I did a little bit of digging and found the commits from the beginning of Twisted Names Life and some of the newer commits that I've been working on and Julian's worked on a bit, I see. So you can see it was originally written by a guy called Moshi Zadka back in 2001. And that was in the good old days when Twisted had a kind of Wild West development process. It was committing randomly to trunk and I guess they hadn't yet implemented what is now called the Ultimate Quality Development System which is a talk in its own right. It's a way we develop in Twisted. It's a way of developing in branches and ensuring that every change that gets merged to trunk has been code-reviewed, that it's fully tested, that the code is fully covered and that there's an audit trail showing between the ticket and the code that lands in trunk. But you should go and read about the Ultimate Quality Development System if you haven't already. So Twisted Names was it was kind of actively maintained to start with then over the years it kind of, I think it's true to say it's been neglected a bit and then I started getting involved about two years ago and I had a background in DNS and thought that's part of Twisted where I could help out. And I've been busily updating the documentation adding test coverage adding some new examples demonstrating how to use Twisted Names and you'll find all of those on the new Read the Docs documentation website. I'll link to that in a minute. So like the rest of Twisted the Twisted Names package is really well tested. It's got comprehensive unit tests which are run using a tool called Trial which is a great test runner. The unit tests in Twisted Names if you care to read the code and probably not the most sophisticated unit tests in the world but that's a reflection of the way these testing techniques improve over time. Twisted is, as I said earlier it's over 13 years old. So if you do go and start hacking on it you'll find that there are bits of it which are kind of hard to read, hard to look at without... hard to look at without bursting into tears but then again you have to recognise this history of the project and it's actually quite interesting to see how particular developers who have been with Twisted from the very start have changed their ideas and their approaches to things like testing. I think that would be another interesting talk in its own right. It's interesting from the point of view of a new contributor who has to deal with this old style code and the new style code and understand what's the current best way of doing this developing the Twisted. So we've got plenty of unit tests and we've got reasonable coverage of the code. Some of the modules I don't know whether you can see it on the slide are not very well covered but those are areas which I'm working on now. Modules such as the authoritative DNS server and the secondary DNS server are not particularly well covered. And in fact that's that's sort of really its ugly head lately in a bug that's been discovered in the secondary name server in the latest release of Twisted. It's partly down to a lack of test coverage. This bug wasn't I wasn't noticed earlier and it's partly down to the old style design of that part of the package. But hopefully I and others are going to improve that over time. Twisted names wasn't particularly well documented but that's improving. As I said earlier I've been working quite hard on improving the documentation for Twisted names and Twisted as a whole is better documented these days. You can go like most other projects these days and read the docs on read the docs and it's nicely presented and nicely indexed and easily searchable. So I recommend if you're interested that you go and read the documentation for Twisted names and for the rest of Twisted because it's much easier these days to navigate the documentation. So how am I doing for time? I'm running out of time. I'm running out of time. So let's crack on. We'll have a look now at the different modules in Twisted names. I'm going to start at the lowest level and work up. Like everywhere in Twisted there are layers of abstraction layers upon layers and on layers and we'll start at the bottom and give some examples of how you can use these low level APIs and then later we'll see some of the higher level abstractions. Sorry. So let's start with the Twisted names DNS module. Now this contains protocol level APIs representations of the DNS records representations of DNS messages routines for serialising and deserialising these messages from the wire. And it's also in this module that you'll find the protocol implementations both for UDP and TCP because DNS operates over both of those transports. So we've got a little example here which I'll try and talk you through. Can you see that? Let me try and zoom in a bit. This is Reveal.js and I'm not sure whether it's going to Oh yeah, yeah. Can everyone see that? Great, okay. So what we're looking at here there's a couple of things I need to explain. We have got first of all let's start at the bottom and look at the last line which is task.react. Now if you've used Twisted before you may not have come across this API but this is a new way for you to start the reactor for a short-lived Twisted program and what it does it supplies a method you supply it with a function that you want to run a function which must return a deferred and TaskReact will take that function and supply it with the reactor run your function and then wait for the deferred that it returns to fire and upon firing TaskReact will then tear down the reactor take care of stopping the services in the right order and it will log any errors that haven't been handled on that deferred. So if we then move up to the main method we can see that having supplied the reactor we're instantiating a DNS datagram protocol so this is in this example we're only going to be this DNS client example we're only going to be dealing with UDP and we instantiate the protocol and then we pass that to react to listen UDP on port zero which means any high ephemeral port and because this is UDP we don't have any connection so we have to so it may look odd to be in a client calling a method called listen but we are going to send a UDP datagram and we have to be listening for the response whereas in TCP the operating system would set up the connection for you and you wouldn't have to choose the ephemeral port that the response comes in on so we listen we listen on UDP port XYZ and then we send a query using the protocol we call proto.query our DNS query to in this case the Google DNS servers and then when that query has been answered we're going to print the result and that's by way of adding a callback to the deferred returned by protocol.query again I sometimes think we should start every talk with an introduction to deferreds but I guess everyone's heard it and more people these days are familiar with the idea because it's now part of JavaScript I think so when the answer comes back we simply we take the result and we take it's the result is a message a DNS message which I'll explain it in a little bit more detail later but a message has three attributes has an answer's attribute an authority attribute and an additional attribute and those represent the three categories of records that might be returned by a DNS server and in this case all we're doing is printing out the answers returned by the DNS server and in particular we're returning we're going to print out the payload which is either the A in this case the quad A record it might be the A record or the MX record we're not interested in printing out the header information which wraps around that payload and so I'll show you the show you the output oh I'm running out of time rapidly okay so we've got an answer from the server a quad A one single quad A record now let's quickly move on to the next example so that was a client the next example is a server and in this case it's quite similar but we instantiate the datagram protocol this time with a controller takes care of handling the the query which comes into our server and when a query is received by the protocol on port 10 10,053 our protocol then calls out to the controller and calls its message received method and it's the message received method on the controller which is responsible for for constructing an answer to that message so this is how we write low-level servers low-level DNS servers in Twisted and in this case we're just going to respond with a canned A record with a fixed IP address so hopefully that makes sense I haven't got time I'd like to go into it in more detail but I haven't got time there's the the server running and there's us issuing a request to it using dig okay so now those are low-level APIs if we move up now to twistednames.client this is a much higher level API a much more friendly way of interacting with Twisted names and in this example we're going to introduce a couple of new concepts we're going to use Twisted names to look up concurrently the reverse DNS records for a whole class C network and so you can see in our main method that we are constructing a list of all the IP addresses in a slash 24 network using a really useful module called NetAdder which does all the construction of those reverse DNS names for us I think there was a lightning talk on it yesterday so I recommend that module and for each of those reverse DNS names we're going to call client.lookup pointer and client has a series of these lookup methods one for each type of DNS record that you can receive from a DNS server it doesn't have all of them that we're working on implementing some of the missing ones but it has a lookup method for almost every common DNS type and so we construct a list of deferreds all of them in flight all of them are then added to what's called a deferred list now a deferred list is a really useful API for collecting the responses to a list of deferreds and then it fires its callback when all of the deferreds have themselves fired or failed and so when we handle the results in this example we are looping through the results checking whether the result was a success or a failure and if it was a success we print the again print the payload and we also print a summary summarising how many of the requests were answered successfully and how many of them were not answered either because the record didn't exist or perhaps the query timed out and so the results to that are as follows so you can see it all happens rather quickly and because everything is happening concurrently so that's a real advantage of using Twisted for this sort of work I might skip now to a better example of that one which follows on from Lynn's talk on Wednesday so let me quickly summarise some of the other modules we have in Twisted modules for creating DNS servers I had an example of that and it's really easy to use because there's a Twisted DNS plugin for the Twisted command which comes with Twisted and so you should explore that and explore all the options that you have using that command Twisted itself runs it's twistedmatrix.com that domain is actually served from a Twisted DNS server so it's pretty stable it's not a fully featured DNS server but it's good enough for some cases you can see that when you start the server it logs to standard out and that we can query that server once it's started up we also have an authoritative server and it's interesting but I haven't got time to go into it that you can load a DNS zone based by defining it as a Python module and so here we have a Python module with describing the zone but the objects that you see there are all globals which are imported at the time that this module is evaluated by Twisted names it's quite a clever mechanism but you should look into that too it's an interesting piece of code and again there's the example of how it runs and again these examples are all on the documentation the Twisted documentation site so I recommend you go and read those there's a bunch of other modules which I have to skip through common contains some helpers and some APIs common to all of the Twisted client and servers resolve I won't go into cache is about caching the responses to queries root is about doing recursive DNS resolution which Lyn talked about in her talk secondary is about transferring zones and serving them authoritatively from another authoritative server which some of you may be familiar with and the point I want to make by describing all of these is that all of these building blocks can be put together in interesting ways and I've done a couple of examples of this on the website for example you could create very easily using the low level APIs a module or a script for compliance testing of DNS servers or clients because you have complete control over the flags and the payloads that you put into messages so it's easy to construct non-compliant messages to see how DNS servers respond to those or it's easy to see how clients respond to non-compliant responses from servers so that's a good use of these building blocks we use Twisted at Work for Twisted Names at Work for functional testing so we have a bunch of code which does DNS lookups and we want to in our tests supply canned responses to those DNS lookups and it's very easy using Twisted to set up a lightweight DNS server and then tear it down at the end of the test it would be really easy to set up a database backed DNS server or a DNS server which looked up its data from a REST API for example where it'd be easy to combine these with other components in Twisted like the web module or the LDAP module to look up DNS records from an LDAP database or to control and manage the DNS records in your server using a REST API so now let me see if I can quickly finish off with a quick example a more complicated example of using Twisted Names so in Lynn's talk it was really interesting that there's a tool called DNS map which is a tool for brute forcing a zones for guessing which names may be in a zone a DNS zone and it does that using a dictionary of words or common sub-domains and as you said in your talk it does it in series it's quite slow to complete because there's about a thousand words in its dictionary I thought it'd be interesting to write the same tool in Twisted because it can do all of these lookups concurrently so this is how the original DNS map is documented you pass it a domain name and you can pass it a list of words which it will then look up each of those words as a sub-domain of the supplied parent domain but as you can see it is quite slow so I started this going against Spotify.com and 48 seconds later it had only reached g so it was going to take forever now I want to compare that to another example the example I wrote which I don't know whether I've actually given you the link but I've put all this code on my github page I've got the link at the end of my talk and in this example we are we're actually sending all of our requests concurrently but we're not sending a thousand requests at a time we're using another interesting part of Twisted called the co-operator the co-operator API in the task module and what we can do using that is to limit the concurrency so that we can say there's only ever a hundred in-flight DNS requests at a time so we're not going to overwhelm the DNS server that we're querying I haven't implemented in this the random timeouts that DNS map actually puts in between its requests so it's not quite the same but if I show you the results in this case it's looked up all it's about a thousand sub-domains in two and a half seconds so that's a good demonstration of the power of Twisted and the power of the APIs and the way that it can efficiently process, efficiently send out requests and process the responses all this code is on my github page now I think I've run out of time so I wanted to talk about my project I'd love to talk to you about it I'm going to be sprinting at the end tomorrow at least on Twisted so if any of you are interested in helping out or learning about the development process if any of you are DNS experts and want to help me with my project then I'd love to hear from you it's all about EDNS DNSSEC, I've got funding so you might get paid for it I've made modest progress and I think that's the summary of this talk it's how I wanted this talk to be but I haven't had time to cover it all so those are the links, the documentation that's the github link to the examples in my talk if you want to investigate those I'll put these on github as well and I'll link to these from Twitter or somewhere I'll make them available on the conference website have I got any time for questions no if there's any questions catch up with me afterwards and I'd be delighted to talk to you about it