 My name is Dimitri. I work for Canonical and I work in the foundations team on Ubuntu. And I work at integrating various things into the Ubuntu software stack, mostly in early boot. And I didn't write Resolve D, but I was one of the people who integrated into Ubuntu to be used by default on client servers, desktops, IoT in all of our deployments by default. There are various reasons why we went to do it, mostly because DNS is hard. Doing DNS correctly is very hard and also it's very dynamic lately, such that there's constant updates in terms of name servers that you're supposed to use and search domains that you're supposed to use and so on and so forth. And loads of software doesn't really monitor well when the networking changes. So that's kind of our reasoning for it. So what is SystemD Resolve D? It's an optional component which is shipped in the SystemD software project, which has loads and loads of components. It's a local caching DNS resolver. I think that's the quickest way to describe it. It's a daemon. It listens on board 53 on TCP and UDP and it offers a local resolver which will do pair link DNS name resolution. And that's kind of important because these days you do not have just a single DNS server which you always have access to. You often have different DNS servers depending on whether you're on Wi-Fi, Ethernet, 3G, LTE, whether you're Rome, whether you enable or disable VPN or your VPN dies, such that all of these things constantly change and it constantly monitors and stores configuration for every single of your network links. It also has an NSS module, so you can enable it and then you can have NSS switch resolve all of your DNS names via that as well, so you don't need to have ETC Resolve Conf at all. So like when I worked at ClearLinux, we've tried to actually remove ETC Resolve Conf from the file system that didn't go that well, but we've tried and it was fun when it did work. There is a D-Bus API as well such that you can query the configuration, change the configuration as well as do all of the different queries and resolutions via that. And there is a very handy command line tool as well such that you can actually debug what's happening and get even like the binary output of your queries and so on and so forth. It is a network daemon as I said, so it is listening on port 53 which does conflict sometimes with other softwares that also wants to listen on port 53. It has a Resolve Conf implementation. Some of you might be aware of Resolve Conf which is like on a boon to daemon for example, it's a lot of shell scripts which try to parse loads of domain servers and just mash them together into a single file. So there is a compatibility interface for that as well inside SystemD Resolves. So it's many things together under single umbrella. So as I've said, removing ETC Resolve Conf is a bad idea because in practice it is a operating system ABI. So it has to have name server space something in it. And SystemD Resolves offers you a couple of options of what you can do with ETC Resolve Conf and mostly we're trying to shepherd all the queries into it. There are three files that you can generally point it to. The first one will point at the stubresolval on your local host but it will also have dynamically updated search domains such that as soon as you enable VPN you'll get all of the search domains pushed to you updated in your ETC Resolve Conf such that everything works. There is a static file as well which doesn't have the search domains but it's useful for like embedded systems where you know that you will most likely not have any search domains or you don't care whether they're updated manually then you can statically link to that file instead. And there is a third one which is basically like a dump of the existing upstream name servers that Resolve is aware of such that it will actually have the underlying name servers there. And I'm going to show how these files look like. So this is the static one which is shipped by default in the packaging, right? It points to your local name server. It does say that it does support EDNS such that all of the EDNS queries are capable. There is also pipelining support being added in master as well such that you can do full queries as you would expect over TCP or UDP without losing anything. So this is an example from my system when I'm connected to my company's VPN such that in slash run there is a dynamically generated file and it also includes all the search domains which I've acquired from DHCP from my local Wi-Fi from search domains that I've acquired over the VPN from my company or from my VPNs for my clients as well. And the neat thing here is that when I do these queries each one of those queries is actually rooted on the correct interface to the correct upstream name server such that my internal VPN company queries are not leaked to a different VPN to the second one or to the public internet. And it also means that I always get the correct answers which I actually have roots for. This is the resolve conf which is a dump of actual upstream name service which I am currently using. So you see that I have two of them and I have all the search domains. This will leak your DNS queries, right? Because here you have no information which one of the name servers you should use for which search domain. And you might also get annex domain results from one but not the other and it might not be authoritative which one is the correct annex domain. So however there are pieces of software that really cannot deal with pointing to a local host resolver. So for example Kubernetes they often want to know your actual upstream name server and like from version 111 they started to actually look at this file to figure out which ones they should carbon copy inside their container deployments. Which is probably not a good idea because I don't think they are actually monitoring for updates because this file will be updated as your networking changes. And in the cloud things do change because like whenever you change your tenant ID or you connect to more VPNs inside your cloud your search domains will update dynamically and such that you have to monitor these things. So you can also not point ETC Resolve Conf at Resolve D at all and still use Resolve D else how. So your NSS module will continue to work, your command line tools will continue to work, your D-Bus API will continue to work, network manager will continue to push DNS servers to Resolve D even if your ETC Resolve Conf doesn't point at Resolve D. And one interesting thing is that in that case Resolve D will start to use ETCresolve.conf as an input file such that it will parse it and it will store the name servers you specified there by hand as the global or the default or the fallback DNS server if it has no other knowledge. And it's useful because this way you can revert back your system to what it was before or how it's configured elsewhere such that you can compare the two types of name resolutions. So how do we store and push all the name servers into the Resolve D? So network manager and network D they have native support for Resolve D and they push the updates via D-Bus to there. If you are a VPN provider or you write a new VPN software you might use D-Bus API to update the name servers to Resolve D or you can use the command line tool in like hook scripts to update the write name servers if you have that need. There is also Resolve Conf compatibility interface such that you can just use existing Resolve Conf integration instead and you can by hand specify things in ETC Resolve Conf which will be read by Resolve D if it's not managed by it. So there is loads of options. There is also compile time fallbacks which I don't quite like but you can compile Resolve D such that it will fall back to 8.8.8.8 and even though you have no network connectivity it will like route robin still try to resolve things. I don't like that because that kind of is the privacy violation and like call home and you're not using what you're supposed to use so I'll get back to what the defaults in Ubuntu actually are slightly later. Apart from resolving just normal DNS queries there are other things Resolve D can do. So it can do LLMNR resolutions, MDNS. You can enable and force DNS sec and then you kind of break your system because loads of people claim that they do DNS sec and then they missign their domains and basically you can't resolve anything. Also the captive portals break DNS sec quite often. As I said, this is a local caching named Resolve so there is a cache which your users can poison each other with such that some paranoid people might want to disable cache. So this is kind of like tunables of how much paranoia you have as well. DNS tab listeners so if you don't want it to bind to port 53 because you're going to run some other better software instead then you can disable binding to port 53 as well and you can ask it not to read ETC hosts because you think that's vulnerable as well and then you can disable reading that as well. So it's loads of options. On Ubuntu you see these are no by default because this is what makes Internet work for people in Starbucks and in like Houston Terminal 2 and various angry bug reports that I had to read. In terms of pair link configuration you can see what scopes are enabled on each link so you can say that I do not want LLMNR on my Wi-Fi but I'm okay for that to be done on the Ethernet because I trust physical network more and you can set various settings as you want on each interface and usually it's a full list but it didn't fit on the slide so I split it in two columns so the formatting is slightly modified for presentation. It shows the current DNS server. It shows the search domains and a very weird tilde dot search domain. It basically means that you can force specify where you want the bulk of your search queries to go to which specific interface link you want them to go down. You also can specify which search domains you want to be used on like a VPN connection and you also can say that you do not want anything else but your company dot com resolved there. That allows you, that gives you quite powerful tools to implement split DNS correctly and as expected and most importantly you can do this automatically if your system administrator provisions those things via OpenVPN correctly then all of these things just happen locally on the client machine without users have to worry about whether they're doing split DNS correctly or not. There are the resolve kittle command line. It was previously called system D dash resolve D or resolve but it was renamed to resolve kittle and later releases so it depends which release of Ubuntu you're on you might have to use one or the other but the verbs and commands are mostly available everywhere. You can query normal A records, quadruple A records you can query other things like OpenPGP keys and etc. and behind the very restrictive captive portal and then you like paid for your Wi-Fi and suddenly you get the rest of the internets and at that point you probably want to flash your caches or if you're like a system administrator or like you write GNOME software to do captive portals please do flash caches after you know you have the real internet connection because that way you can actually restore connectivity to loads of things and you can also reset server features. What we found out is that loads of captive portals really do not like EDNS and sometimes they do not give you the answer to where the captive portal is unless you send the query without EDNS but once you got past the captive portal EDNS starts to work magically and suddenly the DNS sex start to work and all of those things start to be passed through correctly and that's quite scary but that also means that sometimes ResolVD tries loads of things and it says like I've tried DNS stack I've tried EDNS, I've tried TCP none of those things work I'm going to use just UDP and like only do short queries and short packets and after you pass the captive portal suddenly you need to scale back up and start enabling all of the nice features like DNS over TLS and all of those things. Here are some more commands that ResolVD will offer you and you can send loads of parameters and well I did mention captive portals and EDNS they're quite sad we found many other bug reports as well so like for example loads of providers let's not name them by name but they run like a big cloud and then they abuse option 15 and they send multiple search domains there and they do it comma separated because magically that works and our clients use it and we're not going to change that so I'm thinking that maybe option 15 will need to have like non-RFC parsing capability like network D and network manager which sounds ugly but it will make the internet's work We've tried to enable M-DNS by default and LMNR but unfortunately what we found out that NX domains resolutions take way too long so people make a typo or people want to resolve to make sure that something is gone from the network because they've removed a hostname and those resolutions take so long that all of the integration test suites started to time out and bomb out such that I had to disable that because otherwise like OpenStack doesn't work or Mass doesn't work or Juju doesn't work so those things are nice on the local network but in practice they break the internet and also the other query that we often get is that domain-less searches are not forwarded so loads of people at home on their Wi-Fi Wi-Fi they do not set at home domain for their local network and then they expect for us to leak search domain-less queries to the public internet and we don't do that and people want that to happen and we tell them no you're wrong but they get annoyed so if you want to resolve hostnames on your local network make up a domain and set it and then we'll pick it up and then we'll use it because that's how you should be doing things when to default we do trust the domains that we've acquired over DHCP by default because that makes things work in cloud context a lot but if you are paranoid or you don't trust that you probably would want to disable that you can enable DNSSEC you can enable DNS over TLS and you can enforce it you will reach very little subset of the internet but good luck and you can do that in your ETC systemdresolve.conf to modify that and use that we listen on port 53 we do not have any fallback DNS so we don't round robin try to reach internet which we don't have access to we do use tabresolve.conf which I showed earlier we disable nss module because loads of software doesn't actually use nss or they prefer to use ETCresolve.conf instead so we kind of use the tabresolver as our default point of how to do queries we have the network manager if up down integration we will switch resolve.conf implementation to resolve this soon but we haven't done that yet in practice this works as a default setting for a distribution well and it covers loads of use cases for some people it doesn't work and in those cases we do recommend well you have the powers to change the same link of ETCresolve.conf to point where you want it to be pointed at but then you also get to keep all the broken pieces because this works well in Starbucks anything else possibly not and I think that's all for my slides and ask me anything yeah do you support DNS over HTTPS? so the question was whether we use DNS over HTTPS what do you support it? so I don't think DNS over HTTPS is not in resolve D yet and we use resolve D by default so the answer is no however HTTPS is probably something that should be added to resolve D project if it does it's nice because then suddenly you can point all of your system and everything on your system will use DNS over HTTPS right so it's already a single point of integration so you need to edit only once not in every single piece of web browser that you use on your system yeah right so the question was what do I do to disable resolve D and how do I avoid NSS taking over and redirecting my queries to resolve D anyway so LibNSS-resolve is a separate package on Ubuntu and it's not installed by default so you would want to remove that package or you would want to also use the package reconfigure and assess whatever the command is to disable that NSS module if you have it enabled we did briefly enable it by default but we have disabled that since so only if you run like a development release you may have it on your system but in practice we don't enable NSS module by default so it shouldn't be there and changing ETC resolve to not be a sim link is quite easy but he knows how to do that so you do need to do two things yeah has the cache poisoning been solved by now I've delegated that question to our security team and in practice they did risk assessment if I can call it that way and basically the benefits of the cache outweigh the probabilities of cache poisoning and it's mostly to do with local users poisoning each other's cache rather than somebody external and also you have to be aware that resolve the internally flushes caches often whenever the network configuration changes or your links or name service changes it does self flush the caches because things have changed my world view might be different so I don't believe cache poisoning is solved but I also don't think it's a viable thread is that an okay answer so the follow up was that because of cache poisoning possibility resolve D was disabled well you have a conflict option for that so disable caches and then resolve D will not cache anything and then you can use resolve D and you will not be vulnerable to cache poisoning but you will also have slower queries because each time the query will be repeated right because like in cloud context like some of our cloud partners they ask us to enable cache DNS cache because it makes the cloud faster especially when you have multiple VMs talking to each other and network there is slightly more trusted so if you can trust your network and your connections to your upstream name server you shouldn't worry much about cache poisoning yeah yeah so that the question was if I want to use a static name server but still use resolve D where do I configure it so in slash ATC system D slash resolve dot conf is the global configuration where you can specify the name servers you want to use but you also need to prevent other things pushing parallel configuration because that can take precedence over whatever your default system name server is so you would disable resolve D plug-in and network manager for example yeah yes I find that a weird assertion because I've run for 10 years okay explain explain so the question was why did we disable DNS by default or for now I think one of the best answers from my team colleague was that we will not use Ubuntu to debug the internet and slightly more wider answer is that we have found that there are captive portals middleware and network equipment which drops DNS stack packets and that even though certain domains are signed correctly between me and client I cannot get all of the answers right that was one issue the other issue was captive portals where DNS stack works but later such that people type in google.com they fail to get to the captive portal they cannot get the internet they disable DNS stack suddenly they can get to the portal and they can get to the internet they can re-enable DNS stack but who does that yeah yeah yeah oh my god this is absolutely client-focused this is the absolute end of the chain and so we generally speak to cheap routers cheap routers are really awful even the better ones have their own domain which they answer themselves and then everything else so it has this weird behavior where depending on the domain you actually resolve, it has completely different feature sets, right? So, and then you have these mixed views on the world where you start GMS-seq verifying a half of the internet and then suddenly you end up in the other half and then everything is closed, right? So, I think on the internet itself, the domains are generally fine, right? Like, of course Google does verify everything with these things, but it's just a nightmare. And, like, it comes... Could you summarize that for the mic? For the mic. I think the answer there is that between end client and the good internet, there's loads of shitty components in between. I think that's the summary. But for me, also, we found loads of bug reports where people change their keys and don't resign their domains correctly, such that you get invalid signatures, which you're not supposed to trust, because people played with DNS-seq, then moved on, or, like, Bob laughed, and then, you know, things are no longer signed correctly. The worst reason to pick it, because people have expired certificates, you don't disable it in the browser because of that. So, I mean, secure with the techniques. So, I think the browser argument is different because it's shown to people in their face and generally people got taught to use that well. And it's quite user-visible, hence it's generally fixed better, as in more websites have valid certificates more often than it is the case with DNS-seq. But if you would see it very fast. Because you just... I'm sorry. He was the next person to ask a question. Yeah. Couldn't it just cache per user? The question about cache poisoning was whether we can cache per user instead. Possibly that's the answer, but it has performance hits. And also, what we also like to see is that when you launch thousands of containers that they generally hit more or less the same CDN endpoints and things like that, because that makes connections faster. So, I think there is a performance hit if you do per user caching. Better than no caching. We prefer cache everything. So, I think we're on that slide scale. Also, I don't think there is per user caching support yet. So, maybe that's something that you might want to add to SystemD. Okay, one last question. But does SystemD resolve the support answering from the cache, even if it's expired? So, in case, for example, the after name service is not available at the moment. So, the question was whether there are answers available from the cache, even if you like lost network connectivity. I don't believe that that's done that way, because as soon as you lose network connectivity, your cache will be flushed and it will be deleted. So, we're actually very protective there that we flush the cache rather than the little because we think like any different configuration changes, everything might change, right? So, and we even have rules that, I don't know if somebody said so ridiculously large TTL, we ignore it. We will flush every entry after 10 minutes or something, the latest, because we say, yeah, we don't want to play this, so we end up in one network and then we're at the capital portal, sets a huge TTL and you never get rid of it. Yeah. So, we have the policy we flushed too often. Yeah. So, the summary there for the mic, is that Resolvedee by Dearfold has aggressive cache expiry and cache flushing. No, sorry, we're done. Oh, we're done. Oh, okay. Thank you.