 I don't know if they can go their life. Oh, hello everyone. Welcome to the evening session of day one of the device engineering conference. Today, we have a single closing keynote by Paul Vixie. If you don't know who Paul Vixie is, I think you shouldn't be here. So Paul is basically the person who practically did the bind protocol and all the tools around DNS. He's going to give you his opinion about network operations and whether it is relevant in the age of cloud. And he has pretty strong opinions and I asked him to hold back nothing about whatever he wants to talk about it. So over to you Paul. Thank you. So good evening to those of you in India standard time. Speaking to you from Pacific standard time, where is about seven o'clock in the morning. And I am thankful to be included in your conference. I do want to correct a couple of things. Yes, I did have a lot to do with bind bind is named server software. And it originally came as part of the BSD Unix operating system. And that was from the University of California at Berkeley. I took over the maintenance of bind in 1989 when essentially all of the students who had written it graduated and had real jobs. But of course the internet needed DNS software. I was at that time working for a computer maker called digital equipment corporation. They made Vax as you may have heard of them. And so that's how I got started with it. So bind was a couple of years old by the time I took it over. And they did rewrite it and then I hired a team who rewrote it again. So I probably have had a lot to say about that. But really what that has given me is an insight into DNS, the domain name system. And I realized that I'm here today to talk about network operations, but I'm going to use some examples that come from network technology in general. And then I will also cover what I think this means for net ops. Net ops itself is a artificial term. The idea that you would use camel case where you capitalize the N in net and the O in ops and then slam them together as if they were a word. That really came from the agile community, that idea that you were going to camel case things and make up your own words. It's unfortunate in some ways because some of the techniques and technologies that are popular today have been popular for a long time they just didn't have the names that they have today. And it was not always possible to buy a book or hire a consultant or go to a training that would use those words. And that makes it seem sometimes like innovation is occurring when label engineering is what's what's occurring. So, so let me be sure my slides only have really one slide so we're going to be staring at this for a while let's get started. Alright, so we are in the age of cloud. That's inarguable. Very few new data centers are going to be crafted for small to medium sized companies that it doesn't make economic sense to figure out the power and the cooling and the connectivity and so forth. So we are, let's say a company of 20 employees. What it will make sense to do is to buy VPS virtual systems, virtual operating systems, a Linux instance of VST instance of Windows instance. It's possible I suppose that somebody will decide they'd like to have a server in their house in their business. But even in that case they're probably installing some operating system like the Mware or Windows or even Linux in some cases. And using that can that server to create these the same virtualization agent. The so called guest operating system. Why do we do it that way, because to the best of the knowledge of the people in the field now, we've always done it that way. That is the that's just the current wisdom that is the word on the street is if you want to do this kind of thing this is how you do it. So I gotta say we didn't always do it that way. It used to be that for one thing virtualization is a somewhat modern invention is about 20 years old really in any in a real sense. But there was a time when Amazon sold books. They were originally known as Earth's biggest bookstore and Microsoft sold windows. And the Google that didn't exist at that point. So in those days, what those companies had to do was build data centers and fill them with computers and then install whatever operating systems they needed, build their software tested and all the rest. And clearly that worked, or we would not have grown as an industry and as an internet to the point where there were some pressures of scale. And, you know, ultimately, the way that the, the legend goes, Amazon built AWS for itself, and then said, gee, this is also going to be a good separate product. In other words, what they needed was a way to have a lot of data centers with a lot of computers, but not have an army of people, tending them, and to have a lot of human error creep in the needed to be elastic in its scale. And so once you've solved that problem and Amazon ultimately is a logistics company. They did they solve that problem I solved it well. As far as I can tell they began with the Zen XEN virtualization platform which they have since obviously made their own. But if you solve that problem it's almost a waste, if you only use it for yourself and that's really where the age of cloud began was because we had pressures of scale. You know, we can be thankful for that. We have a lot more work getting done a lot more value being created, a lot more value being exchanged because of let's say the internet. And we would have, if let's imagine everybody was still running Apple Talk, which would really not be able to work at scale outside of the house or the business that was using Apple Talk, or Novel or any of the, the other protocol suites that were not the internet. We needed the internet so that we could live this future, and we needed the cloud so that we could live this future. So that sounds rosy that sounds like an unalloyed good. But I think almost all of you listening to me know that there is really no such thing as an unalloyed good there's always a price to pay if you want to get some benefit. No such thing as a free lunch, I guess is what they would say here in the US. Let's talk about food. Let's use that as an example I really want to illustrate this, so that everybody knows not only where we are but why we are here and why it's inevitable that we had to be here. So 150 years ago, most people in the world certainly those here in the US knew where their food came from. That didn't mean everybody was growing their own food, although a vastly larger number of people 150 years ago were creating a large part of their own food. But it meant that you knew the person or the farm or the whatever business where your food was made or if you didn't you knew how they were doing it. It's just general knowledge. And so, you know, if you see any of these science fiction dystopias, like Mad Max, Fury Road, wonderful movie I loved it in that situation, where all of a sudden civilization collapses, you no longer need the benefits of scale but you no longer have them. But during the time that you had them, you've created a generation who no longer knows where their food was coming from or how to make it for themselves. They didn't know how it was made they didn't know what to do. And that is not far from the truth I realized that it's dystopic fiction, in the case of Mad Max. But it's also a real thing. In the 1970s up until 1980. Here in the United States, we didn't really have a diabetes problem. It wasn't a epidemic in the way that it is now. And you can see, if you look at various charts and diagrams you can find on the web. But in 1980 something changed that that is when the reported cases of diabetes started to rise, and it rose every year, and has. It's a big problem now. What happened. What happened in 1980. But the US government changed its recommended daily food allowance for what foods you should eat the so called food pyramid. How much in grains how much in meat how much in fruit. And believe it or not people listened I know it's not very common for culture to listen to the governments that it creates. But they did in this case, and it turns out that that particular set of recommendations is very heavy on the sugars, the indirect sugars the carbs carbohydrates. And so now we have this diabetes problem. I don't think that we could have turned that corner. If people knew where their food was coming from. But by that time it had been over 100 years since the average America knew what what food was how it was made, how it should be made. What you should eat that used to become a knowledge now it is very rare. And so what I'm trying to paint here is the cost. And it is said that under capitalism, half the world starves and the other half has diabetes. So half the world is getting way too little in calories and the other half is getting way too much. That does not mean I am opposed to capitalism, I am a capitalist. And note that capitalism is is a magic and all magic comes at a cost. So yes, under capitalism, half the world starves the other half has diabetes, but without capitalism we don't start. We could not have the world population that we have creating the value that they create. Everybody also had to tend their own farm and grow their own food. We needed both the specialization of labor and the ability to invest capital to create a system. It's, you know, it's remarkable what we let ourselves into under the pressures of scale. And we don't have a lot of food security here in the US right now. We're utterly wedded to the transport chain. Most metropolitan areas are 72 hours away from riots, if something happens to that metropolitan areas ability to import food from the parts of the world that are still making it. This should concern us, especially in this time of climate change, political instability. It's made that the dependency chain is very long and very fragile and very secret. It's not deliberately secret. It's just for one thing impossible to know everything you need to know the details are too numerous. The details would only apply to you if you were a gigantic corporation, a corporate person knows how to make food. Real persons generally do not realize that in India you still have a lot of small farmers and a lot of parts of the of the country are doing better than what I'm described describing here. That's one of the American perspective here. So, let me move out of food and out of economics and the pressures of scale and pull some examples from what turns out to be my recent experience. I wanted to build a Wi-Fi mesh network for a farm, actually, one of my cousins, and I volunteered to do this, and it did not seem like it was going to be all that difficult. I looked around, I found the right hardware vendors, I found a possible hardware vendor, did some research, bought some equipment, plugged it in, played with it. You know, configured it kind of on the workbench. Well, everything looked good. It took me a while, took me a day or two, but you know, I figured it out. Then I went to the farm in question and deployed it and things were going well. Right, so I put a mesh radio very close to the building that had the internet connection. And then I added another building far away using a Wi-Fi mesh radio, and it worked. And it was fantastic, right, because this farm is not used to having good internet connectivity. We've got very much a benefit of scale, which is that if you're in a metro area, then it's profitable to build a commercial internet service provider that will serve a lot of houses, but that's because there's density. Without density, you know, we had to have a government program to even get electricity to happen outside of the cities. We called it the Rural Electrification Program, and I'm glad that we do things like that sometimes, but nothing like that has happened for the internet. And so this little Wi-Fi network with its business-grade connection was looking good. It was very popular. So I said, great, that's working. That means my fundamental assumptions about how all this is going to work are not wrong, so let's build out the rest. And something strange happened. I added another mesh radio that was actually a little closer to the base station than the first one. And it did not come up. The mesh did not automatically accommodate this new radio. And worse than that, my first connection went down. I no longer have the first working building. I went from zero to one. And then when I tried to go to two, I went back to zero. And I got to tell you, I struggled with this because these things don't have whatever our serial consoles that you can plug into and ask it, what are you seeing? What are you doing? Why are you doing it? No, it's not like that at all. There's a GUI, Graphical User Interface. And it's ultimately all of the stuff is based on the Viata. This company's products and all of their competitors are based on the Viata system. So Viata has done some good here. So ultimately, it is a crafted version of Linux and yes, it has an SSH listener and you can connect to it if it's on the network. But if it's not, there's no way to connect to it. So you're left with this problem where you don't know how to get started. It's not on the network and you don't know why. And you can't find out until after it's on the network. So yeah, this was a head scratcher for me. And I eventually found that the way that the mesh protocol in this product seems to work and I don't know anybody at this company so I can't really ask them. But the way it seems to work is if somebody's trying to transmit an Ethernet packet, and it's to an address that you have not seen. So you have no local forwarding database entry that would say, oh, I know where that that MAC address is. So I'm going to use that interface. If you don't have that, and you're supposed to be kind of, I don't know, sending it out as a broadcast so that whoever has that that address can answer and say yeah, I am that person. That only works if you are headed in in the direction toward the mesh you have joined. So that just means that if you let the mesh automatically form, then it will be ruthless. There will not be some place to that has a chance to say by the way that's me, unless you're downstream from it. But what happened, and this is borne out by experimental evidence although I've not looked at the source code is that my, my preferred mesh exit point the thing that had the wired connection to the, to the internet was joining a mesh. And therefore, it was willing to transmit packets upstream toward the mesh it had joined but that mesh was not willing to transmit packets downstream toward this outside connection. So I laboriously took every one of these radios off of its building, because they were all by that time you know up on the roof and so forth took every one of them. I plugged them back in and reconfigured them to say look, I don't want you to automate your mesh I want you to choose exactly who you could potentially talk to. And so I made my own topology using static configuration I just turned off all the automation, and it worked perfectly. So what does that mean. It means that if you weren't building networks before the age of cloud, and you buy this product, hoping that you're going to be able to, let's say build the, the network you need and the one they're advertising as being available. It might be very frustrated, and you're not going to get this solved with the documentation you're not going to get this solved with tech support. And you don't have either the knowledge, or the serial consoles that it would take to sort of diagnose this. And I was very much kind of shooting in the dark, hoping to hit something and I eventually guessed right, but I had to know what to guess, which meant I had to know how the ARP protocol works ARP address resolution protocol. I had to know sort of what is an Ethernet broadcast packet what is an Ethernet multicast packet. And I don't think most of the customers who buy this equipment have any clue about those things. And if you don't know, you don't know what the opportunities were what roads were not taken by your manufacturer, and you're just a prisoner of their you might end up as a very dissatisfied customer returning all the equipment that you bought, because it doesn't automatically do everything that it's supposed to automatically do. So, you know, in that sense it was lucky that I have been using computers since before Ethernet was available. And so I know what alternatives existed for Ethernet itself, let alone all of the, the technologies that have come sense. I'm going to come back to Ethernet in a moment, but I also want to talk a little bit about IPv6. I'm going to use the same mesh network as my example. When the Internet connection was installed. The ISP said by the way, here's your address is a business grade connection so they were not going to use DHCP toward me is it here's your address here's the subnet. Here's our address. You can set your default route to point at our address and everything will work fine. And it did. And that was all great. But after we got this whole thing running and I said, geez, since you also delivered IPv6 and since, you know, there may be a need for that someday. I'm going to also get that working. And it didn't work. Luckily, by this time my little viata based router was on network I was able to SSH into it so that I could find out you know what's what's going on here. And I think all of you are familiar with the IP command I'm not a Linux lover I'm still learning this I'm, I'm still using BSD for most of my work but this is how the rest of the world works and I got to learn it and I'm not resentful. Anyway, I looked around and I found my, my router didn't have a default route. So there's a break up for v6. So that was a problem. But I thought, you know, before I go there and let me make sure that I can at least do IPv6 neighbor discovery on the next top that I was given I was told where to point my default route. I'm not doing that shall we. No answer. Okay, well let's mess around a little longer and see. Do we have an IPv6 neighbor discovery result which is what they call ARP for IPv6 you send a broadcast packet containing the v6 address that you'd like to talk to and the person who has that address answers you and says, I have that address. Here is my Ethernet MAC address if you wanted to send me packet you wanted to send packets to that IPv6 address. Here's your Ethernet address to do that with, but that failed also that that was not working. And so I reported this duly to the ISP and I said, hey, you know v6 is not working. And they said, oh it didn't speak these words but the thing that they said boiled down to. We don't really work at the level of typing commands at routers anymore. We have a GUI, I am your installation engineer, and it all looks good here. Ah, okay. And so I sent, you know, some results I cut I did what we all do. I copy pasted, and I said and this is what I'm trying. This is what's happening this is what I'm getting. This is the state of my router. And, you know, and in the next round that particular engineer who by the way is brilliant, but he's shackled by the fact that he can't touch his own router is stuck with his provisioning GUI. And he said, well, you know, I really think that you should. Don't try and ping us we might not answer we might have blocked ICMP. So, just try to reach the open internet, everything will be fine. I had to explain to him. I'm going to work because I don't. Since I can't ping, or otherwise reach there's no IPv6 neighbor discovery or IPv6 ARP if you prefer. There's no result of that in my router so it will not be able to reach anything until it knows what Ethernet address to put on the packets that it wants to send to the internet through you and your router. And that was a head scratcher for and I gave him the RFC number for for IPv6 neighbor discovery. I gave him the new one with the old one hasn't changed much. And he appreciated that he said, you know, I will research this he could, he could tell that we didn't know enough he and I together didn't know enough. And I'm lucky in that in his case that he actually understood the results of the commands I was running a generation from now he will not his successor. If his if any of his kids go into the same business that he's in, they will not know what the IP command does. And so I'm lucky that I'm kind of on this cusp of generations. This should also concern us the same way that food security concerns us. So I promised I'd come back to Ethernet. And there are many stories I could tell but I'm going to stick to just one which is the MTU, the maximum transmission unit. This is the size of the largest packet that you can carry on some network link. Now we often incorporate this concept into broader ideas. So we have something now called PM to you, which is the path into you. And what that is is for a given destination. There's a set of networks that you have to traverse between where you are and where they are. And one of them is going to have an MTU a link MTU that is the smallest one. And in practical terms they are all 1500 because that's what Ethernet offers and everything kind of looks like Ethernet even if it isn't. It's just so convenient to to pretend everything is an Ethernet that we've now got all kinds of things that that use this same concept. But 1500 is the MTU you're going to see. So let me set the way back machine for the early 90s when the ATM protocol was being developed. This is back when the Internet was still in a fight. There was still a protocol suite called OSI that had its own version, its own equivalent of TCP was called TP4. And there were a lot of European telephone companies who've made a lot of investment. In fact, my employer Digital Equipment Corporation spent, you know, probably billion dollars on really making a really nice OSI protocol suite. Because they thought that that was going to be the future of communications. They should have spent that money on Internet. They might still exist today if they had spent that money on Internet. So they guessed wrong. But anyway, the ATM people were part of the part of the OSI crowd. Yeah, so there were a lot of telephone company people, a lot of people from the ITU, the International Telegraph Union were saying yes to what we need. It wasn't what we needed. And you're not going to find very many ATM links today, other than hidden behind DSL where it kind of still exists. So what we ended up with is Packet on Sonnet as our common link. But one of the reasons that ATM failed in the marketplace was because of their cell size. Everything had to fit in a 53-byte cell. And every one of those had a five-byte header to tell, you know, the receiver of a given cell where it was going. And so you had all this complexity in the network where you were trying to. So every endpoint had to ask its local router essentially to make a phone call. You know, please build me a virtual circuit to a certain place because I want to send ATM cells. And I need everything between me and it to recognize and properly route those cells. But, you know, it's worked on the whiteboard. But at OC3C speeds, which was 155 megabits, which I know seems slow to you today listening to me, but trust me, in 1994, this was, that was a lot of bits, 155 megabits. And the problem is that building microelectronics that could handle cell switching at OC3 speeds was an expensive problem, took a lot of transistors, a lot of watts of power. And it just turned out we need bigger packets essentially so that the microelectronics doesn't have to make so many decisions per second. We have to cut down the number of decisions per second. And so there was Ethernet 1500 bytes much bigger than 53. So you could build cheaper routing equipment and switching equipment. If you made everything look like Ethernet or packet on sonnet or anything that we're, we're the number of routing decisions and switching decisions per second was small enough to fit in the envelope of what could be built at scale. So here again we see the pressures of scale, leading us away from this sort of unscalable technology. Okay, so I got to do some math with you on 10 megabit Ethernet, the original Ethernet protocol. If you sent minimum sized, which was about 60 bytes, minimum spaced, right, there's no distance between one packet and another other than what Ethernet requires for its trailers and headers and whatever. So minimum spaced, minimum sized packets and you just send those as fast as you can, you're going to get about 14,000 of them per second. That's a lot in the early 80s. It meant that you weren't going to be making those decisions with your CPU, you needed your Ethernet interface to be able to recognize the relatively small number of those packets that could be destined for you and ignore everything else. It was very cool and it worked, but 14,000 became a fairly practical number. That's why we had so many switches and routers and host based routers and all the rest. Okay, well let's talk about MTU here. 1500 compared to 14,000. Now it compares well. Obviously 1500 is not a minimum sized packet. So if you have the maximum sized packet, you're probably only dealing with a thousand of them per second and that's really practical, even in 1986. So here's the problem. We, we sped it up. We made Ethernet faster. We got fast Ethernet, 100 megabit, and we got gigabit Ethernet, then we got 10 gigabit Ethernet, 40 gigabit Ethernet, 100 gigabit Ethernet. We keep on making it faster because that's what the pressure scale is the opportunity to make things faster and do more things with the same number of transistors and and the problem is we never changed the MTU. The MTU is still 1500. And the reason we had to do that is every new version of Ethernet has to be compatible with the current installed base. So when we got fast E fast Ethernet, it was 10 times faster at the bit level, but we had to be able to make a bridge between the fast Ethernet and these at that time slow original 10 megabit Ethernet. Because if we said, yeah, we here's a new version of Ethernet by the way, you're going to need a forklift because you got to tear out everything you own and replace it with this fast Ethernet would have failed in the market. So they said, no, we're just going to stick with this 1500 and deal with whatever 140,000 packets per second if they're all tiny instead of 14,000. And we did that every time. And so, at 10 gigabit Ethernet, I got to tell you that the number of routing decisions we're making per second is absurdly higher than what it was for ATM on OC3C. We're just lucky that the transistors are cheaper now, but we're burning a lot of heat. And the video bought a company recently that has a line card that offers to speak these protocol like the IP IP version six whatever it'll speak that protocol for you in its own operating system using its own CPU which is on the line card, because there's no way we're back to the time that there's simply no way that a CPU could tolerate the load of that many decisions per second so we end up with a coprocessor on the line card. That's a lot of complexity and it's a lot of vulnerabilities that we don't know about yet bugs. This isn't a lot that's going to go on there, because we never changed the MTU. So again, we have a pressure scale, and it has created this MTU problem. And I got to tell you when I, half of you right now are scratching your heads because you've never had to think about an MTU before, and it never occurred to you that we had alternatives that there were other roads, and that those roads weren't taken. You don't know, not only what road wasn't taken but why, why you yourself would not have taken that road had it been up to you. And the reason for that is there's mostly not a reason for about half of you statistically speaking to have ever had to innovate at this level. This is what is built and build your thing on top of that so that your company can succeed. And so that we can all continue to get paychecks. And that is why the MTU never changes is that there's no reason for anybody who wasn't working in the industry before Ethernet came along in the first place. So if you haven't been doing it that long, you don't know what an MTU is and you don't know that there were design choices about what to use. So, I'm going to tell you, on my networks we use movie grams and I've told all the switch gear to allow up to 9100 bytes. And that works as long as you have a way to know which hosts, you can reach at that size versus which ones you can't. And again this calls for understanding how a route routing routing table works, knowing what path them to you is what knowing what path them to you discovery is. And if you do that, I hope to inspire some of you to go build a test network, use the NFS protocol network file system, tell it to use both try to try this under both UDP and TCP tell it to use 8 kilobyte payloads and run some performance tests on both a standard 1500 byte MTU and on a 9100 byte MTU that allows 8k packets. I think you'll be shocked about the roads the world did not take without understanding that those choices were ever open to them, or that they could have chosen a different road. Some of you who do that are going to end up using movie grams inside your, your companies inside your houses. For the same reasons I do. But again this is not something that they teach in school, or if they do sort of sort of like Latin, you know they teach it, I learned it that does not mean that I'm ever going to need to know this for the rest of my life. And so you forget it right away. When I was preparing to get on this zoom call. I discovered that my Lenovo laptop running the windows 10 operating system had decided to, I guess scan itself I guess it does that from time to time. And it saw that there was some device driver that really, you know there was a later version available so let's install that. The advantage is that my audio broke, because the real tech chip is finicky, and that is the thing we all use for the headphone jack on our laptop, probably I don't know for this for a fact but probably even a Macintosh is using a real tech chip for that because it's, it's, it's what everybody does. But certainly on windows, the real tech driver doesn't work all that well, and you really need to get into the device manager and delete the wrong driver and switch to the right driver and all the rest of that. And I guess it's lucky for me that I have faced this debacle before, and I knew what to do. And that allows me to speak to you today because otherwise that would not be happening. And that should concern us, because most windows users and all Macintosh users have no idea what that issue is. And you could argue that maybe Microsoft should do a better job with your device manager, or that Lenovo should do a better job on their software updates. And that's possibly so, but we're getting back to the mesh radio thing which is, you may be having a problem that you're completely unequipped to understand. You could end up potentially tying up many hours of some tech support person's time on something that's an integration failure. And yes, Lenovo and Microsoft could probably do better, but they won't make more money, nor will their customers make or save more money as a result of that investment so it will not that investment will not be made. So you can say of things that scale that nothing will ever be built at scale better than it has to be. In other words, if you're inside the bell curve, you're good enough. And once it's good enough, you ship it, you get on to the next thing. And again, I'm okay with this. I know what choices they had I know what choices I have. And I understand enough about what's going on here that I know how to fix it. My descendants I have four children, my youngest just turned 30. They do not understand these issues. So if you invite them, one of them or one of that generation to speak to this conference. 25 years from now. They may not know as much about how their technology works as I did. So it's all sort of comes back to first world problems. The internet engineering task force itf is where all of these protocols come from it's where TCP comes from IP IP version six UDP DNS all that sort of thing is done in this. The volunteer organization has like five employees who have the job of holding conferences where all the volunteers go. And they, they argue and they whiteboard and they present and they, they write internet drafts that ultimately become RFC documents which are in a protocol standards. They're working very well. We have too many volunteers, and they have too many different agendas, and those agendas are often ruled by their employers. Why does that matter, right because the people who participate are doing so as individuals. They somebody has to pay for their airplane ticket to go to this conference, somebody has to pay their salary, while they spend all day arguing on internet mailing lists about the richness of what as to what protocol should be next and how that protocol should work and what its features should be. And so these volunteers have backers, there's no way to do this if you don't have a backer, you can't keep up you can't afford the travel you can't afford the time. So ultimately it is corporate persons, not real persons, but corporate persons who dictate what standards are created, and what features those standards will have. And this is very much a wealth gap is what it means as I travel the world I talked to people who in their wildest dreams will never be able to afford an airplane ticket to go to even one itf meeting let alone three per year. And they might be on a mailing list or two they might not, but chances are low that somebody I meet in my travels is able to affect their own future by participating in this open transparent nonprofit standards process. I just can't, even I can't right and I'm the CEO of my company I could, I could tell myself to behave differently and allocate my time differently. That's not the best use of my time. And if I, if people at my level of seniority can't afford to participate, that pretty much means that the companies who can afford to hire the best people and send them in there are going to be setting the agenda for the internet going forward. And one of the ways that doing that is by moving DNS, sort of into the web, they're, they're, they're pushing the DNS protocol to start using the HTTPS protocol. So secure web transaction based on TLS. So revised TLS. So version 1.3 of TLS now is firewall proof. So any of us who have a next gen firewall that relies on intercepting an outbound connection for tending to be the far end getting one of our own devices to then tell us what it wants to do so that we can make a policy decision about whether to allow it or not that's going away, because the itf has wielded so. So the idea of a secure private network is not represented. Why is that well it's because the people who operate those networks are busy. And they are not going to go to an itf meeting and argue for what they need. Whereas the cell phone makers they are present browser makers they are present surveillance capitalist big tech, they are present. So we're going to move more of the internet protocol into a pattern that will make ad blockers impossible. And it turns out that there's an odd coincidence that the companies who are employing and funding the itf members who are pushing for this. And so now we're going to have something that is ad blocker proof. Now they're not selling it that way that they're saying is, do you look what Edward Snowden disclosed in 2013. Look at government surveillance look at you know how terrible governments are we need to make the internet, totally encrypted so that governments cannot interfere with the communication of their citizens. Well, I got to tell you, there are countries in the world. India is right next to China so China would be an example that would be close to home for for this audience that don't really want their citizens to, let's say talk about Taiwan, or talk about Tiananmen Square or whatever. And, and they got one party rule, and China is not alone Russia's doing something similar, Turkey is doing something similar obviously North Korea would be an example of citizens not having a lot of rights. There are countries around the world, who are not waiting to hear what the itf has in store for them, and they will do whatever they have to do to maintain the old status quo. So, the idea of a bunch of plucky volunteers, whose bosses are happy that ad blockers will become impossible that they themselves have a idealistic mission of making sure that communication is government proof. This is not working it's not going to work it's going to lead to chaos. But that again is the pressure of scale. Since the people who need to be represented can't afford to be represented. They aren't. And so we're getting whatever protocol the 20 somethings of Silicon Valley want us to have. That's not going to be good. So in the future, you know what what can be done to mitigate this. I would say that you have to there's something that each of us has to do that makes no economic sense in the short term. And that is that we have to build something better than it has to be built. And that's something is ourselves. We have to do tinker. We have to, once in a while, look at the assembly code generated by the compilers we use for our programming to find out what's going on down there. Because it may be that you need to make some different choices that you won't be able to make if you don't know what an instruction is or what a register is. And that's the core of all these things of food, internet protocols, CPU design operating systems. Everything about our future is going to be very heavily dictated by technology. So this is not just the age of the cloud. We are in the information age and information is power. Information creates wealth. And the people who understand information and understand information technology are going to be the ones that kind of decide what our future is. And I want that to be all of us. So I want to encourage companies, governments, and especially you individual technologists within the sound of my voice. To learn things that your job does not require. Find out the history of why things are the way they are. Don't just buy a bunch of raspberry pies to build your home lab plug them all in and run a bunch of binary packages that will cause them to behave as a virtual cloud. Look at the source code of those packages. Educate yourself on things that you have no earthly reason to know. But if you know them, there's a chance that you will make some decisions that will be informed by your experiences your needs, your plans, rather than have your future dictated to you by the relatively small and shrinking fraction of the world who actually knows how any of those works be one of those people. And that's what I came here to say. I don't know if I have any time for I mean I'm just sitting here but I don't know if I'm going to be allowed to answer questions or not but I throw it back to the moderator. Oh and let me just say, thank you again for inviting me. These topics are vital to me. I really want my children and their children after I'm gone to have a world that is democratic and transparent for freedom matters and decisions that we make matter. It's not all just sort of this pleasant veneer of technology that we're being led where the technology leaders want us to go I really, I really want you to listen. I'm not here to sell you anything other than yourselves I'm here to sell you a better future. So again thank you for having me. We're now open for questions, people who are on zoom call can paste their questions on the Q&A tab and I will just basically read it out to Paul and he could answer going forward. And if there are no questions, I may probably sneak in a few of mine. Just know one minute, are we copy pasting questions from Twitter live from the live stream or no. Are we copying patient questions from the YouTube live stream or no. The chat is not enabled. Okay. Questions anyone. Whereas on the zoom call. Thank you for your questions. Okay. The first one is from some incur, he asks, you have made it clear that DNS over his GPS presents some challenges for enterprises. Have you changed your positions since the last RSE conference. Paul, you're new. Yeah, I've unmuted thank you. I think I have refined my position but as things are rolling out. We're seeing, I'm seeing a growing concern. So if you've heard me talk about doh, then I'm probably more panic now than I was when you last heard me talk about it. The itf is moving some parts of the DNS protocol, not only into the web for transport, but into web pages. So, we will shortly see meta tags in our HTML objects that contain the DNS bindings necessary to fetch all of the rest of the objects so if you've got a page that includes maybe some style sheets and some graphical files or JavaScript. Rather than hear those references in the browser and then have to make a DNS request to find out well where is that being served from, you're going to get all of those bindings as part of the containing objects he won't be making those other queries. And that means that any of us who kind of depend on knowing the behavior of the devices on our network. And therefore saying hey if you're fetching this if you're trying to reach that host, you've probably been infected with some virus or whatever. We're not going to see the fetches because the there will be no questions. The answers will be delivered as part of the web content itself. And this is because there's no one working on the web, who is also trying to secure the web or secure the internet. So, if anything, I am. I'm even more hardened in my positions than I was when I spoke at the RSA. Oh, so many do you have any follow up questions. Can you just please base them. Okay. And he says, so men as this sounds ominous for home users who do not have the tick know how or the budget to even identify these challenges. Well, it is. And, you know, I think we all know that those of us with jobs in the tech sector might not have those jobs, if the internet weren't growing. And as it grows, we're going to reach more and more parts of the world where no technologist lives in that apartment or in that house. And so those people are going to be at the mercy of their supply chain. So, this is a big problem, but there's actually other big problems that are also being ignored. And so I guess that means we are going to ignore them all. But to give you an example, think about the Internet of Things IOT. And a lot of home appliances like refrigerators or laundry equipment or heaters, air conditioners, whatever, they all want to be on net. They all want to be controllable by your your cell phone. And importantly, they all want to communicate with their mothership back at their corporation of manufacturer. And say, by the way, this is how much laundry is done in this house. If it's a little Roomba robot to go sweep your floors, he wants to report back to its maker, how big your rooms are. And, you know, even if you can make money on laundry equipment, you don't make more money by adding this IOT logic to it. And so all of the investment that is going into IOT is not so that they can create, let's say a light bulb that you can change the color with your cell phone. They want the data, they want to know when you turn that light on and when you turn it off, because if they can learn that about you, they can sell that information. And somebody somewhere will have a better time advertising some product to you, because of all of the IOT data that has been pulled out of your house. So surveillance capitalism is not just big tech, it's also small tech. No IOT technology is ever going to be profitable from its own manufacturer and distribution. All of the profits from all investment in IOT comes from surveillance. And so yeah, you should worry about DOH. But you should also worry about that Nest thermostat you've got there. Did you know what had a microphone? Yeah, Google, when they own Nest said, sorry, we didn't tell you there was a microphone, but we've never used it, I promise. Okay, that's great. But I've also got a baby monitor camera and you know, watching my baby sleep at night that is participating in DDoS because it's got vulnerabilities. So how do I know that the microphone that's in that Nest thermostat hasn't been abused by some malicious software that got into it? So we've already crossed into dangerous territory just by building the network we have and trying to grow it as we have. That's what I'm saying is ominous and I intend it to be. I'm not here to tell you that everything's going to be fine because unless those of you listening to me take action, everything's not going to be fine. But I also just want to say, a lot of damage is already done. You know, we heard last week about something called Namewreck where the supply chain for DNS software has some terrible vulnerabilities in it. And I'm responsible for that. This was originally the bind software libraries that came from UC Berkeley that I was publishing that as recently as 1998 had these vulnerabilities. We fixed them, but the supply chain had copied my software earlier than that, never checked back with us and has now incorporated DNS libraries with known vulnerabilities without any idea of how they would know which cert advisory applies to them without any idea about what the bugs are, how they would go about updating that supply chain. Nobody knows. The companies who make the devices that are vulnerable are not in on it. They themselves are victims of the pressures of scale. And so, yes, I want you to worry. I'm telling you some bad news. And some of that isn't news. So I'm telling you some bad things that used to be news. Okay. Last question from Suman. So yeah, so would it be wise to regulate an outright ban certain products like say smart tablets? I'm sorry, can you repeat the question? This is, so would it be wise to regulate an outright ban certain products like say smart tablets? Would it be wise to regulate is kind of a trigger question. There are a lot of self styled internet libertarians who always answer no to any question that has the word regulation in it. I am not of that view. I think that underwriters laboratory did some vital work in the early days of electricity, by saying, yeah, we're willing to test your whatever toaster, so you had a device for toasting bread. It's going to get plugged into the electrical socket in some house. So we are willing under why underwriters laboratories willing to take that toaster and, you know, try it ourselves and drop it in a tub of water see if it electrocutes people, whatever it is whatever testing they did safety testing. And that gave those the people who were adopting electrical appliances, an opportunity to pick the one that would not burn their house down, or at least that you underwriters laboratories said that we tested it we don't think it's going to burn your house down. So instead of buying one that did not have that certification, and the power of governments would be to say you can't sell something that does not have that certification, or maybe government as the largest information technology buyer in most economies can simply say we are not going to buy anything that doesn't have that certification therefore it will fail in the market, because if the government buys zero instances that product it doesn't matter who else does or doesn't buy it. The government has a role here. But I'm cautious about recommending activities by government because government is historically fickle. Every time you have an election there's a chance that the government's priorities are going to change. It's also notoriously inefficient. We have a lot of people in government whose job is to is very narrow, and we don't really want them thinking outside the box. Just like we don't want our medical doctors doing real science we don't want to be the subject of an experiment we want to be cured. So, when it comes to government regulation I would say there should be a lot more concern than there is, but I don't want to make it sound like the government is capable of understanding these issues and keeping us safe. This is the last question that closed with this observation. Governments are very good at securing physical borders. That is in some sense the reason every government exists is to make sure that the borders stay where they are. And that what that means is if you have a neighboring country who I don't know maybe they'd like to fly an airplane over one of your cities and drop a bomb. Yeah, your government's going to be able to keep that from happening. That's what they do. That's why they were that's why government success, but if the same country wants to attack the same city with a cyber attack, they want to just kind of reach for you over the, over the tubes, and still your information or maybe inject bad information or participate, maybe inject disinformation into your elections that's happening here in the US right now. For whatever reason Russia has chosen to take a hand in the last couple of elections here. But your government can't stop that the information technology moves too quickly, and the government will never be able to secure the borders of cyberspace. That is going to be up to everybody who connects anything to any network. That means it's an unsolvable problem because most people will never understand the issues they're counting on their supply chain, their supply chain has the pressures of scale. So those people are going to be vulnerable. That means the people at this conference have got to scale as what has at least as quickly as the problem is scaling. So there's a difference that you here within the sound of my voice can make. The government should also take some action, but I don't want us to trust that government can solve this problem it's us this is on us now. One more question. The previous comment about the being the last question was only for the particular person. Okay, right. Right. So we have a related question from freedom, who says Paul, any resources that help you to understand our history in a better way, which can help us so starting this journey. So I did. I did not make a third slide that would be a call to action with a bunch of links that you could follow. And the reason is, it's going to be different for each one of you. So there's a Certainly, if you just look at the technology in any device that you have let's say You've got a Raspberry Pi, or you've got a laptop, whatever, you should find out what software it has and where that software came from. You should look on GitHub to see what software like that usually does. I wrote the version of cron which is in virtually all BSD and most Linux systems and also in Mac OS. And if you want to learn how to make unique system calls to create sub processes and manage pipes and look at my software. And a lot of things that are in your life today in the lives of all of us today came from the open source world. Sometimes the software is out of date and has vulnerabilities that originally came from something that probably has been patched. And it's worth looking at the history of some of the open source software that is in the devices in your pocket and find out how it works. And maybe, you know, consider downloading it figuring out how to build it and see if you can add a feature. And I don't mean you're going to add a feature and then try to upstream it, although some of you have and others will. I just mean, see if you can familiarize yourself with what are the opportunities of a developer of software like that. What roads do they have to choose between and how will they choose. You can also look at the network protocols that are spoken by all of these things as a new version of SSH out right now that is finally ready to talk to my Yuba key. And I plan to read about the entire value chain that made that possible and find out which software has been audited by whom, so that I can decide how much confidence I should place in that new feature. And all of you can do that. You don't need specific pointers from me to tell you, gee, you should probably look at this and look at that. Although if you wanted to join an IETF mailing list and just watch the discussions that's another way to get yourself involved. Because it's going to take a while to get yourself up to speed to the point where your judgment can make a difference in the outcome for all of us. And so the universal thing no matter what how you got here or where you intend to go next, the universal recommendation is the same. Learn more than you need to know, because in the future, you will have needed to know it. The next question comes from Lava Kumar. He asks, what is your prediction for the open web in the age where big companies have incentives to create walled gardens like the App Store and the Play Store? Well, the web has been mostly open. And there are some problems and I'll come to the App Store question in a moment, but I call your attention to HTTP version three, which is based on the UDP protocol instead of on the TCP protocol. Now HTTP three uses TLS and we're already using TLS and so this is not in TLS is not the new element here. But what TLS has meant is that it's very difficult to inspect payloads. You can't really do deep packet inspection successfully with if the data has been encrypted with TLS. And it used to be at least theoretically possible to do that with TLS 1.1, TLS 1.2. However, in TLS 1.3 with encrypted client hello, there will be no opportunity. So as that rolls out, a lot of us are going to get blind to what our, what is the behavior of the devices on our network, which is sad because we're going to get blamed for any spam we relay. And for any DDoS as we join so our network is becoming something that we are responsible for how it behaves, but we are not permitted by the IETF to know why it behaves that way. And that's a problem. In the HTTP three protocol, when we move from TCP to UDP, we're also going to move all of the congestion control logic out of the kernel, and into the app. So how you behave in terms of whether it's TCP Prague or TCP Budapest or whatever is no longer up to the maker of your operating system, it's going to be up to the maker of your app. And maybe it's gotten up to date copy of some protocol suite, maybe it doesn't. It's vulnerable who knows, we've made the supply chain deeper and broader. That's never a good thing for security. But the thing I want to focus your attention on is that right now, if my browser wants to reach out and grab a web page, it has to ask the kernel to create a TCP session. And that's a system call. And it doesn't matter whether it's Windows or Android or Mac OS or Linux BSD, they all work the same way, which is first the app asked the operating system to please make me a connection. And then it asked the operating system please transmit the following data which might be a TLS handshake or might be a request or whatever. And in the HTTP three that goes away. There is no system call that by which an app has to ask the operating system to please make a connection somewhere. And that in turn means that anything we had that was inspecting monitoring the app to see if the behavior was correct. And that's the ability to say, look, you're trying to connect to something you, you have asked me to make a TCP session to a destination that I know has only malicious purposes. So I am going to reject that I'm going to, I'm going to cause your TCP connection attempt to fail. And that's something the operating system can do it can have policy, and it can enforce the policy. And you'll put that app under a debugger, and you can start to say, ah, first it did this then it did that maybe you've got a sandbox and you've you collect malware samples and you run them in the sandbox. And you're really depending on being able to find out what is the behavior of a piece of malware. All of that disappears. Once we move from TCP to UDP. So I think the problems from the open web are much larger than just what will happen in the app stores, but I want to answer the app store question also. I have an Android phone and I can install any app I want there's a little checkbox in the settings. So I don't have to get everything from a certain app store. If the Amazon Android app store has something I want I can install it. And that's freedom that the Android team over at Google has decided I should have. Apple has made a different decision, you can only get apps from them or if you're a developer you can get an app over a USB cord or Bluetooth connection. But there there will be no apps running on Apple devices that did not get approved by Apple. This should concern us, because while let's say the Firefox or Brave browser on every other platform is responsible for its own connection management. In the Apple world, they won't permit you to do that. So those browsers all use, I believe the JavaScript engine and the connection management framework that is in the iOS platform. And so the browsers may look different, but they're all kind of the same same engine different car. And that's a decision Apple made. And Apple is capable of making other decisions. There was a period where they were permitting their own apps the things that come with the operating system on the Mac OS they're permitting those to bypass the firewall. They went directly down into the mock kernel and they just bypassed whatever kind of firewall the Mac seemed to have. Wait, all of my Apple apps are getting out even though I've got a firewall that should block that well. That's Apple. Apple explained look, we need to have a good user experience we can't let your firewall get in the way of that. Ah, okay. Now they reversed that choice they said. Yeah, we kind of understand why you might want to have a firewall and so we will, by our own policy, volunteer to be subject to your firewall on the device you bought that we still control. But your device do some do one more thing for you. And, but that's a choice we, Apple are making. That means Apple can revise that choice at any time. So it's not just that you can't get your apps from anybody but Apple, and that you're therefore only going to get the apps that don't threaten Apple's market dominance. So I have Apple deciding exactly in detail how your device the one you paid for that's on your network is going to behave. I was listening in on an ITF conversation about HTTP 3 and somebody from Apple said, yeah, we can't do that. I don't know what the future is some protocol knob somebody was saying, we should do it this way and Apple said we Apple can't do it that way, because we've got multiple things we've got a radio that speaks to LTE the the mobile network and also the radio that speaks to the local Wi Fi. And we use both at all times we are, you know, so some of the things we do like media streams we're going to choose one others control streams will choose the other. And that that worries me, because it matters what signal we transmit from the devices that we carry, and it matters who's able to see them. There's a huge amount of geo geographic location information about all of us is being uplinked by all the apps on our phone. So if you like. Oh God, this is a Pokemon game. And if you like that kind of thing, then be aware they're not making any money from you playing their game they're making their money from selling your location which they have access to whenever that game is running. But there are a lot of worries about the openness going away. And so here I'll give you a historical perspective. I had something that the OSI protocols we did not have. And that was that it was organically developed by the tech community. And this is before there wasn't internet there for there were no search engines or anything else this was the tech community pre commercial. And they built some things that worked that way. And I guess Berkeley Unix was the first one to have TCP IP but then Linux came along and also adopted it and it just turned out if you get more better work done with that than with any of the OSI stuff. The reason that all of that was possible is that we had permission to innovate with the freedom to innovate. So when let's say the web came along, they said hey there's this thing called the internet and it's a way for anybody to exchange digital data with anybody else. What can we build on a framework like that. And they did they use TCP they use DNS they used, you know, the lot of internet technologies to build the web. And that's because the internet at that time was designed to give you the freedom to innovate. You didn't need to ask any phone company, whether you could build something like the web, whereas in the OSI protocol suite you would have had to get the phone companies to think that this was in their best interests. So what have they done what has big tech done what is the web industry done with that freedom to innovate well clearly they've got a global e-commerce and communications platform that is very relevant very influential. But the other thing that they're doing is making sure that that freedom to innovate. Therefore the freedom to disrupt is no longer present. So they've, they've climbed a ladder and now they're pulling that ladder up behind them. You can resist this, but it will come as some cost to you will be getting complaints as I do from my family about the things that our network refuses to do. If you're willing to live with those complaints, you can keep the that ladder from being fully pulled up. But step one, I'm going to go back to my original message step one is you have to understand how this works you have to know how stuff works, you have to know more than you need to know for your job. And that again that's the same as true of network operations. Don't become that person who only understands the GUI, you know, go learn what that router is trying to do. Yeah, you may never have to configure a BGP session by hand there's some automation that'll do that, but you need to know what the automation is doing and what the choices were and why they made the choices they made. Cool, we have a last question and then after that we're done I think. Okay, this is from Lava Kumar he says, what impact do you think the deep platforming of the US President had with some digital rights and freedom be made fundamental rights. We're already having wide disparity in rights. So if you consider the parts of the world where government is very weak. You see a lot of really terrible things happening. And here I'm thinking about human trafficking. You could pick whatever example is the one you most worry about. And when government is too strong. Like let's say North Korea is a traditional example. You get different abuses right there's an ideal set of rights and an ideal sort of extent of the government's ability to protect and recognize those rights. There is a there's a Goldilocks zone there is a middle ground there that works. And one of the reasons I've always been ready to come to India realize I'm only here virtually right now but one of the reasons I agreed to speak at a couple of different conferences in recent years and would accept an opportunity. One son allowed to travel in the pandemic is behind us is because India. You know I'm sure those of you who live there have got complaints but I gotta say, compared to the rest of the world, India is in the Goldilocks zone of what rights are recognized by government and how well the government is able to enforce those rights. I'm not saying there aren't problems and if I live there I know enough about those problems to complain about it. I just want to say that India has been a shining example about how to find the Goldilocks zone and mostly stay within it. Most other countries don't do as well. So it won't really matter what rights, the technologists building the next gen internet or the next gen web want to have recognized what will matter is local power. That's the reason we have nations. And if you want to know what you want to see this reason in action, just go pick some piece some square kilometer of ground somewhere, and put it outside of any nation you find that some nation has quickly annexed it. So, we've got a lot of people who think that nations are not the ones who should be recognizing and protecting rights. Some of those people are building technology. So, it won't matter the answers to this question is, what rights, we wish to have recognized won't matter. It's the ones that people who are in nations are successfully able to fight for, that's what will matter. And that again is what I'm hoping to get all of you to help with is help with that fight. Thank you, Paul. Thank you for an interesting session. You're done today. I'm pretty much I'm pretty sure there will be more people who will be wanting to interact with you once our pandemic problems go away. You're quite a bit in the tsunami zone of the pandemic. Right. Yes. I think you're terrible there in the news. I wish you the best and best outcome. Yeah, and there will be more and we will probably reach out to you once that subsides. Thank you for that. But let me just say, email is great. Talk to me. My email address is in the portal. If you have more questions you want to have more discussions I am available. This is what I do. Okay. Thank you everyone. Let's reconvene tomorrow for the two of the data for our conference. Thank you. Thank you everyone for staying with us.