 coming up on D. T. N. S. Is A. I company deep mind actually profitable? We explain how Facebook's outage happened and what effect it had and why Francis Haugen thinks transparency is the key to fixing this. This is the Daily Tech News for Tuesday, October 5th, 2021 in Los Angeles. I'm Tom Merritt. I'm Roger Shane, the show's producer. Sarah Lane has the day off, but Nate Langson, Bloomberg tech editor and host of text messages here. Welcome back Nate. Hello. Thank you for having me again. We were just talking a little bit about how we were personally impacted, not very much by the Facebook outage yesterday, as well as fostering dogs. If you want that wider conversation, get our expanded show Good Day Internet become a member at patreon.com slash D T N S where you can join our top patrons like Johnny Hernandez, Hi Tech Oki and David Mosher. Let's start with a few tech things you should know. Google will make an announcement October 9th at 10 a.m. Pacific in the invitation to the virtual event. Google cryptically wrote pixel fall launch. What could it mean? Further text associated with the invite may provide a clue. Google wrote, we're officially introducing you to pixel six and pixel six pro. We could parse that a little more. But I guess we'll just have to find out what it means in that event on October 19th. Google also added the Nest cam wired and Nest cam with floodlight to the Google store. The $100 Nest cam wired as the name implies has no battery needs to be wired in meant for indoor use comes in four colors. And one version has a wooden base that cost you a little extra 120 bucks. The original black indoor Nest cam still available for 130 bucks. And then there's the $280 Nest cam with floodlight. That is a camera with a floodlight that has to be wired through an outdoor junction box. It detects people, animals and vehicles using some on device machine learning. So it's not just motion detection. A few other products here from Kobo. Kobo announced two new Bluetooth enabled e readers. The $260 eight inch Kobo Sage lets you take handwritten notes, but you'll have to get the separately sold Kobo stylus to do that. That's 40 bucks. Notes can be converted to plain text though. Once you do the $180 seven inch Libra two is the other product. It's IP x eight waterproof meaning it can survive two meters of water for up to an hour. Both of these are available for preorder now ship in October 19th. A couple other quick hits here Singapore's legislature approved a bill called the Foreign Interference parentheses countermeasures Act, which gives the Home Affairs Ministry the ability to order companies to hand over user details, remove apps and block content if they are deemed to be used by foreign actors to interfere in Singapore's affairs. The bill would also let the ministry designate politically significant persons, either actual people or entities and order them to disclose or give up foreign funding. The bill must be signed by the president of Singapore before it becomes law and Ubico launched the Yuba key bio. It's the first of their keys to support passwordless login and biometric authentication. It also lets users use a pin in case biometric authentication fails. If it's temperatures weird or there's moisture issues, hands wet, the key does not require batteries, drivers or custom software and works with Windows Linux and Mac OS. Yuba key bio is now available in USB A and USB C versions, the A versions 80 bucks. The C version is 85. All right, Nate, are you ready for the explainer of the Facebook outage? Take it away. All of Facebook services from WhatsApp to its building security system went down Monday, October 4 for six hours. You heard us talk about it briefly yesterday. Facebook VP of Infrastructure, Santos Jenardin posted twice with some good details about it, explaining it was a configuration change at the root of the problem and emphasized despite your best imaginations. There was no malicious activity. No user data was compromised. This was a very bad and unfortunate mistake. Cloudflare you may have read yesterday posted a great explanation of what it looked like from the outside and that's where you may have heard people talking about DNS and BGP as explanations for the outage. They contributed but they're not the cause. DNS is often referred to as the phone book of the internet. So when you type in a domain name like facebook.com DNS is a system that tells your browser what server that domain points to so you can go get the web pages. If you have only one web server, that's easy. The DNS looks up Nate's only server.com sees what IP address it is points at that tells the browser gets the machine. However, if your network is larger, like an ISP or Facebook, you've got a more complex system. You've got multiple servers. And in these systems, the border gateway protocol or BGP works. It's often compared to a postal system. So when your browser sends its request in for Facebook.com, BGP figures out just which server is best for the job. It advertises to the rest of the internet. Here are the servers that are available. That way you get connected to a server that is near you for faster service. That's an oversimplification, but it's kind of right. In larger networks like ISPs, what cloud friend notice is that all of Facebook's BGP was withdrawn. BGP tells the internet where to find Facebook's DNS servers that meant any request. Once the BGP was withdrawn for a Facebook domain returned what's called serve fail. You often update BGP to maybe you had to take a data center down for maintenance. You usually don't withdraw the entire thing. So serve fail tells other DNS tables around the world to start updating their files to show that any server associated doesn't exist. If you take away all of Facebook's BGP, that means all of Facebook.com no longer exists to the rest of the internet. That caused some automated systems to assume it was available for sale, which it was not. You could see that on the official ICANN record. But if you saw that, that's what caused that. Facebook says the problem was caused by the system that manages communication between its hundreds of data centers around the world. It's got fiber optic cables. It's got undersea cables. It's got high rises full of data centers. And it's got a system that manages communication between them. On Monday, engineers were doing routine maintenance. Often that means taking down a part of the system, so they did a routine command to assess global availability just to make sure they had only taken down the right parts. The command for assessment, however, is the problem. Apparently, it was malformed because Facebook says that that command unintentionally took down all the connections within Facebook's internal network. Facebook has an audit tool that's meant to catch these kinds of human errors, but there was a bug in the audit tool, so it didn't catch it. Double failure. So the upshot is Facebook's data centers now had no way of talking to the internet. And that leads us to the next failure. To keep the network clear of junk, Facebook's domain name system servers, the DNS servers disable BGP if a data center is unavailable. That way you're not telling the internet to send request to something that isn't there. That's normally normally a very smart thing to do. However, in this case, all the data centers because of that configuration error appeared unavailable. So the DNS dutifully disabled BGP for everything, including the DNS servers themselves, meaning DNS appeared unavailable to the internet. Data centers couldn't be accessed because of the configuration bug and the tools you'd use to investigate that were now down because DNS was unavailable. You can't go to Facebook.com slash audit tool when DNS is like there is no Facebook.com or actually the internet saying like, we can't find a DNS for Facebook.com. So Facebook had to send people to the actual data center. That's why they had to do that. They had to debug the issue directly on the machines themselves. And of course, Facebook wisely, I think makes it hard to get into the data center to access its servers and physically modify them even if you have the right to access because how often are you going to need to do that, right? You want to make it hard so you can catch bag bad actors. So it took a long time to do it. It did not mean that they had to cut things open with an angle grinder, as the New York Times briefly reported then retracted that did not happen. And then even after they fixed the machine, turning something as big as Facebook back on all at once would cause huge traffic surges, power surges that could could could cause electrical failures. So this Facebook had drilled on as part of its storm recovery plans, and was able to bring back a little more slowly than you might have thought. But without incident, just took a little more time to do it right. Janard and finished his post by saying we've done extensive work hardening our systems to prevent unauthorized access. And it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a trade off like this is worth it, greatly increased day to day security versus a slower recovery from a hopefully rare event like this. From here on out, our job is to strengthen our testing drills, and overall resilience to make sure events like this happen as rarely as possible. So Nate, there you go. That's what happened. It all seems understandable. I feel feel very much for that admin or that engineer who typed in the wrong command, who malformed that command. And can I just can I check on your little soundboard there? Have you still got your round of applause sound? I do. I just press it quickly. That's on behalf of everyone listening who is impressed with being able to sum up all of that in five minutes. Because that was that was incredible. And I have nothing to add other than what we're about to talk about. Which which I'll defer to you, of course, for because there is more to say. Alright, should we talk about the effects then? Yeah, people noticed. Down detector takes public outage reports to help you figure out if it's just you are not received its largest number of outage reports ever for one incident at more than 10.6 million for Facebook being down yesterday. That passed the 7.5 million that had reported Facebook's app outages back in March 2019. It also had a halo effect. Cloudflare's post about the Facebook outage did a great job of showing why this affected other websites that weren't involved in any of the network errors like Discord, Telegram, TikTok. Basically, Telegram added 70 million new users on Monday. That made it slow down. Because when people couldn't use Facebook, they all rushed to sign up or use other stuff. Facebook gets a lot of use every day. So when you cut everyone off at once, that's a lot of people flooding into other systems. And it was costly. MarketWatch did a back of the envelope calculation based on Facebook's 2020 2020 revenue, estimating Facebook was losing 163,565 dollars every minute yesterday. So total of more than 60 million. Now granted, some of that could be made backup with make goods and that stuff. But not all. And the losses don't stop with Facebook. A lot of businesses use Facebook, WhatsApp and Instagram to communicate and sell to customers. None of those were available for six hours. Not all the effects were bad. Haystack provides engineering metrics for development teams. And it showed developer throughput among its customers rose 32% yesterday. Monday usually sees a bit of a rise in productivity from developers, but not by that much according to Haystack. The analysis indicates lead time for pull requests increase the most. That implies if you don't know what that means that developers had extra time at the end of their day to close off a bunch of stuff, basically do housekeeping stuff they usually wouldn't have time for, you know, like when you clean your house because you're bored. So Haystack was quick to point out that devs could have just gone to other social networks to engage in the great internet pastime of making fun of Facebook being down, but they didn't. They used the extra time they had from Facebook being down to clean up some stuff. Yeah, and one of the really interesting sides to this that I'm excited in a weird sort of way to see play out is the fact that Mark Zuckerberg has made a huge, huge deal about the advantages of a unified system, you know, from the ground up, you know, quite literally when you're thinking about data centers. And, you know, one of the big fallouts of Facebook, which again, I know we're going to come to is about whether the company is too big. And it's when outages like this happen. I mean, yes, on the one hand, it does also underscore a fundamental weakness that the internet by design can have in extremely rare circumstances like this where you own and operate everything. But I do also think that it's going to be used as a bit of an argument against the unified principles that that Zuckerberg has for Facebook, even though on a, you know, splitting the companies up on paper isn't necessarily going to change anything about this. But it will raise questions about just how joined up should these companies be when a failure like this can cost the communication potential of billions of people all at once. That's going to be really interesting to follow. Yeah, it's easy to armchair this and say, well, why didn't they have redundant systems? It's costly. It's very costly. This isn't just Facebook being money grabbing costly. It's very costly to not run these as one system. And the fact that BGP went away, and hopefully you have at least a somewhat of an understanding of what that means, is it, you know, once in a lifetime event, you don't expect that to happen. And I think Santos Jnardin's post very, very carefully says we shouldn't overreact to the fact that this happened because it's still very unlikely to happen. So I don't think Facebook has to change a lot of things. It is, as you say, an indication of just how big Facebook is. And if it weren't all in the same network because of those cost efficiencies, if it were separate companies, at least Instagram, what's that wouldn't have been affected, right? And what's that being affected? Huge problem for businesses all around the world. Huge problem for families communicating all around the world, which is why you see them driving into Telegram at 70 million. I do think, and I'm curious what you think about this, Nate, it will have caused a lot of people to try something else to get by for six hours yesterday. Yes. And some of those people some percentage of those people will say, actually, this is better. I'm just going to keep doing that. We do see that from time to time. I know Signal said they had also had millions of new users sign up and Signal and Telegram both benefited when WhatsApp announced the big privacy overhaul to its terms and conditions over the last year. But as we've seen before with both outages like this, with privacy scares, with things like Cambridge Analytica, in Facebook's case, again, those things don't long term tend to be the things that dent either a share price or a user base. They always recover and it is far broader issues that tend to take down the behemoths. And it's usually much more nuanced than one event like this. Yeah, if anything, what this is going to show people is, wow, I do have all my eggs in one basket. When Facebook went down, my entire business was unavailable. Maybe I should have a website. Maybe I should diversify. And I think, you know, we could talk about whether Facebook should be legislated or antitrust applied. This could be just good in having people realize, oh, maybe I maybe I shouldn't be encouraging them to be a monolith myself. If you can, if you can take that action, or in some cases, people will realize, crap, I have no other option, which is bad too. Now, there are a lot of conspiracy theories out there that this was all done to take the heat off of Facebook, which doesn't make any sense to me because it just increases the heat on Facebook. But the other thing happening was CBS 60 minutes publishing former Facebook data scientist Francis Haugen's complaints that were filed with the Securities and Exchange Commission. We mentioned yesterday that Haugen came forward as the whistleblower. Eight complaints were filed based on tens of thousands of internal documents copied by Haugen before she left Facebook in May. The allegations include Facebook and Instagram being aware the platforms were used to quote, promote human trafficking and domestic servitude and failing to deploy countermeasures to combat misinformation and violent extremism. Haugen testified before the US Senate Commerce Committee today, opening by saying, Facebook products harm children, stoke division and weaken our democracy. The summary of Haugen's testimony would be that she alleges Facebook knows the harm it does, which I think we all know that we all know a lot of the harm we think Facebook does and Facebook is aware of it too. But she says it intentionally does not take action to remedy these things for fear it would harm profits too much. This is similar to the conclusions from the Wall Street Journal reports from September that are based on leaked documents provided by Haugen. In the interview with 60 minutes, Haugen said if the company made its algorithm safer, users would click on less ads and make less money over time for Facebook. Haugen also called several times for Facebook to open up its research and its procedures saying quote, no one truly understands the destructive choices made by Facebook except Facebook. We can afford nothing less than full transparency. Of course, senators focused as you might guest on harm to children and whether Facebook intentionally markets to children. But this will not be the end. Senator Marsha Blackburn plans to hold more fact finding hearings about Facebook over the next few months. Senator Ed Markey has brought back his kids act, which would put limits on apps developed for children. And of course, never get between Senator Josh Holly and a microphone is what Justin Robert Young always says. Senator Holly drew up a tort bill to make it easier to sue social media companies if you can prove they are harmful to children. How much momentum will this testimony bring so far? Everything that we've seen and everything being planned are subcommittees and smaller efforts. So can area in the coal mine to keep your eye out for is a full U. S. Senate Commerce Committee probe. If that happens, things are leveling up. If committee chair Senator Maria Cantwell, remember that name, Senator Maria Cantwell signs off on an investigation with subpoena power from the full committee. As I said, things have leveled up. If you don't hear that, just expect more of the same of what you've seen till now. Maybe a bill gets momentum. Maybe not. Nate, what do you make of all of this testimony? All of the Sturman drawing? Well, it reminds me a lot of what happened actually following Cambridge Analytica as regards, you know, privacy when the UK Parliamentary Committee here called the DCMS, which, you know, long time listeners will probably remember something about, made a real show of getting executives from companies and individuals who'd worked with Facebook, with Cambridge Analytica and gave them a pretty hard grilling. It was the one that the sessions that Mark Zuckerberg quite infamously never showed up to give evidence at. And the long term fallout of that has just been this kind of deeply rooted distrust of companies like Facebook and other large tech companies, certainly Amazon and Apple and Google, Microsoft, you know, all get fingers pointed at them from time to time. But particularly Facebook because of this kid's angle. And over here, there's been a huge amount of attention being put on getting answers from the company about these sorts of topics. But they're sort of outside of our jurisdiction when it comes to, you know, a fundamental overhaul of how the company is. Like it's really hard for anyone other than the US to do this because they are an American company at heart. So for this attention to be being placed on it for these reasons, it echoes what I know a lot of lawmakers here in Britain, many other countries worldwide have kind of wanted to see. They're going to be watching this. And you can probably expect to see increased attention locally around the world as a direct result, following the footsteps. So it's going to be very interesting. Yeah, I believe Hogan wants to make things better. But when she says only Facebook understands how destructive Facebook is, I would tend to believe even Facebook doesn't really understand how destructive it is. There's a lot of, well, it makes sense that if Facebook has this many children, it would have these bad effects. That's not science, though. That's that's hypothesis. And these are reasonable hypotheses. I would like to see some actual close looking and saying, but does it? For instance, the Wall Street Journal articles that were implying that Instagram caused. Some people think the Wall Street Journal articles say Instagram caused teens to have bad self images. They didn't. What we got from that is Instagram aggravates bad self image. If you have a bad self image and you use Instagram, it's going to make it worse. To me, that's that's much more useful. I don't care to punish Instagram. What I care about is, ah, well, if we know it aggravates, then that helps us tailor our advice to children. It helps us tailor therapy. It helps us tailor policy to be like, well, we shouldn't keep teens off Instagram. Most of them aren't harmed by it. We should be figuring out how to make sure that those who could be harmed will be off of it. That's just one example. I think there needs to be more open. I've said this before, but I think there needs to be more opening of Facebook's own internal research to outside viewpoints that can study it and actually determine what is happening because this is all new. This is like television in the twenties. We don't really know what effect it has. No. All right, folks, if you'd like a different perspective than mine or Nate's on Facebook and tech news in general, you're going to want to get the tech John in your life. Rob Dunwood, Terence Gaines and Stephanie Humphrey host the tech John taking a second look at the week's tech headlines from an African American perspective because they're African Americans. So they just talk about it from their own perspective. You can follow it over at the tech John dot com. They're talking about Facebook in today's episode that came out. That's T E C H J A W M the tech John dot com. Get it in your life. All right, some non Facebook news out there. UK regulatory filing shows that alphabets AI unit deep mind made 43.8 million pounds profit last year. This comes after reporting a loss of 477 million pounds in 2019. Quiet a turnaround. Google acquired deep mind in 2014 for around 400 million pounds and alphabet wrote off 1.1 billion pounds in debt at one point to kind of wipe the slate clean deep minds revenues stem from applying its tech to various alphabet projects ads on Google, sales, shopping, Waymo YouTube. A recent collaboration with Google Maps improved the estimated arrival times of your using Google Maps by up to 50% if you're using navigation. Of course, since all of deep minds revenue appears to come from within alphabet, it would not be difficult to change the amounts credited to deep mind to make it appear profitable and not saying that's what they did. But it's a little murky. One thing that is clear. Hiring seems to have leveled off deep mind says it has more than 1000 people, which is close to what it said last year and staff costs only rose from 467 million pounds to 473 million pounds. So costs at least are leveling out. Yeah, be cool if you could see some outside clients. That's all I'm saying. Yeah, and you know, it's a very difficult company to get a firm grassbomb because it's kind of becoming like plumbing inside inside of Google. And you know, there's an argument to be made that in order to keep people inside a company interested and engaged in working on what they're working on, you need to be feel you need to feel like your works validated, like you're contributing to something. And the fact that this news originally, you know, it comes out of what's called company's house here in the UK. It's where, you know, all companies over a certain size have to report their filings, including in this instance, deep mind. It's not like it's come out as a press release announcing, hey, we're profitable. Hey, look at all this tax. We are probably, you know, insert joke as applicable here paying. This is possibly one of those things that's kind of done in a way that makes employees feel good and the press just happened to notice. This is not a place the public goes to get news company's house. So who knows? But it's interesting for sure. Yeah, I think I think what's interesting to me is, is that alphabet wouldn't mind this getting out wouldn't be shocked if somebody maybe, you know, not someone along the way, because this would be one of the first big, hey, we have a non Google division that's profitable, right? Yeah, and that is how it often happens. But, you know, let's not read too much more into it than that. I would imagine it's, it's, it's not going to be something we're going to keep going back to, right? Yeah, no, unless they get a bunch of other clients and then still look profitable like, oh, okay, no, it was legit. They're doing good stuff. Real quickly, tortoise, a remote operated robotic delivery service is expanding to convenience stores across the US with a two year deal with Idaho based King Retail Solutions under the agreement King Retail Solutions, aka KRS will lease more than 500 sidewalk delivery units to its stores. Tortoise vehicles are suited for deliveries ranging between three to five miles. They move about four miles per hour. So you know, kind of plug in along on Sidebox units are expected to complete eight to 10 deliveries a day during store hours. Tortoise also recently made deals with Logistics company Axel Hire, a grocery chain called Shoprite and another convenience store chain called Choice Market. Of course, tortoise isn't the only company doing this. There's Star. There's a few others out there all around the world. It does feel like Nate that sooner or later, little robots trundling along with grocery delivers deliveries might just become normal. Absolutely. Yeah, you've really taught us all a lot about automation today. Hey, oh, um, yeah, still not buying into the whole Amazon thing. The the the Astro, but but for grocery stuff, I'm much more in favor of and I love the name tortoise. It's such a great creature in the world. And I love that is a company named after it. It's brilliant. And it I think it's descriptive, right? Because it just kind of trundles along like a tortoise, you know, but it's fast enough four miles an hour over a couple of miles to get you your groceries in a recent amount of reasonable amount of time. And of course, even with lockdowns easing in certain places of the world, people have got this is what we're talking about earlier. People had to do online delivery for some things for a while. And now they're used to it. Now they're like, you know, even if I don't have to kind of like it. So convenience stores are you wouldn't have thought convenience stores would be a big market for this, but but they are showing huge growth in deliveries. And these are remote operated. These are not autonomous. But even so, you can have a human do a lot more deliveries than you can if you're trying to pay them, you know, 20 pounds an hour to go out and drive from one place to the other because you have to transit time back. Yeah, the show. All right, folks, if you're a wine lover, but you're not sure the vintage or type of wine you like is available in the location you're traveling to, if you're doing any traveling, Chris Christensen has an app for you. This is Chris Christensen from amateur traveler with another tech in travel minute. Last week, I recommended an app for beer lovers. But if you were a wine lover instead, I'm going to recommend the Vivino app. There's a number of different apps that help you keep track of what wines you like. It can be a little more tricky with wine because you might like that wine from that vineyard for 2016, but not for 2017. It's not always as simple as I like that wine or I don't. Vivino builds itself as the Shazam app for wine. You take a picture of the label and it can search it or you can search by name and if you don't have it in there, you can add the wine just in case it's missing. I do find that even if there are some wines from a winery that I'm at, they may not all be in the app. So I do find that I need to add some wines, but I live in a state with over 3,600 vineyards, so that's not too surprising. But check out Vivino. I'm Chris Christensen from Amateur Traveler. Thank you, Chris. Although it is it's just strike me that calling it the Shazam of wines makes it sound like you're listening to the wine somehow. Oh, that will be similarly as we do that. Yeah, you can guarantee it. You can I hear I hear notes of cherry. Yeah, I folks keep the emails coming. We love to hear from you. You got it. You got an opinion on all this stuff. Let us know what it is. Feedback at Daily Tech News Show dot com. Thanks to our brand new bosses. Man, when you become a boss, if you're out there listening to the public feed, you not only get a bunch of perks, a lot of extra content to fill out your tech knowledge, you get an ad free feed, but you also get thanked right here on the show are new bosses Chandra Benek and Chad Johnson just started backing us on Patreon. And this is why the applause for Chandra and Chad. Thank you both for backing us. You could be in their place tomorrow. Patreon.com slash DTNS. Thank you, Nate Langston for joining us. Remind people where they can go to find out what you're up to. Yeah, I had more comments about the episode we just put out on Sunday, episode 251 of a text message that it is one of the best shows and most entertaining shows we've ever done. And Ian and I, my cohost, we said similar before we even started getting those comments. So it'd be a great first episode for people to listen. If you like tech news from, you know, this voice that you are hearing now UK tech show dot com episode 251, give it a try. Apparently a lot of people seem to think it's very good. So hope you might agree. I enjoyed it quite a bit myself on Sunday. So go check it out folks UK tech show dot com. We are live Monday through Friday for 30 p.m. Eastern 2030 UTC find out more daily tech news show dot com slash live back tomorrow with Scott Johnson. Talk to you then. This show is part of the frog pants network. Get more at frog pants dot com. Hope you have enjoyed this program.