 Thanks a lot for coming. I really appreciate it. It's really late, so I admire your stamina for showing up. I know there's probably food and some kind of a party right after this, so I'll try to get through it pretty quickly, and then we can get out of here a little bit early. When I saw that this KubeCon was gonna be happening in Detroit, a lot of people seemed kind of taking it back and really surprised by it, but I thought it'd be a good opportunity to use that sense of place to gain some perspective, learn a little bit more about this place, and I actually find Detroit really interesting. It's always been a place of contrasts, huge opportunities mixed with severe tension, and so I hope that we can learn from its history. It's got this repeating underdog story going for it, and I know that we can find some ways to apply this to open source and tech in general. There's some obvious differences between tech and manufacturing. Of course, when I wanna scale something up, I just keep control, scale it, but manufacturers need to go buy land and hire people and things like that, so it's understandable that the auto industry can't move quite as quickly as tech can, but that's not always terrible. Not all of the rapid change in tech is good or even really consistent. For instance, a few years ago, I heard that everybody needs Kubernetes, and now I read on Twitter that nobody needs Kubernetes, so open source has so many admirable qualities, but we should also look for lessons from big tech, and the reason that I think this is so important for us in open source as well is that it's still dominated by a handful of players that make it viable, and as they go, so go we all. Before I get too far into this, this is me, the tiny feet dangling from the carrier belonged to my kid. He's two and a half months old, and I just came back from two glorious months of leave where the most complicated thing I did the entire time was change diapers, so I'm kind of jumping right back into the frying pan and seeing how this goes with like KubeCon being my first time seeing outside people and doing work stuff in forever. I've been writing offensively bad code since I was barely older than he is, and I've been actively working in security for 10 years. I'm also from Canada, so all major US cities seem like really large and intimidatingly unpolite to me. Thank you, you're too kind. Here's a rough agenda. The intro, that's what we did first. The agenda is what we're doing now. There's some history, that's what happened before, but we'll be doing it soon. There are lessons, that's what I hope you'll do later, and I'll be doing it less later. And then the conclusion where we repeat what we already did and we'll be doing that last. So to get right into it, this isn't comprehensive. If I could give this the detail that it really deserves, then I would have just solved peace in the Middle East before lunchtime, but hopefully we can get something out of this. So instead, I give you a truncated abridge to proximate it and probably mostly wrong history of Detroit. Our story started with the British and the French fighting as they were want to do over places where other people had already lived for millennia. The French noticed that there was a waterway here, a strait, and for giving my accent, they creatively called it la détroit, which means the strait. The British took Montreal, the French gave up Detroit, and America took it in the revolution. And so I'll state without evidence that as the story goes, in 1812, Chief Tecumseh marched his troops in circles, past Fort Detroit, and Hall thought that there were way more people there than there actually were, got really scared and surrendered. The British kept it for a while, but they abandoned it after some ship sunk in a lake nearby. And in between all of these wars among various teamonger governments, themselves supported by a Byzantine carousel of changing alliances with indigenous peoples, folks continued to settle near the fort and a town was born. So after that, there was a fire, a vote, a road, and waves of immigration that brought mostly white settlers from Europe and mostly black folks from the American South. As we get into the 20th century, we get into the industry that defined Detroit. As far as I can tell, modern historians can describe the mass murder of millions in a cold clinical way. But if you suggest that an individual truly changed the course of history, they get really emotional. So these so-called big man theories are not in vogue and it's fair to say that the automobile was going to be really big. It was probably going to be big in the United States and it was probably going to happen here, but it happened exactly the way that it did happen, mostly because of Henry Ford. Unfortunately, many of the most iconic companies in automobile history were led by people whose economic success gave them false credence to sort of repugnant ethics and Henry Ford was an unapologetic anti-Semite. It gets worse, Volkswagen was started by actual Nazis, and of course this isn't limited to the auto industry, but there is a tendency to mistake economic success for the sort of globally superior intellect and to assume that success in one domain is going to imply expertise in all domains. This is especially dangerous when it's combined with anti-social behavior and in group bias, so keep an eye on your founders. I am pleased to say that thankfully this is not a problem in our age and there is peace in our time. Moving on, we get into just before the 20th century, Henry Ford established a car company here in Detroit. Does anybody know what that company was called? I thought it was Ford, it's not. So in 1898, Henry Ford founded the Detroit Motor Company. He made two amazing cars, not two models, he made two cars and promptly went out of business. So in 1901, Oldsmobile picks up the slack. Henry Ford comes back, founds the Ford Motor Company in 1904 and GM gets started in 1908. So in this way, Detroit at the turn of the 20th century was kind of like an incubator for car companies and we see a similar scene that played out at the 21st century in Silicon Valley in a very different industry. In 1908, Ford releases the famous Model T that's what we see over here. And in 1915, 13 of the most popular car brands were based in Detroit. So you really see the benefits of that specialization in the early 20th century happening here. And I think there's a lot of parallels that we can look at for what happened 100 years later in California. By the 1920s, there was a boom time. There was also prohibition and resourceful Detroiters take advantage of the proximity to Canada where just across the border was conveniently located a massive distillery complex. In 1926, this newfound wealth and the burgeoning industry in the United States had become the largest industry in the United States. And this had some knock-on effects because whatever you need to build the cars is also going to benefit from this enormous growth in the auto industry. So anyone making rubber, anyone making steel, providers of coal, all of those things, shipping and logistics, they all found that Detroit's auto industry was a boom. And in 1927, the Ford River Rouge plant opened in Dearborn. It created almost 100,000 jobs in the region, which is huge today, but even bigger in 1927. And that's not all working at the one plant that includes parts suppliers and things like that, but it's just an enormous amount of opportunity created when that one plant opened. And dozens and dozens of plants are opening every year. More people were buying cars. Even people with earlier models of cars that were still working were just buying new cars because they wanted a new one. They had improvements in the industrial methods that were making them more reliable. They were making them safer. And we've got the Model A here. It didn't even require a hand crank to start. You could just press a button and that thing would get going, which sounds like a really big improvement to me. This also though led to two troubling trends that I think we see recurring today. So one is an indecipherable sequence of model names and the other is an ever-increasing vehicle size. The Model A was two feet longer and 1,000 pounds heavier than the Model T. Now in the 30s, although Detroit was affected by the depression along with the rest of the country and much of the world, employment in the industry continued to grow, albeit moderately throughout the 30s. And by 1940, this increase and migration, especially from the American South, made Ford one of the largest employers of people of color in the entire country. In 1942 though, America joined the war. They stopped all civilian production and switched to military production. So Detroit was building jeeps and tanks and bombers instead of automobiles. As men joined the military, women made up the labor shortfall and joined the assembly line in record numbers. Like in this photo from the Willow Run plant here, it should be pointed out that they started working for lower wages than the people that they were directly replacing and were asked to leave when the men came back. There's a long way to go. It's now 1945, the war ends and we move back to car production. And this is not even quite at the peak. Things are still moving in a positive direction really rapidly. But one of the very first of the second and third order effects that would eventually lead to Detroit's decline started in 1945 and that was the advent of suburbanization. So we see in the 1950s, the population hits 1.85 million people. And that's just the city proper that doesn't include the outlying areas, the suburbs, other towns around Detroit. And just in the city itself, there are almost 300,000 manufacturing jobs in the city. Throughout the decade, Detroit becomes one of the wealthiest cities in the world and the family car becomes a quintessential part of the middle class white suburban American dream. Ironically, the exploding popularity of the automobile repel suburbanization, the construction of freeways that divided the city along economic and racial lines greatly contributed to the struggles that the city would see in the future. As we get into the sixties with a large and diverse population, Detroit continued to have a significant cultural influence throughout the sixties. Motown changed popular music. It found success across racial lines. And throughout that decade, they had 120 top 20 singles just from Motown from artists like the Supremes, the Temptations, Marvin Gaye, Stevie Wonder. But this kind of betrays a lot of significant internal struggles. Plants continued to move to the suburbs and they brought the jobs with them. Detroit, one of the most diverse cities in the country at the same time had become one of the most segregated including in the US South. Social assistance programs and affordable housing were withheld by local politicians even when offered by the federal government. And mostly this was pandering to suburban populations but pitting them against each other sort of limited the poor into crowded slums. And even middle class wealthy people of color found that they were not able to move into these new communities that were being built because of a practice that later became called redlining. This led to a lot of tension. When you have people not living amongst each other, people thinking of each other as sort of a different entity. Tensions increased. In 1967, the 12th Street riots became one of the worst riots in American history. 1700 stores were looted, 1400 buildings were burned and 43 people were killed. The National Guard and the US Army were brought in. They used tanks and machine guns to put down the protests. The city's population began its long decline. As we get into the 70s, Japan's economy was booming. Japanese car companies were using new production techniques and they were just beginning to approach parity with the quality of American vehicles. In the early 70s, in the United States, 400 CC, I think that's six and a half liters. V8 engines were common in most family sedans and you would be expecting to get about 15 miles per gallon in a typical commuter vehicle. And then the 1973 oil crisis hit. That's a totally different story but what it led to was massive gas shortages and people couldn't get gasoline and when they could it was very expensive. This sort of kicked off a positive trend here towards more efficient vehicles but American vehicles were already very heavy. They consumed more fuel and it took time for the Detroit automakers to sort of adapt to this. At the same time the Japanese cars were already very efficient, they were smaller and so for the first time in history, those Japanese cars posed a serious threat to domestic manufacturers in the US market. As the industry faced new competition, the automakers here reduced their workforces, the population of the city continued to drop. When we get into the 80s and 90s, there's a shrinking tax base, white flight continues, the suburbs are mostly maintaining their population at the same level but the city's population is declining really rapidly and the city government is having a hard time keeping up. Job losses are increasing as more and more plants are moving out to suburban areas or closing altogether and as we see that we see more poverty and more crime in the inner city. One thing that started as just a mischievous tradition in the 1930s was this idea called Devil's Night where the day before Halloween, people would commit mischief and it was mostly graffiti and minor acts of vandalism but by the 80s and 90s, this was sort of a tinderbox just waiting to explode. These turned into huge acts of absolutely malicious violence. There was rampant crime and in 1984, for instance, in a single night, Devil's Night over 800 buildings were burned. These arsins continued, there were several hundred arsins every Devil's Night throughout all of the 1980s and 1990s. We get into the 21st century while some American cities did see a trend of reurbanization in the early 2000s, Detroit's population dropped another 20% from 2000 to 2010. In 2008, the mayor was forced to resign and he later went to prison on federal charges relating to corruption. And by 2010, the city of 700,000 now, half or less than half of its previous size, has 80,000 vacant homes. In 2013, it's the largest US city to ever declare bankruptcy. And in 2016, Detroit's population declined, caused it to fail to make the list of the 20 largest US cities for the first time since before the Civil War. In 1850 was the last time that it wasn't in the top 20. Professor John McDonald from the University of Illinois sort of analyzed this in an academic paper around that time and concluded that Detroit suffered more than similar industrial cities for three main reasons. One is that the economy was dominated by a single industry. Two, the political structure encouraged urban decay. And three, the competition for labor and capital combined with racial animosity led to division among the population. Now, obviously, there's no single paper that can show all of the comprehensive reasons that all of this was happening, but this is a fairly well-regarded summary of what was happening here. And since then, things have improved. So the auto industry is doing a lot better. And Detroit has begun to diversify its economy and you're seeing some tech companies starting to build offices here. A number have a presence downtown. Downtown Detroit, as I'm sure you've probably noticed over the last few days, does not look like the sort of war zone you might have seen on TV and pictures from the 90s or early 2000s. And a big part of this is thousands of abandoned buildings that were haven for crime, arson, and other activity were torn down. And now we're seeing some urban neighborhoods are becoming finally less segregated and city services are improving. So now that we're caught up, what do we learn from that? Where do we go from here? The city that built itself up from the underground railroad in the 1800s, that was a production powerhouse employing women during the war that brought people of color into the middle class. It became notorious for redlining, for white flight, for racist police, for indiscriminate violence. So I think the first lesson is the importance of just avoiding tribalism, the importance of creating inclusive communities. And the most important justification for that is the moral one. And I think that's self-evident. And if it's not, like I'm really not the right person to explain that. But there's a secondary argument to the ethical argument and that is an economic one. And I think this is actually really useful in itself because it's compelling to even an amoral, faceless corporation if they can gain from doing the right thing. So employee marginalized groups and increasing pay during boom times sort of created this new middle class for Detroit's products. It wasn't just the right thing to do, it also made them rich. Paying fair wages was sort of a quintessential part of Henry Ford's proposal that I think we can learn quite a bit from. And when they employed mass production, it reduced the barriers to entry for joining the labor force. You didn't need to come from a family of skilled tradesmen in order to be able to build a car. You could show up and be ready to work and earn a decent wage doing that. This, in my opinion, sort of parallels the priesthood of computing that was seen in sort of the 50s and 60s. You had like large universities, defense organizations and a small number of computer makers that controlled all of the computing equipment in the world. And if you wanted to work with them, you had to work for them. When we saw the advent of the personal computer, it reduced those barriers. And so by the 80s, it was I guess a little bit better. All you had to do was drop out from Berkeley or MIT and you could work on these sorts of things in the valley. And it's improved even more since then. Now we have open source and this can be an incredible force for democratization of computing. The barriers to entry that used to exist don't have to be there to join open source projects. Of course, education can still be enormously valuable. It's just not required as much as it used to be. And of course, this shouldn't mean free labor. The success of open source still depends on contributions from wealthy companies that drive benefits from this. Another thing I think we can learn from the story of Detroit is in supply chain management. We saw the rise of Japanese automakers were rival to Detroit's. And one of their advantages was something called the Toyota production system. And parts of this were imitated to varying degrees by automakers in the United States. They didn't have this sort of all-encompassing philosophy like Kanban, which just means signboard. But really what it means was a combination of just-in-time manufacturing and a few other philosophies. So manufacturing sometimes needs to verify that parts came from the expected source. And the reason that this was introduced by Toyota was to ensure that they weren't overproducing. They got rid of warehouses and instead they were just going to produce enough parts for the line to consume them. And so they had to find a way to track what was being built and when and they introduced the Kanban system. And this is actually, I think, much more valuable in software than we might realize. Not because we need to make sure that we aren't storing too much bits, I guess. I don't know, we don't really need warehouses for that. But what we can do with this is identify what's going into the supply chain. And of course it is still a problem for manufacturers of physical goods who are going to keep track of things and make sure that authentic materials and authentic parts are going into their supplies. But it's not as much of a concern from a security perspective for them. Because in order to make fake parts, you need to buy land, build a factory, hire workers, make the faulty part, and then somehow sneak it into the supply system. So that's less of a problem for them. For us, somebody needs to sit down in their bedroom and log in to GitHub. So I think we should emulate the Kanban system for a slightly different reason. That is to identify what parts are being produced or what components of software are being produced and where, and make sure that we're only putting safe, secure software into our end products. So doing this right now means an S-bomb, software bill of materials. And if you're at the security conference that sort of preceded this one, you probably heard that a bunch of times. If you're in Valencia, you probably heard it hundreds of times. There's this huge cottage industry of S-bomb generators and supply chain wizards that have sprung up now. And so it gets a lot of valid criticism. I think that people think that the S-bomb is going to be a miracle cure for supply chains and it's not. Just like the Kanban system and its imitators don't prevent mistakes in manufacturing, identifying the components of a supply chain doesn't guarantee your security, but you need to make sure that it's up to date. It's integrated with your deploy or release pipeline so that you can see what you're actually producing and where it's all coming from. Also the components of a specific build are immutable, but what we know about these components changes over time as new information becomes available. Vulnerability information needs to be tracked dynamically even though the S-bomb or the component list is a static snapshot in time. So in manufacturing, you might find severe defects after the fact that threaten the safety of a vehicle and will lead to a recall. In software, we often find that a component has a bug affecting security or maybe reliability and we need to identify which runtimes are affected by this component. Which ones contain that component? And to do this, we can aggregate the information from an S-bomb into a queryable dependency graph. Unfortunately, even though there are some vendors that will help you with this or you can roll your own, I don't see any kind of an industry standard for this yet, but I hope that becomes the next evolution of the S-bomb soon, because I think it'll really show people what the usefulness of that can be. And so I'm gonna use another word from the Toyota system here and I think this is Jadoka, I'm probably pronouncing that wrong. And I feel weird departing completely from Detroit's terminology to talk about Detroit, but all of the domestic automakers sort of emulate the same thing, but they just didn't wrap it up in like one cool word that I can use on a slide. So this is where you might have, if you're, for instance, putting interior components into a car, you might have a chromatic sensor that tells you if your trim is the wrong color. You might have something that's going to detect and ensure that the screws or rivets are in the right place where they're supposed to be, that they're present, and if they're not, it's going to cause a fault. It's going to stop the line and it can't move from one station to the next until you correct the fault. We're probably pretty familiar with this kind of thing because most of us have probably written unit tests before, but I think the key takeaway is that you need to enshrine a process where a fault in one of those components guarantees that it won't end up in your end product. And I think a lot of times we don't do that right now, either through insufficient detection of the fault or it's detected, it throws a warning somewhere, it's going to be in your build system logs, but the artifact at the end still gets produced and it might be accidentally used in production in the future. To detect problems as early as possible, you can use static analysis tools, things like SEMGRAP. There are lots of common linters that you can use and at Shopify, we've created a tool called QBotIt. We can use that to scan Kate's manifests and identify any that aren't following best practices. And the real takeaway here, of course, is the action that needs to be taken after all of this is done, which is alert someone, stop it from going into production. Standardized builds have been part of manufacturing for over 100 years. On an assembly line, every single machine is a documented part of an industrial process. Manual steps are defined in detail according to a work instruction. You have supervisors and you have automated systems that ensure each step is completed according to the specified process. But standardized builds following an exact predefined process have not been common in tech for whatever reason. We're finally making some progress here with things like Intodo and the Sigstore project that are really good. The Salsa framework is sort of pushing people to start using these things. And I really hope that more people will pay attention to this. I hope that we're able to make more progress on this as we move forward. We saw in the late 1980s that the prevalence of onboard computers led to the need for onboard diagnostic standards. And we can see here a scanner for OBD. I think that might be OBD too. This kind of system can tell you about problems with your exhaust, your timing, your fuel injection system, misfiring and others. And for urgent problems, it triggers the check engine light so that you know to either take a look at it or go find a mechanic who can look at this for you and identify what the underlying cause of the problem is. In this case, the user is the driver. They see it, they know something's wrong. And I'm sure they will diligently stop the car immediately and have it towed to the nearest mechanic. We can use standardized metrics and things like Prometheus, open telemetry to gather information about error rates, latency and leading indicators of problems. If we miss those, then we can see in the logs if we have structured logging, we can compare different applications output or even the same application over different instances output and see what they're doing, identify problems sooner. And then we can use real-time monitoring, alerting to ensure that when something does go wrong, we flip on that check engine light. We saw in the history that outsourcing was a major problem for Detroit automakers. It contributed pretty significantly to the decline. It's common in manufacturing. It's common in IT and in some corners of our industry. I think most of us probably haven't been affected yet in our roles, but will it come for us? And if so, what can we do? So I think one of the most important things we can do here is let's not assume that we're more valuable just because of the country or region that we live in. Even when we have specialized hubs that seem to create enormous value, those can change their fortunes pretty quickly. So find your niche, be flexible and take inventory often. You need to know what your competition is doing and where they have more strength than you do. One of the leading factors in Detroit's decline was that there was an enormous proportion of its industry filling the exact same role in the market. It wasn't just building cars. It was building big, heavy cars. It was building cars that used a lot of fuel. It was building similar cars. And so when that paradigm was disrupted, it spelled enormous disaster for the city. The Detroit automakers were really completely taken aback when Americans started buying Japanese cars en masse. They understood already that there was a niche market for certain German manufacturers. They understood why people were buying Italian cars, but they were totally surprised that people were starting to buy Toyota all of a sudden. And we need to keep up with the times, but let's also be careful. Some changes are just obscene as anyone who's compared to 62 fastback with today's monstrous Mustang can surely attest to. So Detroit created opportunities for marginalized groups. It elevated the working class. It pioneered the most important techniques for standardizing production that we know of today, and it successfully imported a lot of additional methods from other manufacturers. Its enormous production capacity earned it the moniker arsenal of democracy during the Second World War for its outsized contribution of armaments and equipment. But the economy was over dependent on the auto industry. The population became segregated by race and class and a series of leaders pitted residents against each other for political gain, sometimes corruption, by emulating the factors that brought Detroit's success and cautiously avoiding the traps that it fell into that led to so much struggling for its residents during its decline. Open source and tech at large can be more resilient, more successful, and more equitable. Open source in particular has the power to democratize vast opportunities presented by technology in the 21st century. If we fail to heed the warnings of mature industries that have already experienced this or doomed to repeat their mistakes. That's all I've got, so thank you. Please go party. Thank you.