 So if you haven't noticed yet, there is coffee in the back of the room, thanks to our friends at Databricks, VMware, and Mattermost. Would you like a bottle of water or something up here for you and Karsten? For those of you getting in just at the end here, just a reminder, we do have coffee in the back of the room from our friends at VMware, Mattermost, and Databricks. Cool. So let's go ahead, we're just a minute past 10, so we'll go ahead and get started here in just a second. We've got a couple administrative items and then as folks join us, we'll kick off with our keynote today by Demetris about making our community more diverse and inclusive. So thanks again everybody for joining us for the second, for the Sunday morning keynote here at the Hilton LAX for Scale 19. As a reminder, we have another keynote closing this afternoon with Dr. Vint Cerf and if you're not familiar with him, he is one of the creators of the internet. I think all of my early reading about the internet and CS and as I was getting started, I'll talk about him. So this is a very exciting moment for me, also with Demetris as well as we talk about how to grow our own communities. If you haven't had a chance yet to head by the Expo floor, there's plenty of time after this session as well. Please do go by and thank our sponsors for helping make this conference possible as well as all the community groups that join us to present, to share about their open source projects, their lugs, their meetups, their initiatives, they all put in a ton of effort and so wanted to take a minute to also share some of the winners of the Scale exhibitor prizes. We surveyed you all about what you enjoyed at the conference, what you enjoyed on the Expo floor and the ballots and results came in. And so without further ado, we'll drop these off at your booths afterwards, don't feel like you have to rush the stage. The people have spoken and OpenSUSA has won best presentation at their booth. Our friends at the Linux chicks have won the prize for best spirit. We are very thankful to Spot and the rest of the AWS team for all of their support this year and they have won most memorable booth. If you haven't been down there, AWS is doing some amazing things with open source, they've also got some fancy sodas for you and other cool swag. So go on by and talk to them about how they're helping build a more open cloud. GitLab has won most interactive booth and so it sounds like there's some fun interactive things to do at that booth. I encourage you to go by before the Expo closes at 2 and try those activities out. And then finally, people really love the swag that came from the CNCF, the Cloud Native Computing Foundation. They were working with a bunch of Lego sets of all their favorite Cloud Native characters and I saw some people building some exciting things with them both at game night and on the Expo floor. So again, go check them out as well. So we will bring those certificates by your booths later on, no need to rush the stage just now. And so on that note, again, we're for next year, we are planning to have scale back in Pasadena. I know everybody enjoyed this blast from the past here at LAX, I hope you got your in and out in and anything else that you enjoyed from this particular area. But next year we will be back in Pasadena with all of the fun activities that that brings with it. So again today, we'll be here until 4 at 3 o'clock. We will have Dr. Vint Cerf's presentation about how open source helped build the internet as well as some fun raffles and giveaways. We've got a new laptop from Purism that you have to be in the room to win, so I encourage you to come join. And a closing sort of happy hour with drinks and snacks and other things. So again, we'll see you all here at 3 to 4, give or take. But let's transition to what you're all here for this morning. I know we all braved game night last night and stayed up till the wee hours playing video games and winning at casino games and escape rooms and such. But we're here this morning early on a Sunday to learn all about diversity and inclusion. And it's one of the most, I think it's an important topic for us as a community as we try to figure out how do we continue to grow it. We can't just continue to build a community of just ourselves. There's lots of friends that have come from different backgrounds, different perspectives, different areas of interest. So we thought it was an important opportunity for us to learn with Demetrius and later in the conversation with Karsten how we can all continue to grow and contribute to that diversity and inclusiveness. So without further ado, I'd like to introduce Demetrius, who flew in from a very special family occasion just to join us today. And she'll be sharing with us a little bit about the work she's doing at GitHub and her past and elsewhere. And so thank you very much. I appreciate you joining us. Thank you. Thank you so much, everyone. I'm delighted to be here. Oh my gosh. I don't think I've ever got a yell before at the beginning, so I appreciate that. So wanted to let you all know just so you'll know you'll have time to ask us some Q&A. I'm going to speak for about 20 minutes here. And then my friend Karsten is going to join me for some Q&A to hear. And I have been more of a fireside chat type style. And then we're going to save about 20 minutes for you all to ask any questions that you may have. So anything that comes up for you during this presentation, please just jot it down and I'm happy to answer that afterwards. So OK. Let's get started. Let's open source diversity and inclusion and figure out what the heck did I mean by that. So again, I am Dimitris Cheetham. I'm the Senior Director for Diversity and Inclusion Strategy at GitHub been there for about two years now. And I have to tell you about my time at GitHub, specifically when I was interviewing. So I was interviewing with Erica Brusha, who sends her love and regards for scale, by the way, when I told her I was coming here, she was like, Oh my gosh, I remember when it got started and all these things and they are my friends. So she told me to make sure that I told you all hello. But Erica Brusha was the COO of GitHub at the time where I was interviewing. And so she was interviewing me and she asked that all important question. You know the question that's going to come up in every single interview you do. She said, Dimitris, why do you want to join GitHub? You see, she had just finished talking to me about how GitHub was actually approaching diversity and inclusion in a much more holistic way. Most of the time when you hear about diversity and inclusion, they have it right in HR, which is what I used to do at Red Hat and it's so important to be there. But GitHub was now approaching it from looking at their platform. When I started, it was 50 million developers a little under two years ago. And now they're at 83 million developers. They were looking at it from their philanthropy or their social sector, their giving. They were also looking at it from people, which is their HR perspective. And they were also looking at it through their policy efforts. And so when she asked me why did I want to join, I gave her this little tongue in cheek answer. I said, Erica, I want to join GitHub because I want to open source diversity and inclusion. And I smiled at her and she smiled at me and I guess it was a good answer because I'm here. So after I started, you know, they let me go through my own boarding, you know, figure out where the virtual bathrooms were on Slack and all those things. And then all of a sudden she said, okay, Demetrius, now go open source diversity and inclusion. I was like, oh, you remember that? What does that mean? I didn't know. So I just started doing what I love to do anyway. I just started talking to as many people in open source that I can. I got on Twitter, anybody's name that I saw talking about DEI and open source. I was DMing them, asking them could they just have a quick coffee chat with me. And over the course of hundreds of conversations, there were some themes that started to come out. The first was inclusion happens at the community level. You see, when someone new comes into open source, their first one or two interactions in their first community really sets the tone on whether or not they're going to stay in open source. If they have a really positive and inclusive experience, they say, you know what, those open source folks, they're my people and they're here to stay. But if they have a negative interaction, someone ignores their contribution, someone doesn't respond to them, they say, you know what, I don't feel welcome here. And they're gone and researchers saying they're often gone for good. And so then when we started looking at the community level, I said, well, who's responsible for driving inclusion in the community? And overwhelmingly, people said maintainers or community leaders were the ones that's responsible. You have to think about maintainers or community leaders like your managers, if you're working for a corporation or organization, right? You can build all of these amazing corporate programs, these trainings that everyone will have to take, these mandatory, you know, tutorials and education materials. But all of a sudden, if you take these mandatory trainings and you go back into your team, your day-to-day experiences will undo anything that you've learned on these trainings. So it's the manager. You need me to stay right here? I'm not moving. Okay. And I'm a walker when I talk, so I'm just going to keep going back and forth. That's good. All right. So it was really those managers that set the tone for inclusion. So when I started talking to those community leaders and those managers, I said, well, what do you need? What can we give you? And they were like, please don't come with more trainings where they were like, we have education materials, there's podcasts, there's articles, every conference we go to. People are talking about it constantly and constantly. We are overwhelmed with material. And so I said, well, if you have all the resources you need, what's the issue? And that's where the conversation started to just break up a little bit. So when I talked to managers of smaller communities, they said, yes, we have the resources. We also have the influence with our community because we're speaking with our users and our contributors almost on a weekly basis, but we don't have the time and we don't have the bandwidth. We are all about trying to get hands on the keyboards, contributors from wherever we can get them from. We don't have time to focus on inclusion. So I said, okay, well, if this is what the community leaders of the smaller communities are saying, what about of the larger communities? And what they shared was, yes, we have the resources. We have the time. We have the bandwidth, but now our community has gotten so large, we can't easily influence the culture. If someone's not doing something they're supposed to, it gets a little harder to pick up the phone or send a quick message to them. So you have this tension point. The time in which you needed to focus on advancing inclusion in your community was the time when you didn't have the time to do it, right? So what do we do? But one thing, no matter what community leader I spoke to, large or small, they said, here's what we know. Regardless of what we do, there are still some really, really great barriers to entry from people even being able to get to open source. So they said we can build the most inclusive, fun community there is, but if we don't address those barriers, if we don't address that wall that's there, no one's even going to be able to get to our communities. So this was a classic case of if we build it, they will come. Not if there's that big wall that's there in the middle. So that's what we wanted to address. That's where the equity comes into play. And so that's what we've set out to do with this open source community called All Land. All Land is going to answer that question and do that thing that I said in my interview. We're going to open source diversity, equity, and inclusion. And we're going to do so through access, community, equity, and data. So we kicked off All Land this community about a year ago as a matter of fact. And so during that first year, we said we wanted to do three things in which we have checked off all three of those and moving forward. So I'm here to share about those three things. The first thing was we needed to kick off an open source diversity, equity, and inclusion survey. There were a lot of surveys out there. There were many for individual communities, but there wasn't really a broad survey that kind of cut across all of the different open source communities. We had done a little bit of survey work on our platform at GitHub, but I made sure to let them know GitHub is important. We are a major player, but we're not the only player. There are so many communities on other platforms that are in other forms, and we need to understand what their sentiments are about diversity, equity, and inclusion, if we're going to advance inclusion and open source. The second thing was we kicked off a maintainers listening tour. Again, if maintainers are the ones that's instrumental in advancing diversity, inclusion, and open source, we really need to hear from them in a more structured and formal way. And then the third thing, which I'm so excited to tell you about because we are almost finished with this one, we kicked off an all in 12 month pilot, and this first year was focused on students, and we're getting ready to kick the one focused off the one that's focused on maintainers. So let's talk about this survey. So again, we didn't want to just do it right on GitHub's platform. So we partnered with the Linux Foundation, and we introduced this survey over 60 some odd questions, close to 70 questions. And if you are a data nerd like I am, please go and read this 64 page report. If you say that you don't have data for open source, diversity, inclusion, and open source, you have not looked at this report, please go there. And so when we did the survey, it was some surprising things that came out for me because I've heard so much about how open source is not a welcome inclusion. There are some really big issues of things that's making it so that it's not a great place anymore. But when we surveyed, and it was a representative sample, we found that 82% of survey respondents said, I feel welcome and open source. They agree with this statement. I can tell you this, if you go to any company and they get an 82% approval rating from their employees, they're going to be popping bottles and there's going to be confetti raining from the ceiling. That is a darn good number. But guess what? That's not the work of open, that's not the work of diversity and inclusion. We needed to give voice and hear the stories of the 18% that did not agree with that statement. What were their stories? What were their experiences? What were they facing? Because I can tell you, wasn't that 18%? That's probably what's keeping millions of others from even attempting to come into open source. Those are the stories that they're oftentimes hearing. So we needed to hear from the 38% of non-binary or third gender contributors who said that they didn't feel welcome and open source. What are the 29% of people of color, specifically in North America? What are their experiences? 26% of women said that they do not feel welcome and open source. 25% of persons with disabilities. And we can't forget about that 15% over there of men that said that they did not feel welcome and open source. And amazingly enough, that number was composed mainly of white men in North America. And so that was something that was just mind blowing to me. That's where we had to dig in. That's where we needed to see what was going on. And so we got over 10,000 written comments that our data analysts went through. And here's what we found. Oh, you told me to stay still. I started, sorry. Just need to like one of those little stick things up here. OK, so here's what we found. Most people did not feel welcome if they didn't have the technical skills and knowledge that they felt was necessary to contribute to a community. But the interesting part about it was when I talked to maintainers and community leaders, they were sharing with me all of these very important and critical responsibilities of a community that did not require technical skills. Things that people could do while they were building up their technical skills. So why was it that when someone was coming to the communities without the technical skills, they were made to feel inferior? That was something that we knew we needed to look at. Lack of response and rejections to contributions. The interesting part about it is, well, contributors were saying, I will send an email or a response or a message to the community leader and would never hear back. Or they would wait three or four weeks before I could hear back. And so what the maintainers, especially in those smaller communities, they were like, we're drowning here, we're busy. Like, we don't have time to respond all the time. It's not because we're disrespecting them or don't value them, we just don't have the time. Your new contributors don't know that. If you don't have your documentation that's setting up expectations for when you can respond, how to engage, they're thinking that you just don't want them into your community and nothing could be further from the truth. Along those same lines, especially amongst women, they just felt like their voice isn't heard. Their contributions aren't valued. There was one comment that really, really stuck out to me. And this person that responded to the survey said, if I am not white, if I am not male, if I am not rich, if I am not highly educated, no one wants to hear from me. And we know that an open source, that is not the case. But that's how a lot of the people that were responding to the survey were feeling. And finally, microaggressions and stereotypes and written communications. This was a really, really big one, especially when you started looking at international respondents. So you have really high numbers of feeling welcome in India. You have high numbers in the continent of Africa. But when you started looking into Spanish-speaking countries, especially South America, Portugal, Brazil, in which the numbers that are people contributing to open source are going up higher and higher every day. They said they didn't feel welcome at all. And most of the time, there were language barriers, cultural barriers there. So those are some of the things that we need to be thinking about as well. We have to get to the point that an open source inclusion is not the exception. It is the norm. Like it's not looking where we're seeing, you know, giving awards for people being exemplary. Like everyone should be doing that. And I think that it's possible for us to get there. So the second thing that we did was the maintainers listening tour. And so we kicked off a series of focus groups, individual interviews, as well as an online form for people to submit feedback. We asked them questions. What do you need? What are some of the challenges? What's working? What's not working? What are the gaps? We got so much rich information in that. And so I released a talk and you can find it on our website on some of the top six things that we've learned, like those six themes. But I can tell you, we're doing the final report of it every day. I keep looking. I think we were supposed to release it back in May. And I was like, it's too much information. And they keep telling me to cut it and cut it. I was like, but I can't. So we're trying to figure out the best way to release that. Your maintainers and your community leaders, they actually have the answers. They've just never been given the opportunity to communicate it. And so I have a meeting next week with some leading researchers at some universities because I said, I need you all at the table. They're giving us the sentiment. You all need to give us the data and the research. Then I'm going to the corporations and the foundations to give us the money to build the solutions and tools that they're asking us for. Like one of the things that they said was, hey, everybody's telling us how to solve for diversity and inclusion. And everything they tell us is this long, manual process that'll take us hours to do. They were like, can somebody build a tool or a script for that? And I was like, oh, we are developers, aren't we? And we just need to give someone money to develop the tools. So at GitHub, we definitely will be kicking off a major grant program for developers to build some of the tools that these maintainers said that they need. And we're going to make sure that they're freely available to everyone that needs them. So I want to talk about the next thing. All in a 12-month pilot. We were going to do one for the students and the maintainers within the first year. But we got so involved with these amazing students that I'm going to tell you about. I told the maintainers they have to wait. And we're going to kick that off in a couple of months now. So with these students, we went to universities that are usually overlooked and no one is showing up on their campuses. Those are schools like historically black colleges and universities, and not those top five or six, but those ones that no one's going to. And we also went to a school that was founded for the education of Native Americans in the United States. And so we went to the schools and said, we don't want to put up an application process. Because even filling out an application or an essay, that's a privilege and a barrier in and of itself as well. We want those students that really deserve a chance. And we asked the schools to select them for us. That's all we did. We said, send us their top five students. And we're going to work with them. We're going to give them, during their fall semester, they got open source education, which was open source one-on-one, learned about the language. What is a community? What is a repo? What's a maintainer? How do you make your first pull request? What do you do? All of those things there. And we offered them professional development because you can give someone the technical skills. But there are some things that all of us wish we had known when we first started in our first job, especially in a corporation, right? Like what do you do when you're the only one on a team? Like how do you deal with microaggressions? How do you leverage your employee resource groups? How do you deal with equity when you get it? Most of these students were from very, very underserved communities and under-resourced. When they come out of school, they will be making more money than pretty much everyone in their family. So we wanted to make sure that we even gave them financial education. So that's what we focused on during the fall semester. During the spring semester, we then partnered them with Major League Hacking so that they got experience for 12 weeks actually participating in an open source community. We just didn't give them the theory and the academics and just put them through training. They got experience actually sometimes of learning how to code for the first time, even though they were sophomores and juniors. And they got experience with actually presenting their code, learning their communication skills. And all of this was to get them ready for a summer internship experience with one of our corporate partners. And so we are in week eight of 10, so the students are actually almost finished with their internships. When we started, over 90% of the students in our pilot had never heard of open source before. Over 90% had never had a summer internship before. And I'm proud to say that we place all of them, 100% of them, many of them in their first open source internship experience. And I have to tell you about these amazing students. You know, one of the things about it is if you go to any school, if you get those that's in the top 10% GPA, they'll find their way to an experience, right? Someone will eventually hire them. We knew that GPA is not the only aptitude for success. The students in our program, some of them are commuting two hours each way to school every day because they can't afford to stay on campus. Some of them are full-time caregivers, especially during the pandemic, where they have to live at home, or they have students, or they have kids. We have full-time parents that are in our program. You have students that are working part-time, full-time, or they are active duty military because that's the way they're paying for college. And then you also have some that were on academic scholarship because my question to them was, why don't you have an internship? And they were like, we have to train during the summer because otherwise, we can't afford to go to school. We're here on scholarship. And so I'm like, so if you are a scholar athlete, that takes you out of the ability to even be able to participate in an internship. That baffled me. Guess who can solve for that? A very asynchronous community like Open Source, right? So we have some of our students that were on teams that might be in another time zone so that they can work flexible hours. Some of the students had to pick up college courses and we managed to work around their schedule. That's what it meant to be all in. That's what it meant to give students who are off the overlook who just need that shot. We just wanted to crack the door open for them so that they could just have an opportunity. And we did that through the amazing partnership of schools, organizations, and corporate partners. So whenever I show this slide, I say this is what it looks like to open source diversity, equity, and inclusion. And we have plenty of room up here for other logos and there's some that's gonna get up at it already. But then when I look at this inaugural class, this is also what it looks like to open source diversity, equity, and inclusion. This is our first cohort of graduates of all in. And guess what? We started with 30 in the pilot. We are growing this year by 10X. There will be 300 students that will be going through all in. And I have to admit I settled on 300 because Microsoft and GitHub was like, when are we getting to 1,000 and 10,000 please? And I was like, whoa, whoa, whoa, wait. Because one thing about it, I just told you about the experiences of these students, giving them technical education in a training course, that actually was still some gaps that we didn't see. That's why I wanted to keep the pilot small. It never occurred to me, it does now, that if you have students that never had an internship and you're gonna throw them into a corporate internship for the first time in a virtual experiment, what does that mean for them? They're like, oh, you wanted me to show up at work? Like every day at nine o'clock? I thought this was like a class schedule or whatever. Just things like that, their elevator pitch, communication, there were some students that they needed more of an intensive program. So next year, what we're gonna do is for some of those students that need more of an intensive experience, we're not just gonna send them their first year to a corporation. We're gonna build, I'm gonna make this name up right now, like the Open Source Academy, where for 10 weeks, they will be with us getting more and more in-depth experience, more and more training, giving them what they need. That's what it means to be all in. They are not a number to just push through, push through a program and say you graduated, we're gonna get them job placement. If they need training, if they need training for public speaking and presentation skills, we're gonna get them that. We have some students who identified as neurodiverse. We got them with training with a neurodiversity coach, someone that was neurodiverse themselves, because we didn't want them hearing from anyone that they needed to train and assimilate in order to fit in. We did strengths-based coaching. Let us understand who you are and what you need, and then we're gonna help find the right opportunity and the manager for you. And when I say that I figured out how to get them that coaching, it was through my friend Karsten and Sam Kanuta at Red Hat. I called them. That's what Open Source and Diversity and Inclusion mean. I'm not sitting here in a silo trying to solve a solution for something. I'm calling on friends, everyone in this room, to help us. And so I wanna make sure that what you're hearing today is understanding that this is not a program, it's not a set of programs. All in is a community. It is a community of people. It's all of us coming together with your best, your thoughts, your ideas, your energy, your resources, the money from your companies and organizations, jobs, whatever it is. That's what it means. So we need all of you to participate. Here's how you can participate. You can join the community. Go to AllInOpenSource.com. If you are a company or an organization, you can agree to hire interns next summer. Or if you wanna contribute financially, we have ways that we can take your money too. And we need it. Because all of the students, not only did we give them those things that I talked about, each one of them received professional head shots. We actually had a photographer go to their schools because a lot of them didn't even have head shots to go onto their LinkedIn profiles. And we wanted to make sure that access was not an issue as well. Every student received a free MacBook as part of the program as well. Because what we learned were some students didn't take internships over the last couple of years because they didn't have a laptop at home. So we have to make sure that we're addressing those things as well. So lots of opportunity for you all to join. And so please know that I extend this invitation. Less open source diversity and inclusion. Thank you. So I'm gonna be joined up here for Q&A with my friend, Karsten. I moved all these on because I thought I was gonna be walking all over and then they told me to stay still. So do we need to go over there? We just need to sit down though, Karsten. Well, you can swing your legs if you want to. And so we have about 30 minutes but I wanna make sure. So Karsten is gonna ask me a few questions but if you all have questions just raise your hand and we'll interrupt here as well. Yeah, thanks. We'll see if any of these questions are ones that people got in their mind and then we'll do the cleanup. Okay, well good morning. Good morning. Thanks for joining us at scale. This is one of my favorite shows that I've told you before. You've told me. Like five or six. Exactly. Yeah. So Jumita, I know that you've had a really successful career in serving in the chief of staff and in senior director level roles and what prompted you to launch the all-in open source community at this point in your career? Yeah, so when I, maybe about five or six years ago, I had the opportunity to really think about what I wanted to do next in my career. Typically I haven't had that space but one of the things I wanted to do is I wanted to go back to my first love which I was a computer science major in undergrad and through a series of twists and turns which we can talk about cause I think that's an issue with inclusion as well. I kind of got away from tech but then I said I wanted to go back and it was funny enough I moved back to North Carolina I signed up for my first program in class cause listen y'all I was a really good programmer C++ which totally dates me but I was like I'm gonna go and be a programmer again and I think I was like an hour into the class and I was like so I'm gonna be on the business side of tech again. And so then I just thought about what would really make a difference and you hear all sorts of statistics about people of color, women just being underrepresented in tech and specifically with open source I know how important it was for programs for me coming from a really poor background you know it was so funny when I graduated high school my mother with three kids in college was making $13,000 a year. And so I just think about all those program project breakthrough which is a program for HBCU students that IBDM had I was one of their interns project uplift there was so many of these impact for programs and it's so funny when I hear about those programs most people are like are they really effective? And I'm like ooh ooh ooh they're very effective. Like a lot of people like if you can't scale it if you can't help millions and millions of students it's not work the ROI. And I was like without those programs you wouldn't have a me and you wouldn't have a lot of my fellow classmates. And so I said I wanna join an organization which is gonna let me figure out what type of program I wanna create and that's why I created all of them. Yeah that makes sense. So then it's not really hyperbole to say that the advancing diversity, equity, and inclusion in technology in particular has been a part of your life's work, right? It's a part of my life's work. Yeah I live and breathe it every day and my daughter comes like every day she's like what are we doing with all of them? Mom she's 11 and so she'll go to school and start talking about it as well. And so this is what we talk about at the dinner table at night it's my life's work. Oh I love it. And so in your professional roles and your open source involvement were we affected those things so but what are some of the trends that you've noticed that you've got more as you've got more involved in this effort of trying to make open source more inclusive? Yeah I mentioned it briefly but one of the things that I was just I don't know I was astounded by it. There's a lot of people that's working on diversity and inclusion in tech and even in open source. There's a lot of articles a lot of there's a lot of money being spent. And so when I look at how much effort and finances are going to diversity and inclusion and then I look at the numbers as my 11 year old said the math ain't math then. It's like something is happening here and what I started seeing was that everybody's focused in the same areas like that top of the funnel. They want to give students internships or they said we're gonna do this or once you graduate then we're gonna have all these amazing opportunities for you. There are so many hurdles that students and people have to get to to even get to those internships. I love when a company says oh we're gonna go to this school and we're gonna give them all of these certifications for free that they can take and I was like but they don't have laptops and they don't have wifi on their campus. How are they, they don't even have professors that's teaching their coding classes. How are we gonna get them to your amazing certifications? But no one wants to do that work. That's the long term that's not the easy ROI. That's not the work you see in the headlines. That's where you might only impact five students at a time. No one wants to do that but that's where the equity lies. So that's when you start hearing a lot of people take away equity. They'll say let's just do diversity and inclusion or let's just do belonging or let's just do justice but they don't like that equity word because that's where the hard work is. Yeah and that's what struck me first when I thought I'm like wow how are we gonna work on equity? That was the, yeah that was a tough one and also when we first met you were in the middle of your maintainers listening tour and I know you've picked up a lot of things in a lot of ways. What are some of the interesting things that you would highlight or would call for all of us from that? Yeah I think for the maintainers I mentioned earlier like they kind of know what they need and they had never been given the platform for us to really listen to them. Some of the amazing things that I saw come out just in a listening tour especially in person there was one, I don't know if you were at this one it might have been in another conference or I did the listening tour where one of the maintainers she raised her hands small community and she was like one of the big things that I have is that we'll have someone that's new to open source they're coming to our community we give them opportunities, they get up skilled they start learning, they start doing really impact for work for us and then they're hired away by our company and we celebrate that but then all of a sudden we have this big hole or gap in our community that no one can feel and oftentimes they're doing a lot of the work that we need for them to do the advanced diversity and inclusion within our community so this one maintainer she said they had someone that was working on removing the dead names from their code and they were doing that and they got about 80% done 80% complete and then they were hired away and so that project has just been languishing for over a year now and all of a sudden someone from a very large community was like we have like 300 people working on that for us and now they're looking for their next project we're gonna send them over to you next week and they'll get that done for you and it was like a light bulb moment why aren't we doing more of that that literally is open sourcing like diversity that's open source and so one of the things that I found was just this partnership between it's not a competition people want to work together people that are saying Demetrius just tell us what to do and I promise you we'll do it they just say it was so many resources in there that were overwhelmed so those are some of the things that I saw that really really just struck with me on that piece of it and I remember another part of what you found and that really struck me as well was that you found the data that showed that people who were in marginalized communities or minority status were creating communities that were amongst themselves and some folks might see that as a positive outcome but what's the other side what's the concerning side about that yeah so just to give everyone a little bit of context when we were looking at the data from the open source survey we started seeing very high numbers of people of color and women that said that they felt welcome in open source if they were a maintainer and I mean it was like a really huge spike and I was like huh that's something to be celebrate like people who you know make it into leadership roles they feel welcome and so then we started talking to some of the maintainers from the maintainers listening tour and right at the end it struck me I started asking the question why did you start your community and what they said was we tried to join larger communities or communities where we were in the minority and we didn't feel welcome so therefore we went and created our own community over here and then I said are we creating separate but equal like not intentionally but that's what is coming out as and so now we have these larger communities that are like we need more people in leadership and those that tried to be in leadership are like nope we're not joining your community we're going over here and creating our own so what seemed like something that should be a great thing to celebrate it actually has like some you know some undertones to it that I think that we need to address so when I talk about joining with universities that's what I want them to research I like you know no offense to anybody in the research space and they are my friends are working on some amazing work with them but oh like we do not need another research project on how to create a more inclusive communities like there's like a million research papers on that let's start digging into that work that separate but equal work to see what's going on there that's when we can start you know finding solutions yeah I see can we save a question for the end thank you so well let's shift a little bit because just because I was thinking also about what you were doing when we first met was you were right in the middle of that first cohort of students and I mean you brought tears in my eyes in the pictures and I mean I love the professional headshots like that those polishing pieces are such a huge difference or overcoming biases at the first run but so and there are a lot of camps and educational programs and things out there that are for students so especially in technology and even in open source and what do you see is really the different for students for all in for students compared to a lot of the others yeah you mentioned there's a lot of boot camps out there are a lot of programs corporate organizations that underwise but I think not to sound cliche but we're all in with these students like we just don't stop it was I'll give you a prime example of what this meant it was right after the holiday break the students came back and they were supposed to go with major league hacking and you know do their I think it was 10 hours a week and we got an email from a student and the email was very short and curt it was like I can't do 10 hours a week so basically y'all need to figure something out and most people would have been you know organizations or boot camps would have been like oh you can't make the requirement you're out like it would have been no questions asked but I actually picked up the phone and called her professors I called the professors and said what's going on and they sent me the article where she was in a really bad car accident like concussion protocol she wasn't even supposed to be in school this semester but she said she has to graduate on time because she can't afford to extend for another year so here I would have if I did it just quick if I was just looking at her as a number she would have been gone from the program replaced with someone else who wanted the MacBook and you know the free stipends and the academic credit but I said I told these students I was like we're not letting any of you go I was like this might sound a little cultish but I will be all I will call your we've been on the phone with parents we've been on the phone with professors the deans of their schools we've been involved with them even during the internship process we are really really making sure that we go all in and really address the needs of their students we are not letting them fail but I can tell you that my boss reminds me of every day it's not sustainable like this like you're 30 students almost wore me out so imagine 300 students imagine 10,000 students so that's where we're looking now we're gonna be issuing some RFPs we're gonna have to have partners people that do this and do it well that bring what they do to the table we can't solve it all ourselves one of my pet peeves is when like companies try to like create a university like tech company I'm like you're not a university why are you doing that like go and give money to a university and let them do it well and so we can't be all the things right and so that's what we're doing with all in I think that's what makes it different we're not looking at it at a competitive advantage someone asked me when I first started this program he was a member of a C-suite of a company he was like Demetrius is an amazing program but you're open sourcing it what if somebody comes and steal your idea and just take it and they do it and I'm like yes that's what I want them to do because that means it's gonna be 30 times 30 times 30 times 30 and he looked at me like I was crazy he was like why don't you start a B-court nope I'm not doing it so everything that's in my head I am pushing it out and hoping that people not compete with us but join with us to do it and I think that's the difference with all in yeah and I really feel like you've caught that spirit of like what makes open source work when people are inclusive and being able for all of us to work together through this model I mean yeah that's what sold me on and for well you know I just was oh you're in Carson you're in I grabbed you I love it I love it so speaking of the similar thing of you know we've looked at the little bit of the wall of funders aspect and all those and the participation and where does the open source software funding fit in to the picture of all in we gotta pay people like you know I was speaking at a conference it was virtual at the height of COVID and someone put in the chat and they were like contributing to open source as a privilege and I was thinking that I don't know what I thought they were talking about but I actually picked up the phone you know and call the person and it was on Zoom and scheduled a meeting and she was like even during the pandemic if we didn't have time it's if it's not your full time job like some people still view open source as a hobbyist activity extracurricular activity so if you don't have all of the support that you need at home or if you're not working multiple jobs or you know all those things you don't have time to contribute to open source and so when I started talking to a lot of maintainers they were like there's some amazing people that can help them in their communities but they can't afford to pay them and they have other jobs and they have other commitments so they kind of come and go as they can and I was like I think with all the money that we make in tech I think that's like one of the easiest things to solve for but then you know a lot of times people think you need like buckets and buckets of cash one of the things that struck me when we were doing the maintainers listening to I was talking to some folks that were internationally it was a contributor in Africa as a matter of fact Nigeria specifically and I was like yeah we heard about lack of infrastructure you know and you all don't have access to data and blah blah and she said that's not our problem we got access to data all over the place and I was like oh well what is it she was like it's so expensive to actually connect to the data and that's why we can't participate and I was like oh well how much is that gonna cost cause I'm thinking we need to go build some power lines or something I don't know and she was like oh probably if you gave every person like that wanted to contribute to your community like $200 a year and that would get them unlimited data and I'm like y'all aren't contributing and aren't participating you're shut out for 200 bucks I was like what, what does that even mean and so that's where I think we have to really think about and look at and really break down what is funding, diversity and inclusion look like because a lot of people are thinking it needs to be millions and millions and billions of dollars but sometimes it can be as simple as $200 for data for a year or a MacBook or a laptop for them to do an internship and so we're solving for that even through our corporate sponsors program we get up. Well so shifting to just some questions about the advancing of diversity, equity and inclusion and open source and you know from the broad perspective as a lot of people know that contributing open source software for the first time can be it could be daunting for new people even if it's no matter how eager they are to be doing it from research what are some specific steps that projects can do like the Drupal community that to take the foster that more inclusive and welcoming atmosphere. For any community documentation is gonna be critical people are looking at your documentation as soon as they come into your community trying to figure out how they can contribute how can they be a part of you know whatever it is that you're doing have robust documentation and I think that's the thing that's most often overlooked and then the second thing I would say that we've heard a lot of is just community hospitality when someone joins your community you can have through GitHub actions by the way the shameless plug you can have it where as soon as someone joins your community it sends them an instant message that says hey welcome to our community this is how you can contribute we're glad you're here your voice is you know really really valued here just those small little steps like that that was the number one thing that we were hearing out of the survey to say it can advance diversity and inclusion I think a lot of people were thinking it was gonna say hey close captions for every meeting which is also very very important or you know all these other things that you might not have time but they were like no just let us know that you see that we're here and that could be something that's automated so documentation, community hospitality if you get those two things I think that you are going you know you're well on your way to advance and diversity inclusion in your community and then we can start doing some of the equity things as a docs writer that's maybe another reason why you won my heart over from the beginning with the proof in the pudding so from another perspective you mentioned how crucial it is for open source communities to be sharing and working with each other about their experiences with diversity, equity, and inclusion what are some of the mistakes that you've seen open source maintainers make with diversity, equity, and inclusion? I think the number one mistake that I've seen and it's not a mistake it's just an opportunity I would say you know if I had to change the word a little bit what is it? It was, it was funny and so I you know during the listening tour I was having individual conversations and it was always a maintainer of a large community or a well-resourced community and they were saying Demetrius we're doing all of these efforts we're you know having these spaces for people to come and learn about our community we're having these diversity and inclusion talks so that we can start identifying future leaders all these things that we've set up we're pouring thousands and thousands of year into this but no one's showing up we can't find women or people of color anywhere they won't come and so I'll ask them a question and I love asking this question it's almost like and I already know the answer but it cracks me up to see their face but I'll say well tell me about your marketing efforts for those events that you're having and they were like oh yeah we're sending out on Twitter and LinkedIn and I was like who's Twitter and LinkedIn? They were like ours, our leadership team and I was like let me see your friends list on each of these platforms and when I look at their friends list I was like they look like you like all of them typically are white men so if you're sending this out to a bunch of white men who do you think are gonna show up? You have to expand your personal networks and I think that that's something that is often overlooked for maintenance people join communities they're almost like friends that join together I've heard so many communities start from people meeting at conferences like this or they're hanging out at a bar somewhere it's friends and family members and they start this community and then they bring in the people that they know and the people that they know and even myself included most of our personal networks are people that are similarly situated to us so if your community if your personal network looks like this and that's all you're advertising to you're never gonna reach a broad and diverse audience to bring them into your community to introduce yourself so I would say make sure that you're going to conferences not just with your friends and the ones you know that you've been going to year over year keep coming to scale because we need to scale 20, 20, 21 or whatever but there's also other conferences that you can be a part of whether it's AfroTech whether it's some for the Hispanic community go there as well and start expanding your personal network and I think that's the number one mistake that I see people trying to advance diversity and inclusion with people that look just like them are from the same backgrounds are the same abilities. Excellent, thanks. Well, I'm mindful we have got a few minutes left I wanna make sure we have there's at least one question in the queue I did do I once gonna ask you one more thing about what you thought about the future of diversity and inclusion and open source so we can do we want to talk about that one or do you wanna go ahead and jump to I've got a question I've got Gwinnivere's I think over here it was one all the way over here first the mat, yep I think she was first do we have an amazing skirt by the way I think I need that Yes Yeah, so I really love the effort of going out to colleges that have traditionally been underrepresented how do you empower those students to advocate for themselves when they get to their first job offer they will they will be paid less than their more privileged counterparts that's just the numbers Yeah, how can we change that? Yeah, I think the number one thing is just arming them with information and that's why I said as part of our curriculum we went into what does finances look like what are typical salary ranges how can you start reaching out to people to talk about what different salary ranges than those things are but what I've learned through the summer is that the students they still aren't there yet we're talking about how can they advocate for themselves and we still have a lot of these students that are dealing with a lot of imposter syndrome just to ask them hey, what have you been working on this week you would think that's one of the hardest questions that they've ever heard before and so just getting the more and more comfortable hearing their voices in front of those we noticed that when we first started all in the fall semester they were working with a lot of folks on my team and folks that look like them from underrepresented background when we put them into and they were very engaged when we put them into major league hacking and those are they're my friends but mostly white people you saw the students just clam up inside of themselves and it was amazing to watch that but I was like we have to even get them more exposed to people to realize that they're gonna have to work with people from all parts of the world and they need to be comfortable in themselves and I think that's just practice and practice and it's having people that do it and do it well it's not us doing it that's why I said we're partnering with an organization and we're trying to find one that they know how to do this and do it well to get these students prepared so it's definitely a step by step process Thank you Yes Hi I was wondering if you could talk a little bit about resource allocation so I work on a lot of open source hardware and so one of the problems we have with our projects is in order to be a contributor you need to have access to the ability to put together a circuit board you need to have access to Fusion 360 to then maybe if you have a 3D printer or something along those lines so there's like individually each one of those things to your point about speaking about privilege and the ability to contribute now it's not just a time thing you also have to have a resource thing and so I'm wondering if you could speak about how one goes about expanding an open source hardware project to be more inclusive while at the same time recognizing that we are also strapped for cash and can't be oh here's a $2,000 care package to just give you whatever you need to start Yeah I think the number one way is making people know what the work is that you're trying to it's so funny that when you said that my ears perked up I have a student that they said that after the summer they realized they don't wanna do software they wanna do hardware and I was like I don't know where to tell you to go who's working on hardware and so like so us having this conversation and knowing what everyone's doing in these pockets of open source we can then connect because I can solve for the funding like I have this one student if you tell me he needs a whatever it's not like a million dollars or something right like if there's enough in his reason we can get there there's foundations that have been reaching out to us the Ford Foundation has generously funded us we're talking to the Sloan Foundation the Linux Foundation they have been all over the place and they're like if it's money tell us what you need but we've just been trying to figure out where best to get the money and so I think it's just us having these conversations and that's where open source and diversity inclusion you know that's where it comes into play so if your community says we wanna take on five students we would need $10,000 to take on five students and we can match then that's what it looks like when we're all coming to the table and bringing what we can bring to the table I'm sure Cisco Intel all of those companies that deal with hardware they would love to you know if you all are in corporate sponsor get help sponsors to be able to send money or a way to do that especially for those from underserved communities yes entrepreneurism and the networking say of churches and existing can you maybe talk about some of the strategies or ways that you think that we can approach building those type of resources yeah so I would say it's two things one one of the things that we've learned through this pilot we talked about corporate program you know the corporate partners getting the students for internships to these corporate partners not everybody wants to go work for a major corporation they wanna work for smaller you know organizations like startups they wanna contribute directly to communities because they have entrepreneurial dreams they wanna be a founder one day and so that's why for this next iteration of all in we're looking for not just corporate partners I probably need to change the name of that we're looking for any organization that's willing to host students because people want different experiences we're looking at schools in Washington DC and they're saying hey we actually wanna stay in DC are there government agencies that do open source and so we need government agencies there so the quant that you're making about you know churches and non-profit organizations we started the pilot with students because they were readily available demographic but we are gonna expand all in for everyone that wants this type of opportunity because we realize there are retirees there are community leaders there are people that are more advanced in their community and they want to give back so we're like let's teach them open source so that they can go to the communities that have these young folks that they can actually start bringing into the communities in places that we can't reach so we are definitely expanding all in so they can be for like I said retirees for career changers people that are currently and formally incarcerated I always say open source is the lowest barrier to entry in tech and so we need to be reaching out to more and more so we definitely have that in our roadmap yes I have a question there and a question here too okay we have time for maybe two more so we'll start here I'm Sam Coleman I had a kind of a complex question what it is is that when I did my doctoral research it was an open source technology in public education about six up seven eight years ago and a lot of people in this room had a pretty significant impact on my development in open source and all these people don't look like me and I love them and they may or may not know who they are I'm just gonna say that it's important that you know we need to recognize the level of impact that we have on people I wouldn't have become Dr. Coleman unless I had a lot of people in this room help me my question to you is this through your survey have you been able to identify the entry points into open source for people of color because for me I kind of stumbled into scale in 2004 I had no idea what I was getting myself into I just I was looking around like you know who are these guys kind of cool and so I was wondering has your survey gathered that data so that we can kind of analyze what the entry points are to maybe help hasten that process yeah we didn't ask that question specifically but we have enough data and some other questions that I think we can make some assumptions about some things because there was still a high number of people that they got exposure to open source through their undergraduate curriculum but then when we can start looking and there's been some research that if you look at historically black colleges and universities and minority serving institutions they don't have open source curriculum so there's a gap there so we started seeing an entry point that more people of color came into open source later in their careers it was after they had already been you know working for sometimes over a decade before they even got exposure to it and so that's what we were looking at and all in how can we infuse it more and more into curriculum and we're looking at Microsoft Tills which teach they train people to teach computer science education in high schools and so I told them I want open source curriculum as part of your curriculum now so that we can train people if you have a local high school in your community you can go get trained through Microsoft Tills where you can go once a week or once a month to actually introduce open source curriculum so we have some data and the data is open anything that's not anonymous and so we have the data set so I would love to partner with you just to look at your research to see whether or not we can start building some type of you know research and data around that and something we can solution oh awesome well I send me the clip now everybody sends me their dissertation I love you I love you love it okay last question oh my hi my name is Solonette I was wondering so you talked about a lot of the universities and things of that nature have you looked more at like technical colleges and you know community colleges that have the associate degrees because they have a lot of programs there to help bring people up and they kind of know what they're doing in regards to that so like bringing in this to them I think could be extremely useful I used to be a director at ITT oh please so and I totally changed the curriculum to make everything up on the internet so that they have all had websites and portfolios and they graduated into jobs and so it's like open source has that same thing where you can just graduate into a job if you got the code already there correct so to answer your question yes so the pilot we did at those six schools but for this full program this year we're expanding it to any student that goes to a minority serving institution so that's for Asian and Pacific Islanders women tribal colleges Hispanic serving institutions as well as historically black colleges and institutions they can go to a community college or they can go to a predominantly white college as well if they are from and underrepresented background and tech specifically focused on women and racial and ethnic minorities in the US and we are already talking to Brazil about what does this look like to start introducing this into South America as well so community colleges are welcome to participate this year thank you yeah I think that's it oh yeah you said last question oh and I'm over time thank you all so much so thank you both for joining us and for this fantastic conversation very insightful there's looking forward to hearing more about the program as it expands absolutely so with that thank you all again for joining us for the Sunday morning keynote and thank you Demetrius for joining us as well as part of the skill family supply it is in the mail it will arrive at your home and in Raleigh in a few weeks like I said supply chains are fun but in any case thanks again for joining us we've got the expo floor open to for another couple hours still two o'clock if you haven't yet done the expo floor bingo scavenger hunt passport game thing there's still a chance to win some fun prizes we'll be here again in this room at three o'clock with Dr. Vint Cerf and yeah looking forward to seeing you all here as well as in Pasadena March 9th through the 12th 2023 perfect testing test test hello hello hello everyone in 10 minutes we have the cloud native track starting we'll have Joe Smith talking about cloud native home labs so if you want to hang out and watch sweet if not start making your way out thank you all right uh... thank you everyone for coming this is the cloud native track uh... next up we have Joe Smith he is going to be talking to us about cloud native home labs uh... I am particularly excited because I too run a cloud native home lab so I'd like to you know trade notes and see how each of us does this so take it away thank you rock and roll thank you alright thanks so much everybody for uh... for coming to the talk uh... thanks first of all thank you so much to all the volunteers and the organizers of scale uh... you know this is a volunteer conference so I just want to give uh... all those committees a plug if you had a great time uh... if you're interested in contributing and pitching in uh... I'm pretty sure every committee possible could use it hand so whether that's the tech team the dot org committee uh... pretty much if you are interested in pitching back in uh... reach out to the uh... committee chairs they'd love to talk to you about that uh... and also thanks to the sponsors that we can actually do this uh... we can be wired with coffee which is great uh... so we're gonna be talking today about uh... running a home lab and what that looks like uh... in particular we're uh... we're gonna talk about sort of the journey that I went on and kind of how I like proceeded further and further away from hardware and how I got way further up in the clouds uh... you know being disconnected from what it actually means to like run systems and thinking about the actual hardware that's actually running the workloads you're using so you know it's great obviously we have all these different abstractions and so we don't need to be thinking about nicks and hardware and all the different problems we have with that you know we can think about services we can think about the different regions that we're running uh... workloads in and how we make sure that we're distributing that so that way our customers are able to connect that very quickly so when we're building these distributed systems we're able to leverage a lot of the tools and techniques that we've built out over the years what I found at least is that I was sort of losing that passion and you know there's something different you get when you're actually having to like fiddle around with some hardware you're really thinking about how am I going to architect this at a very low level and if you're thinking in terms of thousands of servers or tens of thousands of servers you kind of lose that like you know the feeling like the pet like oh here's my little like you know my home server this is the like hard drive that has all my like you know home videos or whatever that I'm saving and so I think there's a lot of benefits here that we'll get into uh... the way we're going to sort of build this up is through different layers so the first one we're going to really think about how are we approaching the infrastructure like how are we thinking about the hardware and sort of what's that base layer you know this is primarily the stuff that AWS, GCP is going to abstract for you that you're not really going to be able to leverage anymore once you're running in the cloud so I'm advocating that if you don't have a home lab I think it's worth your time pardon anything it's going to be you know you're going to learn a lot and you're actually hopefully going to be more connected to what you're running once that's in place the talk is actually going to pick up speed a lot I think this is where we're able to use a lot of the cool cloud native technologies that the community is built out and so we're going to be able to take advantage of that and then you know kind of what's the point of this like what do you want to build on top you know what are some of the different cool self-hosted projects that you can use once you have this infrastructure that's you know in your apartment or in your house I'm sure a lot of folks are going to ask this was the first question I asked when I saw slides like this there's an open source project written in Python called look at me which you use to generate the slides so if you're at all interested in that I'd recommend you look at this and then I used graph is to ask a generator for some of the diagrams that we'll be going into so I graduated from Chapman University down in Orange County in 2010 with big Rob who runs the network in the Wi-Fi here we we had a pretty good time and you know we're building a bunch of the systems that you know are still in use today which is pretty exciting he was telling me I think the spacewalk server we built 15 years ago is still down there and so you know going from really like tinkering on all of the hardware really like using mini-com and like serial ports to like flash APs think about you know hey the DHCP server and like Smith Hall is down I've got to go like truck out and like drop a replacement in there you know really getting that like hands-on experience I think really was beneficial especially as a student and then sort of over time you know going from Google and sort of like moving up the stack into the cloud you know again we're thinking in terms of large scale but you know we're moving away from that that opportunity and so you know to great pride I think I still have you know at school I was definitely that kid who's like so obsessed with Linux well can you Linux and open source you know whenever the yearly IT team would come by and like reprovision each of the computer labs I would like follow behind them lab by lab with like 20 live CDs and I would drop in like an 8.04 like hardy hair and disk and I would like dual boot everything which was great because a lot of people were using Linux for the first time even if it was kind of inadvertently which was sort of good so so I think you know trying to find that joy again is really the point of this so from a high level what we're thinking about building you know for me one of the goals that I wanted to do like the application that was my killer app that sort of justified all of this was something called task server and it's the synchronization system there we go synchronization system for like a to-do app that's primarily run in the terminal and it's very powerful very feature rich there's a ton of complexity in there and like there's several talks on YouTube which I encourage you to check out if you're interested but basically you can create different projects and have all this running here and it's great by itself if you just want to run on one machine like one laptop like this but if you have several laptops or server if you want to run it on your mobile device you need some way to synchronize that data and that's where the task server comes into play so there's a way you can configure each of your clients and make sure that they can communicate with the server upload that data and then bring it down so you're always able to know you know what is the thing that I'm running a little behind on and I need to keep tabs on so in order to run this application the thing that I decided to use for this was Kubernetes distribution I finally decided to kick the tires on that I decided to use a distribution called K3s which is from the Rancher folks who were inquired by SUSE super super impressed with this very easy for me this wasn't necessarily like my opportunity to learn how to like do Kubernetes the hard way I just wanted to get this up and running and I was really focused on like more of the lower level again I wanted to think about you know I just want to run CUBE and you know for me the exciting part was like thinking about the lower levels how am I going to pixie boot servers stuff like that which is a great transition because the technology we're going to use for that is called MAS or metal as a service and that's by canonical in the Ubuntu community and we're going to spend a lot of time digging in there and that's really going to be the meat of this talk we're also going to talk about what types of physical machines you want to use what are some of the servers how much you think about tying up the network and how do you want to architect things in order to make sure that they're going to be able to communicate with each other and you can actually connect all these services properly and to start we're going to kind of go into the home lab and so like I was saying the you know when I first started I was really going in you know I was really excited about technology right there was just so much potential with the open source community so many people that had their own itch to scratch they were coming together collaborating and building something pretty awesome and so I think over the years you know I really was focusing on you know solving business use cases you know how do we think about the customers how do we make sure that we're really accelerating our delivery how do we sure we're doing things in a reliable and repeatable way and I think you know focusing on that obviously super critical but I think I lost that opportunity to really like jump in to the technology and kind of have fun with it as well and so about a year ago I joined a new company called Neuralink and we're building medical devices for patients with spinal cord injuries and so we're building a brain computer interface it's an implant that's connected to the motor cortex of the brain for patients with paralysis so this is somebody with like quadriplegia who is paralyzed and can actually move their limbs anymore and so by using this implant they're able to just think about moving a cursor and they're able to connect to the computer over Bluetooth and actually control a computer using their thoughts alone which is pretty wild it's like the craziest thing I've ever heard in my life and I'm still super so to work on this so we're at this really really cool place where I think we're going from like more of a research lab like just trying to prove out this technology and I think we're actually you know if you've seen the monkey mind pong YouTube video if you've seen the pig demo like we're pretty convinced this works and it's more about how do we scale out this technology how do we make sure we're bringing the infrastructure and the resources we need in order to make this happen and so from an infrastructure perspective you know being an infrastructure engineer this is really exciting because we're really the early days of this entire industry to be honest and definitely in terms of the company you know there isn't that much infrastructure there and what is there is you know very solid and I'm super stoked for Logan and all the folks that started before me but I think the ways that we need to grow the company and the challenges we're going to be facing you know really require us to make some very hard decisions and I think right now in order to make the right calls we need to have a lot of experience and we need to be able to decide you know what are the things we want to build and how to make sure we roll that out in a safe way that's going to scale with the organization so we don't need to like do a ton of migrations again and again and again and so I think you know when I think about you know when I joined the company and I started to understand you know what are the requirements we have what are the different problem domains we wanted you know we really do need to iterate we're so small we need to take advantage of the best technology that there is available and we need to contribute back to that we need to build it but I didn't want to go off and like leave like 50 half-baked systems in like a trail of destruction behind me I didn't want to like leave a bunch of things that are out there have a couple people try some stuff out you know those become critical but you know I'm really excited about something new and I'm trying to build a new system and then meanwhile I'm also trying to like keep this other system or these other 20 systems you know dragging along and so I needed the safe place to try out new technology I wanted somewhere where you know it still mattered but you know the sample size was one you know it was just myself and you know my wife and you know we were the only people who are going to be using this you know this is just providing functionality for the personal network and not something that was necessarily going to be involved in the corporate setting you know we still want to iterate very quickly when it comes to work but I wanted to have a you know sort of a prototype for the prototype before I really jumped into that thing and so in order to do that you know again I sort of thought about you know what is it that we're doing at the company and how do I start to apply that and sort of create a parallel universe at home and one of the first things was that thinking about you know how even though at work we're still going to be able to leverage the cloud we want to take advantage of as many managed services as possible we want to be able to you know hand over a lot of that operational workload to you know a bunch of the software engineers who built these battle tested systems but at the same time you know we have these physical implants you know and we're trying to we need to measure them we need to understand how they're behaving we need to track their temperature and a ton of other metadata about them so that means you know we're always going to have you know Bluetooth antennas that need to be plugged into some amount of servers we have you know an implant assembly line that you know for each of the different stations along the line we have a bunch of sensors that need to be used to validate that we're building the components to within the specifications we have and there's a ton of software that needs to run to verify that those components are behaving as expected and that needs to run on something so each station needs to have physical hardware as well so you know again as much as I'm you know able to leverage the cloud experience that we have we need to become excellent at running physical hardware and so that meant that I need to remember how the heck to do that and so to start that means building our own infrastructure as a service so what is this base layer how do we think about the infrastructure the hardware and the servers that we're going to use how do we make sure we're able to bring them up in a repeatable and reliable way and then how are we going to connect all of these servers together so they're going to be able to talk to each other and I'm actually going to be able to access the services that we're building and again I kind of touched on this but I think we're really trying to make sure that you know we have this easy opportunity to try new things you know if I'm really kind of like oh you know this was a great idea it seemed like a good idea at the time it definitely is not you know let's scrap this throw it away it's something where I only need to change some configs on my own you know I just need to reprovision something it's not a big deal there we go so when you're looking at the the opportunities for how can you provision these servers how can you create these operating systems and make them run obviously you're thinking about servers and we're going to talk on you know some of the different things you can use for that you also want to think about how can you isolate out the network and so we're going to touch on some of the switches you might want to use if you don't want to use that we'll talk about crossover cables which I'd forgotten existed so I wanted to call them out specifically and then you can also thank you yes exactly I've been in the cloud for a long time so and then also if you don't want to deal with this you know you can just get started with some VMs which is great so as I discovered yesterday when I spilled coffee all over my laptop it is not a good idea to use your daily driver for a bunch of this experimentation so I would highly recommend not having to blow apart you know whatever you're using for your day to day since you again want to have that opportunity to experiment you want to be able to try different things you want to distro hop here and there you want to try you know fedora silver blue you want to try these different os's and different approaches and see if that's something that's actually beneficial and maybe you can apply that in different areas so one of the cool things is the next unit of computing boxes that Intel launched our PAL system 76 who have a booth in the expo floor they have this cool little box they call the meerkat which is awesome so I have two of them running on my desk they're great because they're just super easy to to run a bunch of different things they don't take up that much footprint they're pretty low power so that's been something for me that's been great is just having these this hardware has really empowered and enabled a ton of different systems and a lot of experimentation which is awesome you know if you're looking to upgrade your laptop you can always keep your old one around that's great because you don't need to plug in a monitor and keyboard it's super easy you know again this is something I wish I just like thought of years ago I feel like I really would have benefited a lot from having this environment set up and so I'm just encouraging y'all to think about like you know can you set this up if you don't have this already there once you have the physical hardware's physical hardware set up you need a way for it to connect to things and so most of us probably have what's called an unmanaged switch and so this is just like a little tiny box you can just plug a cable in you can connect to it and make things you know it'll talk to whatever network it's plugged into and those machines will be able to communicate but if you're using something called a managed switch this is something that's a little bit more advanced maybe a little more expensive has more advanced software on there and it's going to the firmware will let you customize each port and you can set different things like vlands you can set which network these things are on and you can actually isolate out different services from each other so if you have like your normal desktop your normal laptop or your wi-fi router those can be connected to each other and they can all talk and do their thing and then you can create a separate isolated network which you're just using for some of this experimentation and so if you just want to play out for a player play around with something like you just want to mess around with mass especially as we get into talking about DHCP if you want to create that isolation if you're using a managed switch you'll be able to specifically siphon that traffic off and make sure it can only talk to ports which you've specifically configured and enabled and these are all different depending on the settings you're using but for me this was a great opportunity to get more involved with Fortinet we're using some more of their switches at the office so that meant I had the chance to like sign up for their support center you know figure out how to upgrade the firmware do all this and again this is something where it's like very low effort you know I'm just doing it on a weekend for fun and it's not you know when it comes time to like upgrading the firmware at you know for the core router at the office I'm like oh yeah I know how this goes you know I know where to download things I know how to follow the steps so I'm feeling a lot more confident which is pretty great if you don't want to deal with the the switch again the crossover cable something I again very embarrassed I forgot about but what's really cool about this is this is an ethernet cable where they've actually twisted the wires and so this cable is taking the input from one laptop and it's going to the output of the other machine and it's the same on the other way back normally like a straight through ethernet cable you're expected to plug it into a switch or a router and the way that that those network cars are set up is it's expected to communicate directly in that like a normal pattern where the network cards are set up with that and machine like laptops and servers you need something like that that switches it so that way you can communicate that way directly so this is great because you can just manually configure the network in your operating system and then you don't even need a switch you can just again super easy to set this up you just need a little cable it's super red so yeah and these are pretty cheap online which is great or you can make your own if you just have it and you can just switch that up and again if you don't want to deal with any of this stuff if you just want to get started any machine that you've got that's running on modern OS you can start virtualizing this you know there's a ton and ton of ton of documentation a ton of tech talks on things like this you know Ubuntu and canonical have multi pass which is awesome there's a great tutorial on Maz so I highly encourage you to check that out if you're not not familiar with that okay so we've got our setup we generally have some network things are plugged in which is great now we need to start talking about like how are you going to you know we need to power on these servers we need to bring this up something I didn't realize but like most of our machines do not have bios anymore this is blew my mind I you know this is very old information to like everybody in this room I'm sure but but everything's been replaced by a new standard called UEFI the unified extensible firmware interface and so for me you know I would say like oh I'm just going to like you know boot the bios and I'm going to connect in and I'm going to set it to network boot or something totally not a thing like none of the machines that I have actually have a bios anymore this is all like very ancient technology and there is no standard like this this is just like a de facto standard it just works the way it does because that's how like the IBM PC was created and this is just like people just sort of like YOLO tried to figure out how to like set this up the same way and one of the things that it uses is something called a master boot record which we're going to get into so your machine the actual laptop or the you know server that you have has some firmware on there and it's very minimal it basically like powers on the box and it's going to hand over control this is the the process for booting bios essentially and so this is still helpful to know like you know kind of the basics like where we're coming from so this is going to hand over control to the bios and what it's going to look for is it's going to look for like a special like set of bytes on the very beginning of your hard drive and it's just like kind of like this magic area that nobody really knows like technically how long it's supposed to be like how much space you have like how things are formatted it's all just kind of like reverse engineered essentially and everybody's sort of like winks and goes along with it and so this here the master boot record it's a way to structure how the rest of your file system is partitioned so where can you look for essentially the bootloader so in Linux we're using a system called Grub and Grub is a little bit more advanced you're able to build up a little bit more functionality here and really the once you're at this stage you're able to think about which Linux kernel version do you want to boot which operating system do you want to boot but again this is like very limited functionality still at this point and really it's mission in life is just get the kernel up and running which again as most of us know is kind of that intersection between the hardware and actually the rest of the operating system that you're booting so BIOS obviously works super well I mean there's a reason it was around for decades it's you know it made things work out pretty well but again one of the several issues including it can only access one megabyte of memory and so if you think about like you know to make fun of myself you know if you think about Slack like there's no way Slack is only taking one megabyte of memory so you know that is really not that much space and so there's a ton of other downsides to this in addition one of the big problems is BIOS is really only good at booting and operating system from disks and so if you want to do something a little bit more intelligent like for us we wanted to boot off the network that's just something we're not going to be able to do unfortunately and so huge props to Intel they obviously recognize this problem and they created the extensible firmware interface and then they realize that you know hey it's great that we've got like this thing that we're building on our own but the PC industry is this huge you know there's a ton of different companies there's firmware vendors there's motherboard manufacturers there's PC vendors there's operating system vendors all of these different components all need to interact with each other they all need to know how this system works and they need to be able to communicate with each other so we need some standard we need to have some way to agree on how the heck this machine is supposed to work like how are you supposed to get this hardware all the way up to something that you know someone like you or I could take advantage of and so that's where the UEFI standard was created and there's like an umbrella organization called the UEFI forum and this group they publish all of the standards online I think it's relatively dry reading personally but like it's all there if you want to read it tons of details tons of documentation on how to implement this how all of this stuff works we're going to dig into a little bit of this and talk about how this this happens so one of the big things obviously now we can take advantage of modern processors so now this is again why I'm saying that you know this is what most of our systems are using is UEFI standards you know you're really having a lot more capabilities here because you have because you have this opportunity to start to define things you have a process to make modifications and one of the big things that it's doing is actually defining the standard for how are you formatting the file systems on disk how are you creating that bootable partition and you're telling the firmware how are you actually going to start to hand over control to the operating system and so this standard means that you can start to build tools that take advantage of that this GPT format is a structure that everybody can start to build against and make sure that they know how to use it there's not too much in there so we won't go into too much detail about it but really the important part here is now at least we have one way for any system at whatever layer of the stack you're in to be able to to query that information and figure out what operating systems are available what are the different partitions and how can we take advantage of that so for example if you're just running you know Linux right now you can run EFI boot manager and you can actually print out and you can query that information like you can look at that partition and you can see the details about it and so here you can see for instance that you know using your operating system you can actually set the boot order for the devices that you have and so here you see the first thing is 2001 and then 000 so here first you know this is this laptop here it's you know if I plug in a USB drive that's the first thing that's going to boot and then you know there's no USB drive then it's going to boot this EFI hard drive which is great and so this is something before you'd only be able to change in the firmware and again each implementation might be different there might be different ways of doing this but again this is something that now way up in user space up in you know once you've booted the os this is something you can configure the firmware on for your device which is pretty sweet all right so we've got our firmware we have our uefi set up we sort of know that you know we have a way to configure this stuff and so using uefi that is the way that we can actually set up our network booting but in order to get there we kind of need to like make sure we run through some of these different systems that you know if you've been in the cloud for a long time you may not remember how they work you may not have experience with that so the first one we're just going to touch on very briefly is just the concept of a subnet and so if we think about a network it's a connection of it's a collection of machines or a collection of servers the intranet obviously is a you know intranetwork group of networks so you have multiple networks that all connect and communicate with each other and then a subnet is a subdivision so it's kind of for you know our use case we think of it this is our home lab so this would be the you know all the machines connected to our wi-fi that's on our home network here and so you know most folks standard standard network it's going to be 192, 168, 1.1 or 1.0 and that's going to be something called a slash 24 long story we're not going to get into cyber math or anything in this there's many folks who will give much better talks about that than i will but long story short is you know typically you're going to have about space for about 256 hosts in your typical network which is green and so you know when we're thinking about like hey maybe we want to create a separate isolated network for joes like wild experiments that's going to be something we're going to create a new subnet for and we're going to to configure that network switch to say hey anything that gets plugged in here we want it to be on this 192, 168, 200 network and so we maybe we don't set up routing rules so there's no way for machines on this special experimental network to connect back and corrupt whatever is going on so if i'm setting up something you know again we're going to talk about DHCP in a moment but maybe i'm messing around with some of the network settings you know that could totally destroy that network and if i were to do it on the wi-fi and i had friends come over they'd be like hey like why is your wi-fi broken i can't connect so again creating that isolation again gives you that opportunity experiment which is really important the next thing we're going to talk about is the distinction between unicast addressing and broadcast addressing and this is super important because we need to think about the distinction between you know once i have an ip address once i'm connected to the network if i want to you know connect to google you know i just looked up dns here this is you know google has its own very complicated thing which again is probably like months of talks to figure out how their network works but but at this point you know if i wanted to connect to google this would be the ip address i'm doing and so when i say hey i want to send a search request my computer knows that it needs to talk to this ip address and it's going to send a message which is only intended for google and so that's the only server that really cares that i'm talking to it and it knows that i'm sending this message over to that server and broadcast is very helpful when you need to send a message to everybody on your network and this is typically restricted to just the subnet that you're on but this is a way for you to sort of blast out a message just sort of like i am right now i'm sending out this whole broadcast message to all of you and you're hearing it whether or not you want to and you're you're basically know that you're potentially intended recipient for this for this information and you know knowing broadcast addressing is important because there's something called dynamic host configuration protocol and so whenever you turn on your phone or you open up your laptop and you connect to the wi-fi or you you know join your friends wi-fi the first thing your computer's going to do is it's going to say hey I don't have an IP address which means I cannot communicate with other servers I can't talk to other machines and so I need an IP address so that way I can send messages and people know how to reach me and get back to me and send me that information and so you know thank goodness all the early engineers I'm sure some of you folks who are involved in this thank you for all the hard work this is really important because this is how we establish sort of the rules of the road and so I think in the knock like two or three rooms over over that way great excellent it's over there so every time you come up on the network your client is sending request and we'll dig into what that looks like here so it follows a process called Dora so that's the first step is going to be the discovery and so that's when your laptop opens up you're going to broadcast a message and you're going to say hey you know I would I'm a new machine I have a specific like hardware address and I need an IP address if the network's been configured appropriately then so there's going to be a server there and it's going to say hey you know I'm going to set aside this IP address for you would you still like this and 99% of the time your laptop's going to say like heck yes I definitely want to I know on a stream YouTube or whatever so I will definitely take that IP address if it's still available and then hopefully the server is going to agree and say yep that sounds great I you know that's yours I'm going to mark that down for you know certain amount of amount of time and that's going to be your IP address until further notice so that's great so at least we boot it up we've connected to the network and we're going to be able to you know get an IP address and send traffic the next thing we need to think about again when we're thinking about you know how do we want to get get an operating system to actually boot is you know we need to get that data onto your hard drive so again if I'm you know taking a server from from scratch and I need to get an operating system probably I think there's a thing called Linux if I want to get that installed I need to get that data transferred over here but this process is happening you know there's no operating system right and so we need a protocol that's very very simple or trivial and so this is going to be something that hopefully is very easy to implement that's very standardized and is very clear how to do this so essentially once you know where TFTP server is you're basically able to make a request to that on port 69 and you say hey I would like to receive this file and the server is going to respond back if it's just a read request which for us is all it's going to be it's just going to respond back with a data packet and each time it sends a data packet over the UDP protocol you have to respond back and acknowledge that and say yep cool I got it give me more data I'm just going to respond back with more data you're going to write that down stay give me more and you keep going until you receive the data the way you know it's done is when you do not when the data message you receive is not the full size of the data packet when it's truncated that means you know you've reached the end which is great and if the last one is like perfectly sized so it's the full length the server will actually send you back a packet that's data packet of side zero so once you get that you know you're done which is great awesome so we've built up a way to get an IP address we built up a way to get an OS image which is important because now we're able to talk about something called pixie and so this pre-boot execution environment is a standard actually it's been around for a long time but it's finally added to UEFI and this is the way that most machines are actually going to request that network image and so you know at work or in a lot of server environments like this is the way this is the process that servers go through in order to get that network image which is great so we're going to build on top of these two protocols that we've already defined earlier the big one that we're going to do is during that Dora process during that initial discover request we're actually going to add some special configuration and we're going to say hey I actually don't just want an IP address I'm also ready to rock and roll and I want a new version of Linux so I want to get you know whatever operating system you've got for me I want to take that and I want to boot that which is pretty great so if you have a special DHCP server which is configured appropriately which you know Soiler Alert Mas is configured appropriately it's going to tell you hey you know here's your network information here's the subnet you're on here's your DNS servers but also here's some information about where you want to go to get a TFTP server to get that data you're looking for that's where you're going to go to get that network that operating system image one caveat about TFTP is there is absolutely zero security built into the image so if you want to do some like extra validation or extra check something that's 100% on you that is not part of the protocol so there's you know no security on any of this stuff so you know if you've heard of the term of like a rogue DHCP server handing out leases like you know make sure that if you were sending out a pixie request you know hopefully you were in control of that network and you know that your TFTP server is you know secured and somebody hasn't put like sketchy images on that box which is you know the whole lab is very unlikely which is great all right so we know how to boot things we know how to set you know we know the fundamentals here we know sort of the first principles about how to actually use these protocols and how to leverage this and so now we can talk about a specific implementation about how to take advantage of these things so metals of service again comes out of canonical and the boon-too community and for us you know this is how we're going to provision our servers very quickly and so we're thinking about again reducing the cost of errors you know making it really easy to you know kind of have that similar cloud like experience like hey you know I just trashed this operating system I was messing around with system D you know I don't know how I like blew up my network manager config this is really bad I'm just going to re-image this box like I've had it like I don't have time for this like this was fun now it's not let's start over so this is pretty great because it provides a lot of that abstractions that make it much easier and so you know we're going to dig into a lot of the like implementation details there's again the maths team actually is like totally revamped their docs in the last like three weeks tons and tons of details in there which is great so I highly recommend going through that or checking out some of the tutorials we're going to spend most of the time on the implementation but going through the web UI it's pretty self-explanatory if you know these fundamentals and you can apply this to the process so there's three different components we need to think about here so the first one is going to be called the region D the region controller or region D and this one you're typically going to use one per office or like one per location for us obviously in our lab we're still we're going to have one of everything obviously but this is something which I think of is like the brains of the operation this is where you're really going to be thinking about you know doing a lot of that processing this is where you're going to locate that web UI that API server this is the thing that you know as an engineer like this is what I'm going to be interacting with as a human day to day it's going to persist all that data in a post-gresql database and so that obviously you know a few you know for us you know it's a home lab it's not really that much work so you know I'm not going to worry about backups or anything like that but if you're deploying this a production environment that you'd want to care about Region D itself has nothing to do with DHCP it doesn't respond to the network requests the rack controller is what's responsible for talking to machines so the rack D server is what machines communicate with so Region D I think about is for humans and then rack D for machines so this is what's going to serve DHCP and it's also going to handle the pixie process which means it's also going to serve the tftp server as well now the important part though is that rack D is what I'm very thankful it's stateless which means that most of the time it's actually proxying the request back to the Region D server when so when our request comes in which we'll see in a moment it's actually talking to Region D which is what what knows what to respond with so rack D doesn't need too much configuration we're doing all of that in Region D which is great so here's the the typical mass flow let me set this a little bit better I think so when we do want to add a machine one of the first things we need to do is actually enlist this box and so enlistment is when you're first connecting this into mass and you're sort of first telling the mass server what the heck is going on here and mass actually has some pretty comprehensive scripts in there whose sole purpose in life is to figure out like what in the world is on this hardware like how many hard drives are there what type of network card is there how much memory is in this box and it's trying to just sort of like query anything and everything I can think about figure out all the little details like what hardware vendor if you are like dealing with enterprise hardware there's something called IPMI like lights out management which is super cool and we're not going to get into but all of that stuff is what mass is trying to figure out so it knows how to communicate and deal with this hardware so the first thing you're doing is you're actually going to enlist this box and you're going to bring it into this state where it's going to run a lot of these built in commissioning scripts and so these are sort of the default and kind of give mass this like first pass pass of just like what the heck is this hardware and how do I deal with it once you've done that it actually creates an entry in the mass database the region D server persists that and then it's going to create a cool page in the web UI and it's going to show you a lot of those details so it's going to tell you like oh you have an Intel network card you have like a Samsung hard drive you have 64 gigs of RAM whatever you have once it's in new state that's where you need to commission the machine and this is a little bit more advanced because you can actually run some custom commissioning scripts and so if you have extra hardware that you've added that maybe aren't being queried for in the default set of commissioning scripts this is what's actually going to query that and make that available in mass if you need to it also can do things like actually update NTP so update the system time and gather some extra information which is good and then at this point it's going to mark the machine is ready and all of this so far has basically been to get us to this point this box is finally like able to it's finally allocated and it's finally ready to take like be pixie booted and actually like install an operating system here so we need to all of this like you know 1975 knowledge or whatever just like install the next for the most part so the last step there is when you actually deploy the box and this is where you decide like hey I want to like install boon 20.04 or 20.20.04 I want to install this cloud in it configuration on there which we'll get into but basically here you've like got a ton of the details you know what sort of hardware there is you know what architecture it is you know what images appropriate and you're going to be able to deploy this onto the box if you needed to add these custom templates they're on the region controller and they're just located in this directory and you can actually take a look at some of the scripts and you can see you know what is it that they're doing how are they querying this information and there's a lot of cool things if you want to dig into some of the hardware and see how this is presented and how it's available so there's again a ton of information that you can find here which is pretty cool and highly encourage you to go into the maz docs because they're actually very comprehensive here if you have more questions so now we're going to really dig into commissioning and really understand you know what is this process look like and how does this work so when you first plug in your laptop or the server that you're going to be dealing with the first thing that it's going to do again is you know make that DHCP request and if you've set up in UAFI if you've set that to network boot it's going to add in those special configuration and it's going to say hey you know not only do I want that IP address but I also want to connect to a pixie server and I want to actually get that TFTP image so I can run those commissioning scripts so rackd is going to you know which is configured as your DHCP server it's going to respond to that and before it sends that response back it's actually going to query the region controller and it's going to ask it hey I need some of this boot metadata like I need to get some of the details about where's the location of the right kernel where's the TFTP let me make sure that I'm going to be the TFTP server you want me to use and it's going to query region D to get that information and all that stuff is you know the defaults for us are just going to work just fine so once rackd gets that information it's going to pass that back to the server that you're provisioning and it's going to send that pixie configuration back to the box and the box is going to do that thing where it uses TFTP to connect to that server it's going to write out that ephemeral image it's going to boot from that temporary image and then it's going to run all those commissioning scripts and get that information there it's also going to fetch something called cloud init from the region D server once the box spins up which we're going to get into a lot more detail and then it's also going to query the rack D server and tell hey like here's all this metadata that I found I want you to to talk to the region D server and I want you to persist this way down in Postgres so this is how the commissioning process works and how we get all that information we need in mass so then as we go through and say like oh like you know which server do I want to deal with you have all of these details here and you're able to figure out you know oh is this the right box that I want to re deploy or re image you know you have all that information there available once it's done with that process it's like publish that metadata the box is actually going to shut itself down so there's this is again just an ephemeral image it's just like a temporary thing that just gets you this information you're not really installing the operating system when you're commissioning that's the job of the deployment phase and so once you deploy you're using the same you know again assuming you're using Ubuntu you're using the same installer you would on a live CD and that's a tool called curtain and curtain is short for the curtain installer it's like whole mission in life is just install stuff really quickly and on first boot it's going to execute a program called cloud in it which we're going to talk about in just a bit but to really dig into curtain the whole mission here is it's supposed to be able to handle a few different configurations and make sure no matter what you're going to what setup you have you're going to be able to write a new operating system image onto a hard drive here and so it's basically it's just like a really like robust like DD command essentially so the first thing that's going to do is again you've kicked off that boot and so that's whether you've like plugged in a USB thumb drive you've got a CD ROM or you've done the network boot and you're basically bringing up this like live image which is going to run the curtain installer and so the first the next stage is going to do is run what are called early commands and this is basically loading in some modules for some configuration you're going to set up some of the environment to make sure that the hardware is primed and then if you want to you can actually update your apt cache so that way make sure you have all the latest booted packages when you're installing so you're not installing some like super out of date version once you've done that the next step is actually partition the hard drive so again we actually want to persist this we want to write an OS image and so we really need you know we want to create the partitioning scheme correctly hopefully we're going to use like you know we're going to make the EFI boot partition hopefully we're going to use the GPT format again because we're using UEFI because BIOS is deprecated ha ha and we're also going to create the template for the file system tab or the FS tab and so this is where we're not actually writing F-stab we're actually just going to write out a template which will be written out a little bit later the next step is then going to be doing a little bit of network setup so this is actually going to again write out a template to configure your network and the reason we're writing the template is because when we write the disk image we want to add that configuration after that's written out so here typically you're going to get a tar ball with the Ubuntu OS this is what's going to get extracted onto the disk that you just partitioned and you're going to write all of that information out onto the hard drive the last thing I actually for this talk I discovered I didn't even realize this was a thing but you can actually add a little like webhook at the end of the installation process so if you have like a Slack team or like Discord or whatever you can actually like configure it to like post in a channel and say like hey like you know I've been finished installing so this box is ready to go so I thought this was pretty cool it's an extra little hook that you can configure and add that in there which is awesome sorry that was too early so this is extra configuration which you can set up maybe install the webhook script and then here's where you can actually execute that script when you put that in there all right so again we're like building up these layers right so we have an operating system installed pretty much we're mostly like a stock Ubuntu image at this point and so we really need to like you know we want to install like the stuff that we want right like we don't want to just like have you know it's great that we can install an ISO but like you know we could just plug in a thumb drive and it's not really saving us that much time what I think really gets interesting is using a tool called cloud in it and if you're running on any of the cloud providers they all support this tool Amazon you might be familiar with user data it's it's really use it like it's using cloud in it to run this stuff and it's the same format which is great so this program is what runs on the first boot so after you've created the image it's the first time that you're actually going to bring up the system it knows that it needs to execute this code and execute the configuration that you've given it and so an example would be this is you know very basic but you're basically what this is the cloud native part of the talk because it's YAML but it's basically the first thing you're going to do is the package update running app to update getting the latest packages in the repository and then you can upgrade them and then you're also going to create a user ID there's a ton of like additional features that you can use this is awesome because you can actually connect to github and then pull down your ssh keys it also has launched pad support for anybody who still uses that so this is great because like you know again even if we just did this this means that we can just provision a new image we can provision a new server and you know we always know that we're we're able to start from scratch and that's awesome but unfortunately you know sometimes you're just like I want to try a couple different instances or I want to run this application you know maybe I don't want to deal with you know all the common problems we think about when it comes to isolating different application versions or different libraries so we need probably another platform and as one of the quotes I use all the time is obviously the solution is that another layer of indirection or abstraction which means I finally need to learn kube which I try to avoid through most of my time at Slack much of the chagrand and my colleagues so here's where we really get to move very quickly and you know there's a ton of documentation for running k3s so we don't really need too much you know like if you wanted to just get a Kubernetes instance running on your server you can just add this to your cloud in it and it's going to be there it's wild so huge props to the Rancher folks huge props to Patrick at the Sousa booth yesterday for giving me the tools to try and save my laptop yesterday so props to all these folks but yeah k3s is super solid again it wasn't something that I wanted to like go too deep into so you know there's a ton of opportunity here to like really play around with you know what are the best kube distros you want to use how you want to run this and how are you going to configure this appropriately but if you just want something super easy this is going to be great so we pretty much like made it most of the way through we've got you know a platform and we've got kube right and so now we have a way to get servers on our desk running kube if you wanted to create a second kube second cloud in it file if you wanted to have multiple servers that are all connected to the same kube cluster you can actually set some environment variables here and you can pass it the name of your initial kubernetes server that you provisioned so the first one you spin up probably looks like this the second server you spin up you're going to pass in a few details like the the ip address for the kubernetes server as well as like a little pass code basically like a little certificate that you're going to pass over so it can authenticate and join the cluster so again doing this and having this means you can start to like provision you know a ton of like raspberry pies or a bunch of other servers you have lying around and you can really add them together pretty easily and if again any of them have problems you want to add a new one or replace one again using cloud in it you can do this super super quickly which is really cool so again I was using macOS for a long time which you know was fine but there's a great to do app called omni focus but again I was thinking about you know for me the killer app was you know running this command line to do app which is very appropriate for scale I think but there's also a bunch of other applications like home assistant which is a really awesome Python project has a ton of community support behind it you can think of it kind of like an open source home kit or like an open source Alexa and it's really great because it actually works across a lot of these ecosystems and so for folks who use different components from like you know the Google Google home ecosystem or Alexa you can use something like home assistance to bridge that gap and I believe the plugin is called home bridge actually to tie home assistant together with home kit which is awesome and again you're running all of this locally so you don't need to worry about you know extra network latency you don't need to worry about you know who might giving access to all this data this is all code that's actually you're running on your desk at this point which is super cool you can also run like the Pleximedia server if you have again a bunch of home videos or something that you want to show off and then even there's like cool like self-hosted apps like I think it's tenduri recipes like if you want to have like a recipes server that you're running but again this is really giving you a lot of control you know you can run different apps you control the data you don't have to worry about where's it being stored what happens if like you know Google Reader gets shut down things like that so you know I think the usual benefits we all know about open source start to to come into play here which is pretty great so I think again we as we're thinking about apps though we get kind of the benefit of kind of diverging from sort of what I would consider like SRE best practices you know I am very hesitant to kind of diverge from sort of what the like the best practice of the upstream project is and so the task server is expected to be installed like directly on the OS and it's not necessarily like there isn't like a ton of folks that are necessarily running this in a containerized environment but since it's just me you know if this thing goes down or if I need to troubleshoot something like I only have myself to blame and it's something where I can fix on my own schedule and that's fine so here it's totally something that's that's great but fortunately there's Yehengi I don't even know how to pronounce this person's name but this person on the internet decided to dockerize the task server and actually create some kube config which was awesome I made a couple tweaks we'll dig into but huge props to this person for like blazing the trail and getting much of the setup so if you want to look at the docker file it's up here on github I kind of broke down the docker file here but again this is all the stuff that we're used to doing right but it's just that we're able to run this on our desk which is pretty great so we're just going to check out the repository of the code we're using CMake to configure the build and then we're just going to build the actual application we're going to expose the ports so we're going to tell docker that hey I'm going to be running something that's going to be listening on the network on these ports so make sure this is available so I can actually connect this application and then we're setting the entry point which is what the actual application that's going to run there that we've built one interesting thing about the task server in particular and the reason that I'm actually like comfortable running this is that it does actually use TLS for off and so you do need to generate certificates there's some decent documentation in the upstream project I wrote a little little like helper which is basically the more rise version basically but basically here if you generate some client certificates and the server side certificate so long as you're transferring the client certs to your devices securely you'll be the only person that's able to access this data which obviously is important so so you'll need these and we're going to use this as we deploy this to the kube cluster so again so I have a little fork we're going to just dig into the actual script that's running but basically here this is just you know again if we're used to kube this is how you run all these instances and how you deploy this so first we're creating the namespace to make sure that we're isolating the pods and making sure that all these resources are within their own area within the kubernetes cluster we're creating a persistent volume claim which again is you know k3's by default gives us as like local path provisioner which is just basically just like writing files directly to disc which for us is perfect right like we don't need some like web scale solution that like persists to like 17 regions around the globe like we just you know I have a cute little mirror cat on my desk right and that's what's going to take the data so that's fine so we're setting that up we're setting configuration and then just for my own sanity because I kept forgetting as I was like iterating on this a few times I just added this little helper to remind me to set up the dang tls certs and then we're actually going to upload all of those certificates once we've generated them which is great so we've done it you know all of this work you know going back to like 1975 just that way Joe can actually run his like to do to do app in his terminal but you know we've created this thing it was you know may not be what you want to use but it's what I wanted to use so that's the beauty of the home lab so that way we can actually take advantage of you know using this and creating this fun environment you know I finally got to use Kubernetes I have to be cranky at it for you know five or six years you know we got to have a lot of experience with mass we're able to provision these servers and make sure we understand you know from first principles like how are we going to build this out what are the protocols that are necessary and again now that I'm going back to work you know come Monday I'll be able to do a great job of building this infrastructure and I have a lot more context and I have a lot more opinions about what I like and what I don't like because I've gone through this process you know we I finally had to deal with physical machines thank god it did because yesterday I had to do surgery on my laptop and again we've we've kind of created our home lab and now we're hopefully you're excited to build your own as well so thanks so much for the talk we're hiring at Neuralink comes check us out in the expo floor and thanks again everybody for the time does anybody have any questions we got like ten minutes for questions going once oh sweet I may have missed it but where did you install mas on that is a great question I actually just you snap to install mas on the box itself just so you do need to have like that is a I did not add that that is so wild so you do need to have a separate server that I provisioned so that's why I was saying like you have you know multiple machines for that so for that I just dedicated something to run mas and so that's the thing that's serving DHCP and so that's where I didn't want to get too wild there is I think the airship I think does have a mas running in Kubernetes but I think that's just running on 2.8 and now I think we're on 3.2 just got released so exactly correct Linux box running mas is the TLDR good question about the hardware setup I'm trying to set up my home map as well awesome but I'm thinking about like server ITX motherboard and use ITMI for accessing to it what's your recommendation for hardware setup hardware setup for using for using IPMI or just regular setup like the whole home lab like motherboard CPU like oh interesting I mean I think like I think if you wanted to build your own like hardware or like try out like if you have some vendor that you use at work that you think is cool or something definitely finding that overlap is great I think for me like I think just finding these like mircats was great right because I didn't necessarily want to go through like you know which motherboard is compatible I didn't have I don't have any IPMI at home and so that is downside but I think that's definitely you know for me at least I think that's definitely a next step is like automating that and then getting mass to like automatically power on and off boxes is super awesome so yeah so I'd love to see your blog post your talk next year what you decide all right great talk next year is there's a great talk there any considerations for security in your lab or is it is it walled off so that no one can really get to it whoof I am so thankful for all my colleagues who actually know how security works I'm very very stoked I think I like like for me I was like oh yeah like there's certs like cert off this is great that's all I need I think I'm definitely I'm actually starting to look to my colleagues at Slack built something called Nebula which is like alternative to like wire guard or tail scale so I think something like that so you don't necessarily need to like expose ports on the open internet is really really appealing right now naively I am I have opened the like task server port on the router so I do have that like exposed so yeah I think like you know that is something that I'm definitely thinking about so I think something like tail scale if you haven't looked into it it seems super cool so but that basically creates like a like a very like interesting virtual private network so you don't actually need to like make this connection and you don't need a ton of extra servers you basically have this like network where only your servers can communicate with each other no matter where they are in the world it's like it feels like magic it's super cool so so Hey Joe so did you have any problems with UEFI on the mirror cats or were that was that pretty straightforward from like configuring it to do the the netboot fortunately I did not have many problems I think I just had like I think was more like a personal problem where I was just like oh my god it's not bios anymore like when did this happen like I don't so I think that was just something that like I needed to like correct my misunderstanding so yeah I think that just like appreciating that I think helped a lot because then I think like it helped me not be afraid of like going to the firmware anymore if that makes sense like I think just like feeling more comfortable like knowing what's going on like knowing how to partition things like knowing which disks I'm dealing with I think having that for those fundamentals I think was actually super valuable but yeah I think I think overall the UFI was was pretty solid so and then the lemur pro is rocking core boot which is awesome so yeah open source firmware very sweet any other questions you're making me run nope nope my ip is 127 001 I think same as yours I think all right what do you do for electrical power protection for your equipment great question I grabbed some I don't even know if it's actually good or not but I did have the same question and I found something on Amazon for like 70 bucks or whatever and it's not a battery backup I don't really care about a power supply or sorry I don't care about a battery backup just yet but it is something that it's like it seems like just a fancy surge protector so I am like I didn't necessarily want to like plug stuff straight into the wall just in case like there's some like dirty power coming in or something so it should I think I forget the details of that but it should like even out the curve power curve or something is that a thing that thing yeah that sounds that sounds reasonable cool cool cool thanks everybody yeah I appreciate it yeah I would definitely look into that if it sounds like you're thinking about that already so okay so more about your presentation how did you make the slide so I really liked your theme there yeah that's the project is called look at me I cannot take any credit for that I should have added it back here but right there the first bullet point right there is the GitHub repo so it's a Python project so if you just pip install that that'll that'll work yeah I was like blown away it's it's pretty awesome so yeah you just write markdown and like each markdown document is a new slide so it's super cool so for a home network Kubernetes cluster how small and portable can you go there are there I missed the beginning on the hardware you're running but can you just can it be as small as Raspberry Pis or smaller yeah and also can the can the cluster be based on a wireless network sure yeah out of the box yeah so I think you can definitely set up like your servers on a wireless network I think this is where like me being a little annoying I think I just wanted to have that you know there's nothing better than just plugging in you can totally do wireless and it's probably going to be fine and then I mean like if the if you could UEFI boot over wireless if it wouldn't that be cool do we have any experts that actually know the answer to this question Logan yeah I'm hearing not yet probably coming soon I don't much I mean like the time scales we're talking about like soon as relative before scale 30 before scale 30 that might be a thing yeah I don't think we'll get like WPA 7 support though but you know maybe we'll see so but yeah there's a ton of youtube videos like if you just search for like home lab there's a ton of like awesome folks who are like super stoked about this this one vlogger Techno Tim just started this thing called 100 Days of Home Lab which is awesome wow this is wild so I think this is just a few months ago actually so there's a ton of folks who are like talking about their home labs like making a ton of videos kind of like really building and encouraging a lot of this stuff so there's like infinite ways to set this up so I also want to give a huge plug to the self-hosted podcast just great folks if you're not subscribed to like the destination Linux podcast like the tux digital network like a lot of those folks are doing awesome work and like really like encouraging a lot of people to do cool stuff with technology so there's just like a ton of like momentum that's being built to this community which is super cool yeah and I think somebody made like a 40 raspberry pie like cube cluster or something that they were talking about on youtube which is cool yeah that's awesome all right give it up again for Joe rock and roll thank you so much thanks everybody all right next up on the cloud native track and I believe a half hour we have Nathaniel McCallum for Weirkan the container bootloader saga oh there is a break test test test test test check two one two check check check check check check all right test three one two three check test test test check check check check all right um gonna check check check okay here I am I'm gonna read the the schedule oh no oh okay check check check check oh okay so we've got some I think uh tweets here check check okay just gonna read some of these uh tweets my favorite part of my talk at oh check check check check check one two three perhaps the most oh sometimes the questions of people who don't contribute as much is can we get them to do more instead we should dot dot dot rather than asking people to do oh congratulations team GitLab won the Soquel Linux Expo most most interactive booth hoard great copy I'm still talking to the mic check check check last day at scale 19 sometimes the question of people who don't contribute as much is can we get them to do more instead we should okay my favorite part of my talk at scale 19 today in an hour in the marina room is a personal note that says q dot dot dot perhaps the most important sentence in our Spake's entire talk testing testing no not surprised at all this is God you need to make some changes to your life okay I've got uh 130 so we're going to go ahead and start we've saved the best session for last here at the cloud native track of scale and we've got Nathaniel McCallum here to talk about we're canned which is a cool mechanism for booting containers into a computer potentially even including an os image so with that said I'll let you take it away Nathaniel great thank you very much how's everybody doing today last last day of the conference has it been good yeah what's your favorite talk we'll just take a take three people favorite talk so far don't say this one because you're cheating you like the keynote yeah home lab pod man also great talk Dan Walsh is always a delight former co-worker of mine at red hat and spent many years with him he's a great guy so I need to start off this this talk with a bit of a warning this presentation does contain gratuitous fantasy and Dungeons and Dragons references so you have been warned and with that we're just gonna we're gonna jump right in so we this is this talk is called working the bootloader container saga and we're gonna start here with with chapter one hopefully all of your character sheets have been prepared and you are ready to play a round of Dungeons and Dragons we're gonna start off by identifying our hero our hero is actually not the project in this talk it is the project called Enarchs Enarchs is an open source project which runs web assembly workloads in trusted execution environments does anybody does everybody know what trusted execution environments are good I am glad nobody knows except for a few over here so the idea of a trusted execution environment is that there's a mode of the CPU which encrypts all of the pages of the memory so for a particular application or a subset of that application and then can do hardware based attestation so we can use Enarchs to bring up an application in an environment that's actually encrypted while it's running and we can prove to others that that environment is tamper proof and encrypted and then deliver keys and such into that which is a great way to write micro services because you don't actually have to rely upon the security of the underlying operating system just the the trusted execution environment platform itself Enarchs is also a Linux foundation project within the Confidential Computing Consortium in fact we were the inaugural project for that division within the Linux foundation so let me give you a little bit of background at Enarchs history Enarchs started in 2019 and it started basically by us going to Intel and AMD to explore their new technologies Intel SGX and AMD SEV the first commit I think was in February of 2019 something like that and it was actually written in Austin, Texas while we were talking with the developers there and basically the idea was how can we make this portable you have technologies that are on different platforms they work in very, very different ways how can we create trusted execution environments so they're the same across all hardware platforms and in 2019 we decided late 2019 we decided to focus on WebAssembly and we merged WebAssembly support in 2020 late 2020 and kernel development at this point is happening in earnest so you have lots of patches for these new hardware technologies coming into the Linux kernel and then in 2021 SGX the first for version one lands in the Linux kernel version two is I'm very excited about this is about to land in the next kernel release just now and SEV the first version of that began to land in the Linux kernel in 2021 and in mid 2021 my company Profian which is a startup we launched to provide commercial support to NRX and to allow you to be able to deploy applications in the cloud and on the edge in ways that actually don't depend upon the security of the underlying operating system and of course in 2022 we're going to be releasing Profian Assure which is our enterprise support for for NRX but this is let me give you a brief demo of what and how NRX works so you can do this on your own there's no special magic here and what I'm about to show you you do need the latest generation of hardware in order to take advantage of the hardware capabilities so this is ice lake Xeon for Intel or the Milan generation for AMD and since not everybody has these servers sitting in their back pocket we decided to make a website called try.nrx.dev and you can click start now and we'll randomly pick one of the platforms we ended up with SGX in this case and on the right hand side of the screen we have a small echo server this is written in Rust using the Tokyo IO framework and we're the only thing that's web assembly specific in this whole application is this line because we're going to create or we're going to use a socket that was already set up by the runtime and other than that this is all just native code so we're going to listen for new connections which we're going to accept when a new connection comes in we're going to spawn a background task in Tokyo and we're going to handle its output so we're going to read whatever the input is from that socket and we're going to handle a disconnection and then we're going to write it back so a very simple standard echo server and we're going to build this for web assembly so normally we would just do a cargo build but since it's web assembly we're going to specify the target wasm32wazzy and now our application is built for oh I think somebody's playing with the server they just took it down let me try the let me try the other one nope that's how you know it's a live demo anyway we could have uploaded this to the server and we could have shown it running on try.enarchs.dev if it was not down so maybe I'll come back to this at the end of the presentation see if it's back up maybe somebody's deploying a new patch or something okay so that's actually not what this talk is about this talk is actually about working and so one of the challenging things that we had with Enarchs is that we need to move on now to chapter number two testing your metal and yes I know this is a pun I hope I didn't injure anyone testing your metal is of course spelt M-E-T-T-L-E but we actually the problem we have is that we need their metal because when we're developing this software and we want to test it we want to test it rigorously because this is security software and so building Enarchs needs bare metal we test to development kernels we test hardware cooperating user space demons we do application development and we do it on heterogeneous hardware in heterogeneous clouds so all of the clouds basically are offering bare metal in various forms these days but how you actually boot an operating system across all of these different clouds is incredibly different and they're not really the same and we had other problems as well but this is a this is really a many headed dragon as you can see team at here and so we have a variety of different ways that are in use for booting bare metal you have the standard sort of pixie system which is you use DHCP it tells you the TFTP place to get your kernel in an RD it fetches those over the network and boots them that's one way another way is IPixie this is commonly used by equinex for example and so IPixie is like pixie but it's scriptable so you can basically write some scripts to do things and it also has like HTTPS support and stuff so it's not not just TFTP but it sort of has TLS support because IPixie only supports really old Cypher suites so if you actually try to download images from say you know github or pretty much any modern web server it's going to fail because it can actually do the TLS negotiation you also have the ISO method some places will allow you to upload an ISO and to boot that which is great usually you'll have to go through an operating system install although there's ways you can sort of make that automated then you have the EFI direct boot route which is where you put an EFI image on an EFI partition and that gets loaded directly then you have the operating system image this is usually a derivation of the ISO format or some other thing there's a a three letter cloud for example that requires you to have a custom image format for their servers and then lastly we had to mix this also with stuff on prem and so being able to just do like a usb mass storage boot for example is another way we may want to boot these things and so because we have all of these servers doing rigorous testing of of all of the hardware and we have all of these clouds and we have all of these different ways to boot things we really needed a system that could sort of wrangle all of these together so we really need to tackle that that multi-headed dragon and we have specific goals first of all we want our process to be iterative and building a whole operating system image is yes it's iterative but the cycle is just really long to be able to build a whole OS image we want to do something that was faster more lightweight second was we wanted something that was stateless we didn't want to install an operating system and then have to manage the state of that operating system over a long period of time we wanted to boot into an OS where everything came up with the with the bare metal and then when we rebooted it was exactly the same state as when it started before third was we wanted everything to be declarative so we wanted a clear recipe to be able to build our operating system images and we but we also test on multiple different operating systems and so while for example we could use something like kick start to do an install on fedora or red hat and other systems provide other automated systems as well that would have required us to maintain a bunch of different methods for composing these images across multiple operating systems fourth is we want the process to be reproducible this is incredibly important for any development or testing environment particularly when you're dealing with a very rapid development across changing situations you need to always be able to reproduce something at a particular point in time so we wanted to be able to reproduce every single OS build the fifth thing is that we wanted to be homogeneous so in other words if we're booting an operating system it shouldn't matter which hardware we're booting it on and it shouldn't matter which cloud we're booting on the entire process should be the same and further the development procedure for building that operating and booting that operating system should be the same for everyone across the line lastly we wanted it to be infrastructure-less and now I know that there's not really there's not really such a thing as no infrastructure but basically there's a lot of infrastructure that's provided for free from a lot of different places and we didn't want to have to set up infrastructure to manage our infrastructure easy and lightweight particularly as we had a small development team we didn't want to have to start doing IT ops immediately and so we simply needed more agility we needed the ability to move faster and to create environments that were reproducible and so forth so let's move on to chapter three Arming the party we already had as I mentioned some resources available to us so we start with Git right I mean if you haven't heard of Git yet you should probably go learn about it immediately Git is pretty much the de facto way in our industry these days to manage iterative changes and so we wanted to be able to use Git to manage the control of producing images that we could use for for booting and we wanted the whole thing to be declarative right so the image is produced in a declarative way from the recipe that is in Git and this gives us those two things we really wanted which is rapid iteration and change control second we do have available CICD resources there are many many examples of this all over the web I listed two here we have GitHub actions where you can run some code on their server in the cloud on your GitHub pushes or the same with GitLab for CI you know if you tag something if you do a push of code or whatever you can run code on all of those events and so this is essentially some free compute resources for us and we wanted to make sure that we could do this efficiently and lots of people are doing this today right this is a really well worn path committing a Docker file for example and building a container in GitHub is not exactly the bleeding edge of technology the last thing is that we have available container repos and so basically forges like GitHub and GitLab both have the ability to start with that Docker file that's in GitHub do the build of the Docker image and then push it into a container repo and now it's actually staged for deployment and so we didn't want to introduce additional infrastructure on top of this we just wanted to say like look we've got our whole operating system packaged as a container in that container repo I just want to boot the dang thing how can I get it to boot and so there's there's an easy part of this and there's a hard part of this the easy part of this is actually distributing the kernel and in in at RD because if you look at a container image it's basically a set of layers it's a set of tar balls that are layered and contains all the changes you've made to a container over time and we can stuff things in this and then we can distribute this right typically you don't distribute a kernel and in it ramfs in a container because it's not going to use that kernel or for anything and it would bloat the size of the container so but so we needed a way to do that and that's really easy actually you just kind of install the distros kernel package that's easy step out of the box there's a few other little tweaks you have to do but it's it's not very difficult the second part was the harder part and that's container aware boot loader so we needed something that could start up on the system no matter what boot process we had right whether we were doing iPixie whether we're doing native boot whether we're booting off an ISO or whatever we wanted we needed some bit of code there that could recognize how to get a container image and then figure out a way to actually boot that kernel and so we came up with the idea to make containers more metal and we started a project called working working is available on GitLab so gitlab.com slash working slash working and it's written in rust and I believe it's under the Apache 2 license and this is the bit of code that does that harder part in the early part of the the bootstrapping process working provides these four things these are the main things there's a few other things it does but these are the main four things that it does first is it is it does container muxing it basically takes a bunch of different container layers from a container repository starts downloading all of those it downloads them in order as they would be actually laid out on the file system when the layering was collapsed and we stream this whole process so we're actually un tearing them as they're downloaded and then we stream it into it in in our D image so we actually convert the docker format into an in it or D format one of the other things we need to do as part of this process is we need to actually extract the kernel from this image because although we can boot the inner MFS we need to get the kernel out separately to be able to handle that so that when we do the container muxing basically we're just grabbing the kernel from the container image as well as converting the container image into an inner MFS the second thing that we that it provides is K exec booting how many how many people are familiar with K exec here? Okay, we've got a few so K exec it was designed as a recovery mode for the Linux kernel basically the idea is that you can load one you've got your Linux kernel up and running and you can load a backup Linux kernel into memory and it's sitting there waiting to go and when your primary kernel crashes it can jump into the secondary kernel and now you can be able to do debugging or keep the system running and so K exec is an interesting tool that we can actually use so working is both a binary but it's also a that binary plus a very small operating system which includes in a Linux kernel and so first we boot the work in Linux kernel that's in the in the work in image we download the container do all of that stuff prepare the inner MFS and then we do a K exec pivot into the new kernel so you actually as part of your booting you're going to boot the Linux kernel twice the first time for the bootloader and then the second time from the kernel that's been extracted from from the container the third thing that work and provides is config persistence so we did have some way we did need to have some way to persist some information about that particular deployment so when you're actually deploying a container on bare metal hardware for example some clouds have fairly sophisticated networking setups maybe DHCP doesn't work at all and you have to set up bonding and static IPs that are associated with that particular server now because we have a goal for working to be stateless we can't just install an operating system and then configure it all so we needed some way to be able to persist that configuration across reboots and so the way that we do this is working takes that local state and persists in EFI variables so basically the first time you boot the system you're going to have to set it up for whatever it's networking whatever the needs are tell it which container to boot once you have it all working then you save that config into EFI and from that point on that server is just going to boot automatically the last thing that it does is network setup and I've already hinted to this on different clouds you have different networking scenarios some will just have it working with DHCP some will use IPv6 slack others will just have static configuration you're expected to set up things like bonding and your IP addresses manually and so work and also provides the ability to take that networking setup from EFI and then actually set up that network inside the the work and as it boots and so this is what it actually looks like to prepare a bootable container if you've ever used a Docker file that's exactly what this is basically the top just says we're going to take in an argument which is called tag and this is just so that we can iteratively build this on multiple releases of Debian and we're going to say from Debian so this is we're just going to choose to to base this on top of the the core Debian image and then we're going to do the next step which is to install the packages we need in order to get it working so the first thing we're going to do is we're going to update the metadata with apt and then we're going to upgrade all of the existing packages that are there so now we have the freshest base Reaper then we're going to install in-it-remfs tools and the reason for this is actually surprising the reason is is that if we don't install in-it-remfs tools it will be installed as a dependency of system D and the kernel which we're going to install next and the problem is we actually don't want to to generate in-it-remfs because the whole image the whole container image is going to be the in-it-remfs and so that can easily double that if you're building a small container for example generating in-it-remfs can easily double the size of the container image so all we're doing here is we're installing in-it-remfs tools and then we're linking over the in-it-remfs process with bin true and what this will do is that whenever the system tries to generate an in-it-remfs it will just immediately succeed and now we don't actually generate an in-it-remfs so lastly we install the kernel Linux image generic followed by systemd and now we have all of the packages available the last line by the way just clears the apt cache so that we don't store that in the container makes the container image smaller so now we have all the packages installed that we need to make a bootable container we need to do two additional things the first is that we need to sim link the kernel that we've just installed into the special file name working dot kernel and this is just so that working knows where to find the image you may have multiple kernel images installed or so forth and we didn't want to have to like do magical interpretation so you make the decision you decide what kernel you want to boot and just link it into the right place for us the last thing that you need to do is you need to actually set up systemd to run when the system starts and you just do that by by simlinking the binary to slash in it and that's it we've created a bootable container image that's that's really all it took this is the an example for debian but we have I think five or six different os's examples on the git lab site so you can go and check them out there all right so now we're prepared we've gotten our party we're armed for battle and the question is are we going to fight the battle yes we're going to fight the battle so we're going to do the battle for boot and this is what it would look like to actually boot this thing the first two lines here curl we're all we're doing is downloading the pre-built work in kernel and in not rd so all of this is is built in a reproducible way on git lab as you can see exactly what goes into those images there's no no surprises the last thing we're going to do is we're going to do direct boot so if you mentioned if if you remember before I talked about direct boot is one of the options this is where you basically load the kernel and interface directly and that's what we're doing with just we're using queemoo for this so we're going to launch queemoo and we're going to come back to the kernel command line in a moment but go to the second option which is the kernel and in order to you just see that all we've done is specify the files that we downloaded and that's the that's the work and boot loader the next three lines are just basic setup we're going to enable kvm and we're going to say we don't want any graphics for this particular image and that's because we're just going to dump all of the stuff to the terminal and record it that way and then the last thing is just to set we want four gigs of memory for this vm so that's that's pretty straightforward now let's go back to the kernel command line we can provide a command line here but think about it this way if working is an onion there's an outer kernel and there's an inner kernel and so you need to have the ability to pass arguments to the outer kernel which is the work and boot loader but you also need to be able to pass arguments to the inner kernel which may be different from the outer kernel perhaps the outer kernel on the inner kernel have different hardware support because they're different versions and maybe the the software behaves differently so all we do in this example the first thing we do is we just tell it which console to use which is the serial console and that's why we disable the no graphic well you can see all of the work and argument start with Wyr so Wyr dat dash arg this is passing log level equals three to the inner kernel all right so anything that you specify is just a normal kernel command line argument here will be passed to the outer kernel all the work and arguments all the things that start with Wyr are going to apply to the inner kernel the last one here is Wyr.image which is just the the slug to the the container that we want to boot and that's pretty much it and I'll actually give you a demo of this so first we're going to open up this this shell script and all this is is just a convenient way to put all the same stuff in there so you see all the same arguments formatted in a slightly different way and you can see we have the the command line as well as the kernel in an rd there and we're just going to run this Q image and so we're now booting the outer kernel in working that's what's happening right now it's it's being done silently and as soon as that kernel boots and the networking comes up we are going to start okay we've already started when now that we're going to deploy this container we're downloading that container right now where as I said stream converting it into an inner mfs and then finally we are booting it and so now we are actually booting the inner kernel which you can see here and that's it we've done we've done the whole thing so but there's other forms of booting as well so this is an exact same example the only difference here is that we're setting up ovmf as our efi and in qemu and we're booting an iso image instead in this case we don't have access to set the kernel command line directly so now we're looking to boot at a a little simple boot loader here to give us a menu to type command line stuff so this is what would be like if it was on bare hardware and you're booting off an iso image maybe from a usb key or whatever and so we're going to say there's the working argument again specify the console we're going to specify the image which is that debian image and the last thing we're going to do is we're going to save these parameters using the efi equals write command to efi and so now that efi image that we enabled with qemu when it was being booted has now persisted all of those options that we we typed and so on the next boot we don't even have to type anything that we did the in the first case because everything has been persisted to efi other than that it's exactly the same experience as before we're downloading our container image we are stream converting it into an nana ram fs and I do have a note here there's some very very old very obscure hardware that if you attempt to write to efi it bricks it and I just wanted to warn people of that fact but it's very unlikely you'll actually hit that anywhere in production and so we've written all of the variables now and we're now booting a second time so this is the second time of our boot the only reason I'm editing command line image here is because we have to put stuff over serial console but all of the other variables are now being pulled from from the efi and we'll see it again we scanned meaning that we found we found that information from efi we've loaded our kernel and now we're booting the inter kernel and that's it now keep in mind the entire operating system is memory resident now doesn't that cost you a lot of memory not really the reason for this is even if you are deploying a large operating system if you keep your container image small yes it's all going to be memory a resident in memory however if you're just keeping your container image the stuff that you use all of that stuff is going to get loaded from disk into memory anyway and the operating system is going to cash it so even if you installed that same exact container image to disk and then booted it normally you're still going to get roughly the same amount of stuff in memory plus or minus so it's it's not actually super painful I'll give you a quick overview of the work and boot arguments as we saw before weird.img tells you which container you want to boot weird.arg is any arguments that you want to pass to the inner kernel and that argument can be specified as many times as you have arguments to pass to the inner kernel the next one is weird.net and this one takes a little bit of explaining so there's a kind option and then a file section and key and ignore the kind option for a moment the file section and the key are essentially the structure of an any file so if you're familiar with what an any file looks like that's exactly what this is it's the name of a file there's a section in the file and then there is a key and a value pair in any file and that's because surprise surprise we're generating an any file using using this argument and the purpose for this is that it's actually generating the network D configuration so if you saw before we installed system D system D comes bundled with system D network D which has a full list of networking operations you can do very sophisticated networking setups with network D and so there's really the sky is the limit and this is a very very repeatable pattern basically if you say that you know if you put these arguments and it's just going to generate the network D configuration file during its boot and then network D will start up whatever that networking is when it's booted so we don't even have to document that for you as long as you can understand that those things transition to an any file just go read the network D man page and it will tell you more information than you ever wanted to know about networking the last two boot arguments are the EFI write and EFI clear options and these allow you to persist whatever changes you've specified to EFI or conversely clear them if you've said done something that's wrong and you can't get it to boot and you're just stuck in this boot loop you can specify the clear option it will clear out all of those options and then you can try again from scratch so the most important thing about working is that it is also itself a bootable container the only difference is that working is a pre-generated kernel internet ram fs so where normally you boot them on the fly the kernel internet ram fs that you downloaded are just generated ahead of time and that's the only difference and so what you're actually seeing here is that we are chain booting work-ins so in other words we're booting into a one version of work-in and then we're telling it go boot this other container where the other container is itself a work-in image so right now we've loaded the work-in itself as a second boot loader and we've now chain booted and so this can be useful for example if you were to write work-in into a ram for example where you could then use work-in to chain boot onto a network downloaded work-in and now you can have updates of the boot loader itself as you're actually doing your boot and so yeah that's it now we've we've chain booted work-in now you may be wondering after all of this what it feels like this is what it feels like to use work-in and anybody's Stranger Things fans yeah all right so that's that's my talk any any questions if you've got some hold up your hands so we can have the audience hear it it might be people will so how do you track whether or not you've already written EFI to individual devices isn't there some opportunity to actually mess things up there I'm not quite understanding the question the question was how do you keep track of whether you've written EFI to the devices or not you can always overwrite anything you specify as a boot parameter is going to override what is actually an EFI so there's always a recovery mode in terms of whether or not you've written before it sort of doesn't matter because you could just rewrite again and it is sort of the intent of booting work-in and if you haven't noticed from all the metal puns we're doing this for bare metal right and so the intent here is that you're going to have to be setting up a bare metal system even as good as the clouds are today and provisioning bare metal there is some manual process presumed to be involved we want that process to be as minimal as possible and then to rapidly iterate from there on out so hopefully that answers the question any anybody else do you support sorry that was not um do you support Cali Linux for this I haven't tried you're welcome to try it's a very simple Dockerfile recipe so if there's Cali Linux recipe you know Dockerfiles for doing containers with Cali Linux then install a kernel and make sure you have the right sim links in place and it should work but is Cali Linux system debased yes okay anything that system debased is going to be a really easy experience it starts getting much harder when you get outside of that realm because then you have to do things like modify handcrafted scripts to do that has certain assumptions about things but anything that system D is going to work very well anybody else okay I've I've got a few I was just curious about using this for the use case of handling edge devices that are unintended and wondering if this could be a mechanism to install updateable OS images out there at these edge locations absolutely and in fact it would be a great companion to use with NRX which is the thing we talked about at the beginning NRX if you if you think about it this way it draws in the security profile of your application so that when you're deploying that application on the edge which has a lower security domain than a traditional one you want that security domain to be as tight as possible around the application so you write and deploy your application using NRX and then you could deploy the operating system on which it runs using working and that's that's a great pair you might ask about the security of the host operating system in that case well it mostly doesn't matter that's the purpose of confidential computing the idea is to install essentially what is an insecure operating system and then have a secure trusted execution environment within which your app runs and the only thing that the operating system provides is resources resource management so how much memory is the application using right is it balanced effectively with other applications that are running and so forth but because it's running in a trusted execution environment the the kernel can't see even in ring zero you can't see into the memory of that application because it's all guaranteed by the hardware so yes it is actually a really nice setup for particularly those purposes follow-up then aligned with this trusted execution environment is there any value on that device of spending extra money to put a TPM module or some other module on the physical hardware or does this take care of the same kinds of problems anything you can do with a TPM you can also do with a trusted execution environment and you can do better with a trusted execution environment with a TPM you basically just get limited key management as well as secure boot secure boot is a great marketing phrase and what it means is that you when you turn on the operating system all of the components of the operating system are measured throughout the boot process and at the end the result of that you can perform cryptographic operations using that measurement in the TPM in the case of confidential computing the security of that chain doesn't really matter so much or at least not as much as traditionally it was required to and so you can deploy applications in a way that's a very efficient they have a much smaller much tighter security boundary any key provisioning for example doesn't need to go into the TPM because the application itself is encrypted while it's running so you can do an attestation to a remote party you can get in cryptographic keys or generate them internally so that no human ever sees them and get the public key signed by another party with attestation and then your application can run in this way that's provisioned but it's a very tight hardened security boundary somebody over here had a question but I'm still sort of struggling with containers in general overall and I'm a security person essentially what we're looking at here is just nested containers right I mean this is no that there's there's no actual container involved with this we're using the container format as a way to package and deliver an operating system that is downloaded over the network that's network boot booted essentially and so it allows us to reuse all of our development infrastructure for preparing our operating system images when it downloads the container image it's just a set of tar balls and then we unpack those tar balls in a very careful way so that it efficiently creates an init ramfs and then we boot that init ramfs so once it's booted there's no more container involved there's no container runtime there's no namespaces there's no container at all you just have an operating system but we use the container packaging format in order to efficiently deliver it okay that's cool so if the operating system if you're trying to actually boot something like again edge use case if you're trying to actually boot something that's effectively going to run a single application okay how much of the sort of a knit tooling do I actually need so I think it's if you're just if you're just trying to boot a single application yeah since you're like the the actual the use case that I have is is it's actually a device that collects data for sensors sure if you want to run your own in it process on that device and have a single process there's nothing that stops you from doing so so you could you could completely eliminate the system D step remember when we installed system D you could install the kernel but you could leave system D completely off of off of that you could install your own binary as in it and now it's running as PID 1 and that's all you have right you don't need any other infrastructure you probably do need some resources though for like when it goes down and when it crashes that's where system D starts to become very helpful but if you but if you have you know a hardware watchdog that can determine if the kernels crashed and reboot the system then yeah go out go go to to your heart's content we've got a couple more in the back but I'm sure I think we still have time okay follow up to his device access this might be more of an nRx question but at that edge use case is it okay for this inner OS to get to IO devices storage devices etc so if you're using nRx to actually protect your application then your application is encrypted at runtime all of its memory pages there's no way to tamper and look inside in fact if you go to nRx.dev slash crypto we have a hack challenge right now we're basically challenging anyone to see if they can hack the nRx system and get that data that's inside we yeah once you've basically built and deployed your application with nRx that's a very tight package right and just your application is encrypted using a separate key that key is governed by the hardware all of the memory is encrypted or decrypted in the CPU directly by the memory controller and the operating system has no access to any of that your application then runs nRx provides transparent encryption for all of your network operations provides transparent encryption or at least will in a future release for all of your disk operations and so once your application is in the nRx deployment system there is it never comes out unencrypted unless you intentionally do so and so basically all of the secret material you're used to trying to have to protect in the operating system level you don't have to try to protect that because you can't actually read into that memory inside of the nRx keep at all so the we can we can de escalate the necessity of having a really tight like secure boot image which is really really hard to provision I don't know if you've noticed we've had we've had TPM devices going on oh it's at least 15 years at this point and they're still poorly integrated into our systems and the reason they are is because it's an incredibly fragile system and one chain breaks the entirety of the system in fact I've even filed a bug there's a situation in Fedora where if you install if you install on a Windows device with secure boot the fact that it reordered the boot process breaks the encrypted drive in Windows right that's just how fragile the systems are when you when you have to depend upon secure boot for this and so what we're trying to do is we're trying to create a much better system where you can have full general compute available with the same level of hardware protections that you would expect out of a TPM talking specifically about working yeah you mentioned edge you can't say edge without arm sure is there any arm support on the working side there is no reason it won't support arm all the major arm systems these days are EFI everything's built using Rust which compiles find to arm I haven't actually built it and tested it but I would absolutely welcome contributions for supporting arm so I was wondering about the kernel do you have the kernel config like what's the is it like a standard kernel it's Debian it's Debian Debian skirmish okay yeah so there's no modifications on top of it yeah if you if you can be if you can boot the Debian kernel you can boot working okay last call any other questions I just went to the GitLab project I didn't see any documentation or anything on there is there anywhere where we can look up the documentation on there's pretty extensive documentation in the in the readme so if you go to oh maybe I just didn't look at the read in the right place and just scroll down there's there's quite a bit of documentation okay I think I didn't see that because I'm on my phone yeah no worries all right thank you yeah I I have the same problem too anybody else does this work with pixie or ipixie yes yeah there's basically you have the work in produced kernel and innet remf s you can boot those any way you want we also produce from that an iso image that just boots those that kernel and innet remf s so whether you use the kernel and innet remf s to boot or whether you use the iso image to boot you can configure it any way you want on your hardware and it'll it'll work so using pixie for example pixie just fetches the kernel and innet remf s from tftp over the network so just put the work in kernel and innet remf s on your tftp tftp server and you there you go it's booted any others okay well I want to thank Nathaniel this has been a great talk thanks for having me let's let's give them a round of applause safe journey home everybody okay this is the Canadian network of oh no is it no this is how how does that go there's this great wonderful pun but I can't get it rattled out oh yes this is the dominion network of the Canadian broad corp incestration check that does too check sounds good enjoy can you hi everybody we're still a few minutes from getting started but feel free to grab a seat feel free to grab some ice cream or some coffee or some snacks from the back of the room as a sweet way to end scale so if you are in the back of the room again there are plenty of seats come come come grab a spot even socially distant we are good and again enjoy some ice cream in the back of the room we'll get started promptly at three o'clock which is just a few minutes away so give away for some of the activities we did it this is the convention in Maryland yes all the parts it was a kind of a snap clock that he proposed for the moment yeah and I went to the room it had only gone out on the internet at the time Joe yeah it said I stood up and said is anyone technically yeah cool thanks for making it out Rick cool good afternoon everybody I know we're all super excited for the presentation in just a moment so I'd ask if we can start to start to settle down start to grab some seats there is not not many sessions that will will pack a room at four o'clock on a sorry three o'clock on a Sunday I know we're all super excited about this one so thanks for coming out again at the risk of you all running out of your seats one last time which I don't encourage there is still some ice cream in the back of the room for those of you that are looking for some there is some there is a okay maybe it's gone there is definitely some coffee and some other things again thank you to our amazing break sponsors with with that that that supported the event you know VMware matter most and linking this is what happens after four days of being here I will I will get that in a second and I will thank them hopefully they're not watching the recording data bricks yes sorry data bricks as well so thank you again so we are super excited to have everybody back here together after that two and a half year break for those of you that here we're here in the morning it's maybe closer to a four or five hour break but the thanks again we've got a few administrative things and then we'll get we'll go ahead and get started with our our main attraction with Dr. Sir so how many of you participated in the expo floor passport program slash scavenger hunt how many you feel like you got all the answers right okay wash less hand much fewer hands turns out only one did and he was tenacious in doing so he was tracking me down asking me for suggestions tips trying to figure out if I knew the answers I did not because I didn't put together the program and it looked hard so I didn't personally do it I'm also don't I also don't think I'm allowed to give myself a prize so we've actually got I see him in the room here so Kyle do you want to come join me on stage and tell us a little bit about what they have one and then I will tell you who it is everybody so what we have for the raffle winner today is a fully decked out leave room 14 laptop what makes it special so one of the things so when we made it we looked into what are all the things that we learned from all the past laptops we made that we think would make this our dream laptops so in addition to having hardware kill switches that shut off the kill the the a webcam and microphone and the Wi-Fi and Bluetooth it has it runs core boot firmware it also runs in it of open source embedded controller firmware as well that we that we developed and in general it's just like the best locked up that where we've been able to make so far so thank you very much Kyle so I'll give you one fun fact on Kyle Kyle is the only scale speaker that I know of in the last 20 years that we've been running the conference to have a literal explosion occur in their talk please I know it might sound hilarious but please don't leave cans of Coke directly behind the hot fans on projectors in the session rooms I'm sure nobody intended to do it but it made for a fun surprise for for for Kyle and a sticky situation for the hotel so I'll stop keeping you all wondering if you won the one person who did win is Dr. Sam Coleman in the back of the room come on up and for those of you that know Sam you've been coming to scale since what 2004 2004 2004 and you were at the time you helped us run the education track among other things of our call correctly the time or two the time or two open source software and education and so at the time I don't think you were Dr. Coleman at the time I think you were just Sam but congratulations I hope you enjoy this free software powered laptop open from again courtesy of the purism team and Kyle and before you take off today we'll want to do a quick picture for the purism folks as a thank you for that awesome so again for those of thank you for coming out for the closing keynote this is the first time we're doing this but we thought it was an important opportunity given we had we had what's had to do an auspicious speaker joining us and so mind just the last few things again we will be back here well not here but in Pasadena a few miles away next year March 9th through the 12th returning to our returning to our home after all of this COVID fun I appreciate you all sticking with us through the masks through the vaccine mandates through the the tests and everything else I know it's not fun I hope that by next year we can skip a lot of this no promises it's not pandemics are not in my control but I do know that we will be back together all things all things being being cool so with that in mind our next speaker doesn't I don't think needs an introduction but I first heard of him in high school I was reading a book called where the wizards stay up late the history of the internet if you've not read it it's a fantastic overview of just all of the early days of the internet how things came together at various at various institutions to build what we now most of us run our careers on today everything from how this conference was organized to what we did through the pandemic to how I get cat litter through the mail all organized and coordinated over to CPIP I don't know if Dr. Surf had that in mind when he did it but that was interesting so so yeah we're super excited the other thing that the interesting piece here is Dr. Surf did this while he was at UCLA and for those of you that don't know when scale started 20 years ago that was done by students at UCLA and USC and UCSB and CSUN and a few other local schools when we were all just but we we children in our freshman and sophomore years of college thinking that running conferences would be easy one thing I can tell you it is is that after 20 years the only thing I have learned is that it is not easy but but yes so super excited to have Dr. Surf and one of the things that has been a pleasure for me over the last 20 years is just getting to meet my my internet and sort of CS heroes as we bring them as we bring them on stage here for keynotes and for other activities and so without further ado I give you I give you Dr. Surf is joining us all the way from Virginia and appreciate appreciate him making the time as well as and for joining us so thank you Dr. Surf thank you all very much you know when people clap before you said anything my first reaction is to sit down because it won't get any better than that now you all clearly understand the tactic that's been employed here you wait until the last session of the four days of ruling fantastic I looked at the at the list of sessions holy moly you guys have got an enormous amount of useful information and I'm sure in the corridors in between lots of other very useful ideas have been flying back and forth so what you do on the last days you get the talking dinosaur to come out and and it doesn't matter what the dinosaur says the fact that he can still talk is amazing so I was I think I have been given a few suggestions though about things to talk about and I understand that when you get to be my age you tell anecdotes and so so we're gonna go looking to the past and look at a few anecdotes and I see more than a few gray hairs in the audience which means yeah I don't have any left but my beard is doing okay so for some of you this will be I hope a kind of a fun reminiscence of things past that have taken us to where we are today and where we are today is a big challenge and I'll try to close my talk with some thoughts about things that we might collectively do to meet some of those challenges and what I find most appealing I think about you in particular is that you've been doing this conference now for 20 years and there is this fellow feeling of collaborative sense of responsibility that you bring to the table that I wish everyone who writes software would bring to the table so we're going to come to to that in the last few thoughts of this talk but let me start out by jumping into the time machine for a second and take you back to 1969 at UCLA not very far away there were four nodes by the end of December 1969 in operation at these four locations which were selected by the Defense Advanced Research Projects Agency for particular reasons UCLA was selected as the first node because Len Kleinrock who is still a professor at UCLA ran the Network Measurement Center and my job as a graduate student there was to write software for the Sigma 7 machine to measure the performance of the ARPANET and compare with the queuing theoretic models that Len Kleinrock students were developing so we could compare what the predictions were from the queuing theory models and see what actually came out from the measurements and I can tell you that the queuing theoretic models were always beautiful and mathematical and pristine and everything else but they didn't always predict what actually happened in a real network and I suspected every one of you knows exactly why that's the case so UCLA is Network Measurement Center then SRI International was the second node and that was because Doug Engelbart was at the Augmentation Research Center at SRI International his belief and the belief of JCR Licklider who was running the Information Processing Technique Selfless at DARPA believed that computers could be a way of augmenting our capabilities and I would say that they were right our capabilities today what we do every single day are often augmented by the kinds of software that you write and other people write to help us do things that we couldn't do on our own not least of which for example was searching the entire world wide web so that was the second node and then third one was UCSB because they were doing some really interesting work on the presentation of complex functions on a screen so you could see what the computations were producing and finally University of Utah because they were very much involved in computer graphics at the time now you can imagine computer graphics in 1969 are not like what they are today but one of the guys at Utah invented one of the first hidden line removal algorithms so that you were doing 3D rendering and you wanted to show what it looked like what the surface looked like he had to remove the parts that couldn't be seen so he found it was John Mornock whom some of you will know started a famous company called Adobe some years later so that's 1969 many years ago and this is a picture that was taken in 1994 it was the 25th anniversary of the ARPANET on the left you see John Postel who sadly passed away in 1998 in the middle of Steve Crocker who led the network working group that did all of the host-to-host protocols the telnet protocols the FTP and eventually SMTP for email so he was the leader of that group and he was at UCLA as was John Postel all three of us went to the same high school Van Nuys High in the San Fernando Valley I don't know must have been something in the air but but we became good friends at UCLA as graduate students it took us all day to organize this shoot for Newsweek Magazine because we had to draw all those pictures with the backdrop hanging there then we had to go find the zucchinis and the yellow squash and the five five pound chins of coffee and then string it all together now this is Newsweek Magazine 1994 and we thought we would put up a geek joke for those who understood it so if you look carefully at this network you notice that it's mouth to mouth and ear to ear but there's no mouth to ear this network would never work so that was our little geek joke in 1994 but we were celebrating the that anniversary of the ARPANET because without it we would not have even bothered to do the internet so now I'm going to skip very very fast forward here past the original early designs of TCP and then later GCP IP this was a demonstration that I called for after I moved to Virginia in 1976 to run the internet program for the defense department and I wanted very very much to show that the TCP IP protocols could really work across different kinds of networks over considerable distances so we had a mobile packet radio network in the San Francisco Bay Area driving up up and down 101 radiating packets like crazy through a gateway which had been configured to send those packets all the way across the ARPANET which by that time extended into Europe over an internal satellite hop from ETAM West Virginia to Tonham in Sweden and then a landline to Norway to the Norwegian Defense Research Establishment and another landline down to University College London and then popped out of the ARPANET at that point into another gateway that led to the Atlantic Packet satellite net which was based on Idlesap 4a hanging over the Atlantic with multiple ground stations all vying for access to one single satellite channel so it was like having an Ethernet channel in the sky and then through the packet satellite network to ETAM West Virginia down to another ground station back into the ARPANET all the way across the US again to USC Information Sciences Institute so if you do the math of course going from the Roving Vehicle in San Francisco down to Marina del Rey is about 400 miles but the packets had gone through two synchronous satellite hops back and forth so it's about 100,000 miles and I remember when we did it and it worked I was jumping around in my office saying it works it works you know like it couldn't possibly have worked listen if it's you know you know that if it works if software works it's a miracle so so so that was a really important demonstration from my point of view as the program manager for this now we'll skip forward now we're in the early and mid 1980s after the ARPANET and the and those three network demonstration was done and further standardization was done then we implemented the TCP IP protocols on every operating system we could get our hands on in 1982 John Postel announced that we are going to switch over from the NCP host to host protocols of the ARPANET to the TCP IP protocols of the internet on January 1, 1983 so this is January of 82 we make this announcement and everybody sort of wumbles a bit you know you know everything seems to be working okay with the host to host protocols and filnet and so on so why do we have to do that and you say well you know if you don't do that I won't fund your program next year oh oh okay I get that so I had Dan Lynch whom some of you know was the founder of Interop was measuring how many implementations of TCP IP he could detect on the ARPANET and so he would report to me you know once every couple of weeks and I could see the you know the career was going up and then somewhere around the summertime it flattens out so you know I'm a bureaucrat at this point so what do you do well you clearly create incentives so I called the defense communications agency it's now called DISA and I said will you shut off the capability of the ARPANET to carry anything except TCP IP they could do that turns out there was a way to do that and I said turn it off for a day so they did of course the phone bringing off the hook what's the matter with you you blankity blankity blank you know can't can't get any email you know files nowhere and I said I just want you to know I can do that so you know oh okay so curve starts going up again then somewhere around late summer it's you know October or something it flattens out again so I called DCA turned it off for two days phone rings off the hook everybody made it on January 1, 1983 except for two guys and they pleaded you know for some kind of mercy and we said okay we'll give you another month or two so everybody made it up now how many everybody's were there there were only 400 computers on the internet at that point 400 as opposed to 400 million or whatever the numbers are today if you include the IoT devices and the mobiles we're into the multiple billions so now so it's now 1983 NSF is starting to invest in internet technology for hooking computer science departments up to the ARPANET and then to the NSF network backbone and they quickly run out of gas they essentially sent out an RFP asking for a higher speed network now remember the backbone speeds of the ARPANET were 50 kilobits a second okay that was that was broadband in 1969 so NSFnet launches with 50 kilobits and and basically runs out of gas it switches to 1.5 megabits and lasts for a little while and then it switches to higher speeds and we just kept going you know eventually we end up in the multi gigabit range so NSF makes this big investment they build a backbone network they build about a dozen intermediate level networks to serve about 3,000 universities the purpose for which is to help those universities get access to the five supercomputer centers that NSF and the Department of Energy are investing in so the NSFnet comes up and the supercomputer centers come up and not long thereafter NASA and the Department of Energy say this looks like a good thing so DOE builds the ESnet backbone and NASA builds the NSI or NASA science internet backbone so during the mid to late 1980s we're starting to see substantial implementation of course if we go to look at the internet today and now some of you will know I'm cheating right because this picture was actually generated from the BGP backbone back in 1999 but it was so colorful that I thought I would keep it because it lets me illustrate something that you know and that is that the internet is very dispersed and there are literally tens of thousands if not more networks running each operator picks hardware and software to run decides who to connect to and on what terms and conditions there were no central decisions about your business model who you connected with the only thing we asked of everybody is please run the same protocols so that you could interoperate we wanted to encourage Bob Kahn and I to encourage people to build pieces of internet find someone to connect to and let the system grow so that's what it looks like today except bigger and more colorful it is true that there is only one centralization element and that is trying to coordinate IP address allocation and domain name assignment so that they're unique and that's pretty important but ICANN doesn't dictate what is done with those things it just tries to make sure that if you want a domain name that it's assigned uniquely to one organization or an IP address block to an autonomous system so this is just my little memory of milestones I'm sure that every one of you who's been engaged in this game have your own favorite milestones so by no means should you take this as the definitive list of important milestones they're just the ones that I happen to remember one very important one comes after the internet goes operational in 93 and Cisco systems figures out that they can make money by selling routers to universities that want to get hooked up to the net and put local area networks in place oh by the way I did leave out something important 1973 Bob Kahn comes to visit me in my office at Stanford and says we have a problem what do you mean we and he says and he says well the ARPANET worked and now the defense department wants to see if it can use computers in command and control but he right away realizes if you want to do that the computers are going to be in mobile vehicles and ships at sea and airplanes and the ARPANET was built out of dedicated telephone lines connecting everything together well you can't connect the ships together that way because they get all tangled up the tanks run over the tanks run over the wires and they break in the airplanes never making off the ground so he was already starting to work on the packet radio net and the packet satellite net at the time that we met so we started working on the TCP design about a mile or two away Bob McCaff and David Boggs are busy inventing Ethernet at Xerox part so that was the fourth packet switching technology that was born during this 1970s early 1970s period so Cisco is the first to start commercializing this stuff the way we used to build routers was to find a computer and a graduate student and you wrap the graduate student around the computer and turn it into a router problem with that is we were running out of graduate students so Cisco figures that out also in 84 it becomes very clear that this thing is scaling up and that we can't go keep sending the host.txt file around to map the domain names into IP addresses so we needed something that was more scalable so Paul Maka Petrus and John Postel invented DNS and it evolves over time but it certainly has scaled dramatically it's a it's a pretty brilliant piece of work so around 1988 by this time Dan Lynch is doing Interop and it's up in the San Francisco Bay Area and I walk in in 1988 he started in 1986 this small thing it was mostly lectures but then it was an exhibit oh and the deal was you can't exhibit unless you show that you can interwork with everybody else so they bring out the show net this big fat yellow coaxial cable you all had to plug into it and then show that you could talk to everybody else in the show so Eric Benamo who is then the CEO of 3Com the company that Bob McHalf started to sell ethernet and I walked into the show and the first thing we encounter is a two-story Cisco thing you know display and so I turned to Eric and I said Eric how much do those displays cost and he says about $250,000 this is 1988 and I'm sort of sitting here and saying that's a lot of money and and he said that doesn't count the people who had to stay there and man the booth for a week and so I'm just standing there and my jaw is dropping thinking somebody thinks they can make money out of the internet that's amazing so so at at this point I'm starting to wonder because you know how are we going to let the rest of the the general population get access to this thing because up until that time all the networks were funded by the government agencies NASA, DISA or how to well yeah DISA wasn't also NASA DISA, NSF, DOE and DARPA so there was an appropriate use policy that said no commercial traffic will flow on a government sponsored backbone and you could imagine a rationale for that they didn't want government resources to support commercial activity but it became increasingly clear that even the people who were doing research under government grant actually needed access to commercial services that could be reached on the internet if the if it were permitted so at this point I'm working with Bob Khan at his company the Corporation for National Research Initiatives and I had taken a little break from the internet from roughly 82 to 86 to build something called MCI mail which is a commercial email service so in 88 I'm sitting here thinking okay so what could I say to the federal government that would let me break the appropriate use policy without really appearing to want to do that so I called the federal networking council which at the time was mostly NASA DARPA NSF and I leave out NASA DARPA NSF and DOE their program managers were the people who managed the internet policy so I called them and I said would it be okay if I tried to connect the MCI mail system to the internet to see if we could get the email stuff to interwork and to my surprise they said well okay you know you can mail for a year and this is just an experiment so by summer or so of 1989 we announced we have a gateway between the MCI mail system and the internet and as soon as we make that announcement all of the other commercial email service providers which are islands into themselves you can't talk to anybody unless you all have account so things like telemail and on time compu-serve yes all of those say wait a minute these MCI guys can't have this special treatment we want on too and so the government says okay so they all get hooked up and two things happen the first thing that happens is they all discover that all of their customers who used to be trapped in a wall of garden can talk to all the other customers of their competitors plus everybody on the internet because they were all compatible through the internet mail protocols and that was a little surprise and then later mail becomes almost free so much for that business model and but the second thing that happens about the same time maybe a little later in the years three commercial internet services pop up because we've just broken the AUP limitation so UUNet in Virginia and PSINet in Virginia and Surfnet in San Diego all get started in 1989 okay yeah this is anecdote number no whatever 17 Surfnet used to be spelled S-U-R-F-Net and of course you do that you're in San Diego what else would you do so so they had a whole campaign all laid out t-shirts you know surf the internet and all that stuff and then a couple weeks before they actually launch somebody discovers that there's an organization in the Netherlands called Surfnet which is you know it's a Dutch acronym and they are building a network to connect the universities in the Netherlands so they can't call themselves F-C-U-R-F-Net so they're Susan Ostrada was the executive director at the time and so somebody says well why don't we change our name to the California Educational Research Foundation Network because you know it sounds the same and then somebody says maybe we should call Vint so they call me up and they say can we call it Surfnet and my first reaction was you know if they screw this up am I going to be embarrassed and I thought about it it's more wait a minute now people name their kids after other people and if the kids don't come out right they don't blame the people they name them so I said sure so I flew out to California in 1989 and we Susan and I had one of the plastic bottle full of glitter and we smashed it on a Cisco router and we launched the Surfnet so by that time we're starting to see real commercial services pop up Rick Adams was the founder of UUNET and so we're talking 1989 so in 1997 he sold the company for two billion dollars to metropolitan fiber systems which on the same day was acquired by Worldcom for 14 billion dollars so he made out okay that worked that was good so just you know picking a few more of these things the big deal after commercialization of course is when Tim Rensley announces the World Wide Web which is late December as I recall and I don't think too many people actually noticed he was doing it on a next machine which was very cool at CERN but not too many people noticed and except for these two guys Mark Andreessen and Eric Bina at the National Center for Supercomputer Applications in Urbana-Champaign and they look at the text-based interface and they say boy it would be really cool if we could make a more graphical interface wouldn't it so they do Mosaic comes out around 1993 everybody notices because suddenly the internet looks like a magazine with formatted text and imagery and eventually streaming audio and video so that was a big deal and Jim Clark the founder of Silicon Graphics takes one look at the Mosaic browser and he says this is a big deal remember he'd started Silicon Graphics which had turned out to be based on another chip that ARPA funded called the Geometry Engine so he flies out and he takes Mark Andreessen and Eric Bina maybe a few other people back to the West Coast to start Netscape Communications in 1994 and by that time I had left Bob Conn's organization to rejoin MCI to put them in the internet business and the first thing they wanted to do is build the MCI mall okay so I fly out to Netscape and buy 7 million dollars worth of licenses for Netscape's browser and server and the first thing I asked them to do is to figure out how to avoid having my servers filled with partial transactions that won't ever get cleared away and I won't know when to get rid of them so I please store the partial transactions on the user's computer and so they went away and came back with cookies so if you're wondering where cookies come from you can blame Mark no don't blame me so so that you know of course they go public in 1995 the stock goes through the roof and the dot boom is on the venture capitalist in san francisco were throwing money and anything that looks like it might have something to do with the internet and this goes on for a while 1998 in the midst of the dot boom Google gets started by Sergey Brin and Larry Page Yahoo! gets started a little bit before that to surf the internet the interesting thing about the arrival of a worldwide web is that it triggered an avalanche of content that flowed into the net it was so interesting people were not looking for money for the content they just wanted to know it was useful for somebody else kind of like what you do so this avalanche pours in and pretty soon nobody can find anything because there's so much of it so they need they need a search engine some of you will remember altavista which came out of the digital research labs on the west coast and then Yahoo! comes along which was kind of more manual I think than some of the others were and then Google of course with its clever strategy called page rank which was very successful it didn't have a business model to start with by the way there was no business model but not very long after it got started and after they brought Eric Schmidt in as the CEO sort of adult supervision then this three-way business model evolved which was quite successful I can get started in 1998 and the original idea was that John Postel would be the chief technology officer for ICANN and it would manage the domain name system and IP address allocations through the regional internet registries as I say John passed away in September of 1998 but ICANN was really needed and so they proceeded now one thing I have not done is to tell you about the 1978 to 1993 protocol wars between the open systems interconnection model and TCP IP to say nothing of X25 and X75 and X29 and so on that would take too long so I won't do that but a lot of people imagine that this is just smooth sailing and it was just step by step and everything was all planned out and they get nope it was Sturmundrang for many many years and it's still Sturmundrang today so just pushing a little further into time here a few other things the dot bust happens in 2000s around April big lesson there a lot of the startup CEOs apparently didn't understand the difference between revenue and capital and in economics 101 says you have a finite amount of capital to get your revenue engine going and if you don't get the revenue engine going you will run out of capital and then what so lots of dead bodies you know of startups in 2000 but the internet kept going the demand for that capability was still very strong so YouTube get started in 2005 Amazon web services comes up in 2006 the iPhone shows up I want to emphasize the iPhone for a moment because all of you I'm sure know what a transforming event that was but now this is anecdote number 17 or 18 I guess some of you may know that the mobile phone was invented by a guy named Marty Cooper he was working for Motorola at the time what you might not know is that he started working on it in 1973 which is when Bob and I start working on the internet and Metcalf starts working on something going on in 73 I don't know what it was whatever we were drinking so so anyway this thing gets started in the same year 73 and it gets turned on in 1983 the same year that the internet gets turned on so Danny Cohen another name you might know is very much influential in the internet's splitting of TCP and IP because of real-time operation he was all involved in packet speech among other things so Danny calls me up in early 1983 and says come have lunch I have something to show you so I show up and he's got this thing sitting on the table it's about this tall it's got a whip antenna on it and it weighs about two and a half pounds and what's that he says it's a phone and he said well where are the wires there aren't any how does it work so he says we talked about it for a while and he says I don't know the answer to that why don't you call the guy that invented it so I called Marty Cooper on a Motorola brick which is what we called it then and the first question I asked Marty was how long does the battery last and he says about 20 minutes but you can't hold the phone up longer than that anyway so so Marty blesses heart you know and presses on he's still around he's still around he's in his 90s now he's just written a book about the whole story of the invention of the mobile phone but the iPhone is really triggering as every one of you knows when jobs figured out something that none of us realized that we wanted which is a device that had a camera it had the access to the internet it had access to the telephone system it had a touch sensitive display I mean it's got all of these amazing features all of which existed as a technology never put together in such an interesting way that transforms everything because suddenly the internet is more accessible anywhere you can get a mobile signal you get to the internet and of course the mobile phone gets more useful because it gets access to all the applications that are running on the internet so the two are mutually reinforcing it's a really powerful event so 2007 is a big deal in the world that we now inhabit in 2008 2009 several developments came out of the academic world ethane open flow and NOx this is basically software defined networking which really has transformed the face of building networks today and in fact in 2010 Nick McEwen and his colleagues started a company called NICERA to build software defined networks and they were acquired by another company it's very successful there's lots more I'm not going to try to repeat the last 10 years of development and you've lived those last 10 years anyway so you know them as well as I do but they are pretty astonishing and you guys are a part of that so somebody asked me to look back and say what would you do differently? and so I decided I'd put a little list together the first one is obvious right I would have done IPv6 first instead of IPv4 but because it's been damn hard to cause an incompatible introduction of a new protocol at that low level in the architecture that in our own defense though we actually did a calculation when we were doing the original design of TCP to see how much address space we ought to have and remember it was an experiment and we didn't know if it was going to work so we said okay it has to work everywhere in the world because it's going to be supporting the defense department command and control system so we said okay so how many countries are there well how many how many networks per country and we thought well how about two so there would be some competition then we said how many countries are there and there wasn't any Google at the time to ask so we guessed it 128 because that's in power of two and you know we did the math 256 that's eight bits okay so we know how many networks we got a deal of 256 networks and then how many computers per network how about 16 million you know which is at the time they were millions of dollars and they didn't move anywhere they were in air conditioned rooms and they were hooked together with wires but we thought what the heck and besides it rounds it out to 32 bits which is cool so and we thought you know that's 4.3 billion terminations if you could if you could allocate them densely of course you never would but even if you could and that was more than there were people in the world at the time so we thought that ought to be enough for an experiment now I want you to imagine you know that you're a young vint serf in 1973 and you go and your future self goes back and whispers in your ear and says 128 bits of address space and your younger self says WTF that's that's 3.4 times 10 to the 38th addresses and you say yeah and I don't think I can sell that it doesn't pass the red face test your network has never even been demonstrated and you're telling me you need 10 to the 38th addresses so I don't know if I would have gotten away with that now there is a huge mistake that I made in for mobility support and this this one it's amazing how you can fool yourself into thinking you solved the problem when you actually haven't I remember splitting TCP and IP and then we had to figure out how the TCP identifiers would work on an end-to-end basis so we created a pseudo header sucking the IP addresses up out of the IP layer into the TCP layer and use that for socket identification and since I had an operating mobile radio network at the time I thought we had dealt with mobility except for one thing I didn't think about the possibility that your mobile would move from one network to a different network that had a different IP address space and at the time I was thinking okay I know I can put another address space at the TCP layer so that it's okay to switch IP out from under the TCP and those of you who know about Qwik know that the Qwik protocols establish a cryptographic shared variable at the Qwik layer so if an IP address changes not both but if one of them changes you can reconstruct the network if they both change of course they can't figure out who to talk to so that doesn't work anyway I thought that the mobile problem had been solved clearly it has not been solved and so that was a big mistake and I regret that but at the time I was patting myself on the back for saving bits in the header so you know be careful what you congratulate yourself for the other thing that has occurred to me is that radio has wonderful features one of which is broadcast you can transmit in all directions if you want to and we don't use that in any of our protocols in a serious way we kind of could do it especially with synchronous satellites you can imagine a protocol where a bunch of stuff gets sent to lots and lots of receivers and the guys that didn't get it could raise their hand and say please send me another copy but we never really built any protocols to take advantage of that maybe that's something we should think about another thing that we could have done is put crypto into the system sooner than we did and a lot of people come and say you know blankety blanky idiot why didn't you put more crypto in at the beginning and we wouldn't be in such a mess that we are today I don't actually think that's true but at the time in 1976 when Whit Diffie and Marty Hellman published their first paper on new directions in cryptography it was stunning and I'm sure the guys in the UK were especially stunned because they'd invented this idea in 1974 but they didn't tell anybody because they didn't want anybody to know about how clever this was so anyway the paper gets published and the next year or so in 78 or so the RSA algorithm to implement this idea gets invented now you could say why the hell didn't you just immediately implement the public key crypto and my reaction remember it's 1977 I'm trying to get the damn thing demonstrated and implemented on as many operating systems as possible and I look at the RSA idea and I said this is retrofitable I can't put this in anytime and so I wanted to get the system up and running and demonstrated first so we did that we were actually working using DES which is a conventional symmetric key to build a cryptographically secure system and we were demonstrating that with a program called Black Crypto Red I'm sure some of you know the red side of the net is a sensitive side the black side is post crypto and we stuck DES in the middle but key distribution is not nearly as nice with the symmetric keying system as it is with the public key system but anyway so we were clearly working on on the crypto support and of course the NSA was busily doing some of its development work as well but I remember thinking at the time okay if I were serious about insisting on cryptographic implementations who are the users of this thing it's graduate students and I don't mean any offense because I was a graduate student once too but I can't imagine graduate students being really good about key management and all the other things that you have to do and so it kind of was comfortable not doing that and not insisting on it too early in the game and of course multi-factor authentication would also have been a good thing to have thought about if we had any technology to support it because even then everybody knew the passwords were a terrible idea so here we are we really want to add more security into the system and so from my point of view we really need work on BGP I'm sure all of you know how easy it is to either be hijacked or just make a dumb mistake configuring something wrong almost all the really bad stuff that happens most of the time is somebody just making a mistake sometimes it's not somebody did it on purpose RPKI is another thing that I would like to see more implementation Ditto DNS seconding so those are all things that we should be working on I also think that more strong authentication in the system all the way down to identifying a hunk of hardware and being able to authenticate that it is what you think it is would be very helpful and the internet of things really really needs something to keep your simple IoT devices from being hijacked like the what is that half a million webcams that were hijacked to do the DDoS attack against Dyn which had cascade effects because Dyn was doing everybody's domain name resolution for a lot of very important companies and they all disappeared off the net because Dyn fell over which by the way hang on to that thought for just a second because if you're like me you may be needing a change of underwear because if you think about how dependent we are on certain key parts of the architecture and you can see when they don't work there are cascade failures if your mobile doesn't work batteries dead can't get a signal some other problem then you can't do two factor authentication maybe or you just can't get into your email so the business deal you were about to close doesn't happen more than once even just over this weekend I had to go through a whole series of login and authentication steps some of which involve the mobile I now get two devices that I have to make sure work everybody in this room knows that the probability of success is multiplicative so 90% successful times 90% successful is 81% successful if you rely on both things working and if there's a third thing it just gets worse so we are steering right now into a very fragile future in my opinion and I hope those of you who are thinking about software development architectures and things like that will really give some thought to making this more robust this isn't just a question of more security this is more resilience more alternatives like every screen ought to be useful for this authentications if you need it things like that now some people have criticized the basic assumption that we made that every device on the internet should be able to talk to every other device and the reason that we chose that as our principle was that we didn't know which devices we're going to need to talk to which devices at the time and so we had no rationale for inhibiting communications now today we look back and realize that since everything can talk to everything then the bad guys can talk to everything and they do and they cause trouble so it could very well be that we should reconsider how to hide parties from exposure and there are some suggestions and you know the sort of clean sheet designs that are coming to that I've already mentioned about the dependencies so and I'm also very conscious of the fact that I can't even read my own watch now those of you who have good eyes will know that this is a Ronald McDonald watch and I only have 10 minutes left I got this for teaching a class in networking at Hamburg University just outside of Illinois I am not kidding there is a university they teach people how to run McDonald's and they were about to network them all together so they could keep track of the sales and the use of resources and everything else instead they gave me a Ronald McDonald watch to commemorate the thing so anyway let me just riff for a second on open source and open standards because that's what you do and I tell you what you do is really important I still consider it to be one of the central engines of the evolution of the internet your willingness to share your code and your thoughts and your ideas and everything else it's wonderful think about what happened with the worldwide web there was no class in being a webmaster but what you got to do with the browser was say show source so you could see how did they make this really cool web page because you could see the html and so lots and lots of people learn how to be webmasters from each other and that's what you do you learn from each other which is great it accelerates the pace of development it also open source also gives you an opportunity to find bugs now there is a little problem with this sometimes open source leads people to think it's open source everybody's already found the bugs because it's open source therefore you don't need to look yes you can stop laughing now so I do worry about that and I have pushed very hard wherever I can to argue we need to support people like you more so that you can help us make more resilient and more stable and safe software it's really tough because the curation of this software is really important some of the bugs that some of you know lie around for 20 years and they just don't get noticed until they surface it exactly the wrong time there's lots of supply chain risk factors that are involved because the open source software could be anywhere in the stack and a bug that gets in deliberately or a malware that gets in deliberately can be really troublesome so there's a real challenge for us as a community to support open source software in a sustainable way so I just want you to know I'm a big fan of trying to find ways to do that we need this stuff the standards and the open source for interoperability some people say standards inhibit competition but I don't agree with that I think that having commonality and interoperability allows for competition on top of the standardized platform and of course we all need to remember that we want to adapt our software to whatever new platforms that come along Kubernetes and containers being a very good example virtual machines and so on this is a slide which I don't have time to talk about but I want to just pause and ask you to look at it for a second because these are all problem spots and they are not the full list they're just some of the challenges that lie ahead some of them are really tricky because they involve international agreements of some kind whether it's treaties or norms or something else think for a minute about the world that we wish we lived in one where accountability is enforceable and that agency is given to people and companies who are using the internet accountability and agency and in order to make that work you may have to give up some anonymity because if you can't hold people you can't hold anonymous people accountable and so if they're deliberately doing harmful things you have to have a way of tracking them down it's a little bit like license plates it's not a perfect analogy but you know license plates or govilliga except mine it says serves up so most of them most of them are just random stuff but the police department is permitted to penetrate the veil and find out who owns the car that may not be the person who was driving the car but they can penetrate that veil because it's their job to hold people accountable so I think you should think a little bit about that the other one the thing that the last two bullets especially I want to say straight to you that getting rid of bugs is really important making mistakes is easy in the software world we do dumb things you know like buffer overflows or off-by-one bugs or hey we just we just read a variable and did a compare and a branch on it except nobody ever set the variable so it's a random number that gives you really predictable behavior so there's an ethical component here every one of us who writes software has an ethical responsibility to do the best we can to make it safe and secure and reliable you know in all fairness the programming environments that we have are not exactly helpful in that endeavor I was going to say suck but you know but this is a challenge to the academics especially to figure out how to design and build programming environments that actually alert us to the dumb mistakes that we might make so I think this is the last slide final thoughts and I am happy to do some Q&A if you have any so first of all we have huge challenges to keep the internet open safe secure sustainable reliable and connected and the reason that this is a big challenge is that governments around the world recognizing that there are problems in the online world there are harms are being committed against corporations and people and they want to do something about it now some governments are trying to protect the government other governments are trying to protect the citizens but it's a big challenge when the governments are trying to enact laws that aren't necessarily implementable from the technical point of view so we're faced with trying to preserve the value of this connected internet while we're protecting people from harm I also think that your work an open source is important for digital preservation I want you to imagine you have a lot of digital stuff and you want it to last so that your great-grandchildren can have access to it well you know when you have digitized anything you need software to help interpret it most of the time what if it's a spreadsheet or a photograph if you don't if you can't run the code that created the digital object a hundred years from now then it may not be available so we have a huge challenge in maintaining the accessibility of digital objects over long periods of time there's something ironic about this there are clay tablets in cuneiform that were written four thousand or more years ago and if you can read cuneiform which maybe three or four people in the world can you can still read that clay tablet because it was a warehouse receipt and the warehouse burned down and the tablet was baked and that gave it longevity then there's vellum sheepskin calfskin and that stuff lasts easy a thousand years and if you keep going forward to recording media and then you get to five and a quarter inch floppies three and a half inch floppies CD rooms and they last for one two three four five six maybe 10 maybe 15 years how about seven track tape nine track tape so we have a big problem and that's preserving our digital future and that is going to mean that old software needs to keep running somehow interpreters emulators all those sorts of things uh so we really need to do that and the second of the last bullet talks about accessibility and by this I mean making software accessible to people with disabilities and I'm I wear hearing aids and I've worn them for about 65 70 years but there are people who have all kinds of you know vision problems and motor problems and everything else and they often get the short end of the stick when it comes to accessibility it's hard to do that it's not falling off a log you really have to think about how am I going to make this application work for somebody you can't see or can't hear or doesn't have fine grain motor control or some other problem but figuring out how to do that is worthwhile because we're losing out on talents of people just because you can't see doesn't mean your brain doesn't work and finally making open source sustainable and trustworthy is the task beholden to you so I apologize that I didn't leave any time unless you want to stick around for a bit for Q&A but I really thank you for listening and I thank you for what you're doing keep that thank you thank you I am confident that a few people will take your offer up for a Q&A okay as long as you're as it was as it was a genuine offer and you're ready to spend a few more minutes with us and it seems like it was so I think Hannah you are somewhere in the room with a microphone if you would like to ask a question of Dr. Surf please yeah let me let me warn you that if I have trouble hearing the question don't shout that doesn't help it's just a matter of clarity but the nice lady with the microphone is wondering any ills yep can I so we have the first usually I run around like her all though with the microphone but but if everybody's wearing a mask it doesn't do any good because you can't lip read through the mask so not so much a technical question but having done all of this and let your life story what's next oh well I didn't tell you about the project that started in 1998 at the jet propulsion laboratory just after the Pathfinder landed in 1997 after 20 years of failure to get to the Mars remember the two Vikings in 76 and then nothing works 97 this little rover lands successfully everybody cheers so I show up at JPL the next spring and I get together with a team that did the comms for for the Pathfinder and we spend about you know a couple of days together and we're trying to figure out what should we be doing now that is to say 1998 that's going to be needed 25 years from now and we said okay we want an interplanetary backbone network so we started working on the design of the solar system internet we are now at the point where we have new protocols that have been standardized by IETF and the consultative committee on space data systems we are running on the International Space Station have been there for a decade we've done deep space tests with some of the NASA spacecraft that have visited the comments and things like that we are prepared to go to the moon with the Artemis mission there's a Lunanet design that Goddard Space Flight Center is working we are working together with ESA and JAXA and Cary as well and of course NASA so this is all coming together there is a group if you want to look closely it's IPN Interplanetary Network IPN SIG Special Interest Group IPN SIG.org is a chapter of the internet society so we are located nowhere on earth we are in the rest of the solar system I've been trying to trying to get JPL they issue me a badge that says Resident Alien which would be really cool fun fact several of the folks that running our AV here today are involved in some of the fun and exciting missions at NASA JPL so thank you all for running those cameras that are letting us share this talk on the internet so we've got one more question in the back yeah so there was a point on the previous slide talking about accountability and traceability accountability and traceability or something of that sort right and it's talking about how we wanted to to make sure that people were behaving well and to protect you know corporations governments and individuals from these bad actors a better question from my perspective is how do we protect ourselves from government bad actors also very good question again interestingly enough accountability makes sense in that context too because we would expect governments to be accountable for what they've done not all governments are willing to be accountable you and I would probably agree on that but if we establish a practice of accountability and we insist that that's part of the architecture and design then at least we have a shot at it I would not stand here and say I promise or guarantee that that will work but I believe that we should hold ourselves to the philosophy that people should be accountable and government should be accountable for what they do and of course one way to do that is to vote Hi so it's starting to look like with the rise of VR technology and AR that we're going to be in a lot more of virtual worlds but we're having a lot of problems just deciding on what standards to use and even the underlying communications technologies do you have any advice tips tricks that we could possibly use while creating these? So remember I probably don't have any better idea than you do so be careful whatever I say you should take with a grain of salt I had the impression that there were some really interesting ideas a while back remember Second Life and VRML now I don't know VRML intimately so I could be saying look at that thing and you would say I looked at it and it's a pile of crap but there have been attempts to find ways of sharing the descriptions of multi-dimensional spaces I think we're going to have to experiment with this a lot here's the one thing I'm hoping for lots of people are going to try 3D stuff because it's all very exciting we got chipsets that can do it we got headsets oh that reminds me this is going to be weird you know we've been spending the last two years doing the video conferencing and Google Meet and all these other things and people are saying yeah I can hardly wait to do this with the 3D you know headsets and I'm sitting here thinking how's that going to look because you got the camera here but you're wearing this thing and so you look like Darth Vader in fact we all look like Darth Vader and so in order to do this 3D conferencing you're going to have to have an avatar which means it's a new business model right you can have my avatar can have hair for example and you know you can rent suits from you know Ralph Lauren I think we're a long ways from getting this right the one thing I will notice is that the headsets have got to the point where there is less problem you know people get nauseated because they get signals proprioceptive signaling isn't that a great 50 cent word the proprioceptive signaling doesn't match your eyes are telling you're going like this in your bodies as you're standing here like that and your brain says this doesn't make any sense and so so then it makes you throw up which is you know thanks a lot so so I think we still have ways to go but it's getting better but I I think there's a whole ecosystem waiting and a lot of design and standardization I would love it if it turns out that those three dimensional environments could inter work somehow but that would require a lot of agreement and cooperation and commonality of you know headset capability and coding and everything else so guess what you get to try that out this is going to be cool we got one one more here Hannah it's up in front and then and then one on the side well you can yell but we want other people to hear you too so so you mentioned supply chain issues and specifically provenance software provenance do you have any thoughts on systems like Nix Gwix or Gytian have you heard of any of those those are not familiar so if you want to say a little more about what they are supposed to do is that well Nix is about reproducible build systems all the way to the executable okay so essentially your hash is part of the binary okay yes yes got it so I like that a lot I mean I like the fact that it's that you can't make alterations without they're being visible you can make the alterations but if you check first there are a lot of hardware boot systems do that right so I'm a big fan of that sort of thing it's another example of trying to make sure that the devices are equipped with the ability to figure out where did this stuff come from has it maintained integrity on his journey before I actually loaded in and booted up so I like that idea a lot I think we have one other question over there and then we need to wrap up no problem so thank you thank you for answering my question this is actually confirming an anecdote that Professor Douglas Comer told our class on the construction of the TCV IP protocol when in discussions with the DoD and that one of them thing was running around a city that has been destroyed by nuclear war and I was wondering if you could confirm or deny any of that right okay so there's great confusion in the history here Paul Barron when he was at RAND Corporation 1962 to 64 published an 11 volume series called on Distributed Communication he was talking about mesh networks this is 62 mesh networks and packetized didn't call it that call it message blocks for digital speech he imagined routers he didn't call them routers either relays on the tops of telephone poles across the country and so this big mesh could you could blow holes in it as long as there was some connectivity stuff would get through used hot potato routing which basically was send it to everybody and you know try to get it try to get it you know to the destination so that was his model and never got built but it was published then Arpanet comes along now the Arpanet was driven by an economic requirement Arpanet was spending money on a dozen universities to do artificial intelligence and computer science research in the 1960s and every year everybody said you have to buy us a new brand new world-class computers so we can keep doing world-class research and Arpanet said we can't afford that so they said we're going to build a network and you can share and everybody hated that but they said we're going don't don't worry we are going to fund all of you so you don't have to hide your results in order to have an edge on next year's proposals we want you to share your results share your software and share your computing capability so we can accelerate the pace of artificial intelligence and computer science so they in fact did build the network but it was not based on nuclear holocaust or anything it was just based on trying out a packet switching technology which we believed at the time would work a lot better than dial up a computer send some data hang up dial up another computer send some data and hang up we didn't think circuit switching was going to work for our bursty kind of applications of course the telecom cannon of the day was of course you do circuit switching that's how we've been doing it for 100 whatever it was 70 years or 50 years and so we asked AT&T would you like to participate in this thing they said no it won't work but we'll sell you dedicated circuits if you like so you can build your stupid network and so they did and we did and it worked so now when the internet comes along I was worried about exactly the problem of recovery from major failures but and we even did an experiment to figure out what happens if you had a partition network Radia Perlman figured out how to do the routing system to recover from a perforated or bifurcated network but all in all honesty none of those protocols were ever tested in the kind of nuclear holocaust that really blew a lot of pieces apart I actually did however just to demonstrate it fly packet radios in the strategic air command bombers and and and basically cut up pieces of the Arpanet and then glued them back together using ground and air based mobile packet radios just to demonstrate the TCP IP would link the pieces back together again but it wouldn't it would was not mature enough to deal with a serious post nuclear scenario so the real answer is none of this was really built to do that it was built to figure out how to get computers to talk to each other okay got to wrap it up but thanks again really appreciate you have a good evening thank you for joining us and for for sharing with us and and for helping to you know be part of creating many of our careers so so with that scale is officially over at least for those of you that are attendees for those of us that are wearing orange shirts and jerseys and and other embroidered materials we have a couple more hours of four days of cleanup and and wrap up if you want to make that easier for us next year we are always looking for additional volunteers and team members but with that we will see you again March 9th through the 12th and Pasadena it's only about six months away which I think just I think I just scared my team quite a bit nine months away they're like they didn't want me to steal three months you can be part of that make this make this journey easier for the rest of us thank you very much