 All right, so this talk is about being a wizard. Like, you know how sometimes you see someone doing things with the computers and you're like, how did you do that? You're amazing. Okay, so computers aren't magic, which is a barrier to us being wizards, I think. So instead of talking about magic, we're going to talk about how do you learn hard things and understand really complicated systems. I'm going to talk about me for a second. My name is Julia. I have a blog. At work, I work on Kubernetes, which is some container thing. It's not important what it is. But the reason I say this is that last year, I didn't really know what Kubernetes was. And Eric just gave this great talk about containers. And a year ago, I was very fuzzy on the concept of a container, and now it's like my whole job. And now I'm one of the experts at my company on this thing, which I fairly understood last year. And so I kind of want to talk about how do you go from being like, I don't know what this thing is, to I manage an extremely important system based on this thing? And I feel safe doing that, and I don't think I'm going to get fired. So I need to work with a bunch of different kinds of technologies, right? I spend a lot of time thinking about Linux and networking and TCP, and how does TCP work? Anyway, we use AWS. And if you've ever worked with AWS, you know that there are three bajillion different AWS sub-services. Anyway, I learned about a whole bunch of different things this year. So my slides are a little whatever. It's fine. We're all fit. Right. So I want to talk about kind of like having this attitude of like how do you go from being like, oh, I'm bad at this. I don't know how to do it, to like, I can learn this. No problem. I can just be a wizard. So we're going to talk about like six different skills that I've tried to get better at over the years. So the first one is like understanding your systems and trying to figure out how to understand like really complicated systems, which are the ones that we deal with and how to ask really good questions. We'll talk about reading the code. We'll talk about debugging. And then we're talking to talk a little bit about like designing. I'm like, how do you get better at designing systems? OK. So let's start out with understanding your systems first. I'm going to talk more about like understanding like low level systems just because that's what I do in my job right now. Because I got excited about like, I was like, I want to understand the guts of the computers that I work with. And I want to know all about like networking and like how that works. So that's what I do now. But hopefully some of this will apply to anything, even if you are not like immersed in the details of computer networking and that's not like your dream. OK. So like, why is it important to like understand how stuff works? Because you can actually get by without understanding how a lot of things work, right? Like it's pretty easy. So one reason I think it's useful is just to understand like jargon. Like if someone says words, like if someone says like user space, like if you're working with containers, it's like useful to know what that means. Or if someone uses the word like event loop, for example, and you're doing async programming, it's useful to know what an event loop is. But more importantly, I think if you're trying to debug like a really tricky problem, I think it's really important to know like what lives underneath that problem, right? So you're not just like what's happening and like googling sadly. I mean like why is error message help? If you understand like what lives beneath the systems you're working with, I think it makes it a lot easier to debug things. Another reason I think it's useful to understand what you're working on is that it can let you like innovate, right? Like if you want to build a new library, like if you want to build this cool like robotics calibration library, then you need to really understand the details of how everything underneath it works, right? And I think it's fun to innovate and build cool new stuff for other people to use. Okay, so like let's say you have a whole bunch of stuff you want to learn about and there's this like huge mountain in front of you and you want to start like chipping away at it to understand little bits. One thing I really like to do is to try to understand just like fundamentals and if you like okay I want to understand like what we were learning about like calibrating robotic systems and there's this idea of like a kinematics model. It's like what's a kinematics model or like what are these parameters like alpha and theta? Like what do they mean? And like I like to kind of like you like start at the beginning, right? And you're like okay, what does this word mean? Oh no, to understand that word I need to understand eight other words. Cool, great. I'll do that. One thing I did when I was trying to understand TCP and networking is and you can do this as homework if you're excited, is I built a TCP stack from scratch in Python and it took me like about a week to do it and then that helped me understand really exactly how TCP works. And now when I try to debug TCP programs I'm like I know the basics, I did it all myself. And even though like the networking stack in the kernel I think is like four million lines of code and mine was like 200, like writing like those like 200 lines taught me the basics, right? Even if I do not understand like whatever the four million lines of networking code in the Linux kernel. So another thing I like to do is try to figure stuff out is to do experiments. So like let's say you want to understand a little bit better about how memory management works on your computer. You can just be like okay, let's just make my computer run out of memory and see what happens. That could be fun. It turns out that what happens is Linux will just start killing things for you because it will be like, oh no, there's no more and it will just kind of start killing stuff at random, which is exciting. One really exciting thing that can happen is if you on purpose run out of memory during a presentation and then Linux kills your presentation software and it makes a very dramatic end to your talk, which is not a real thing I did but it's just really that I partnered on purpose and I think it was like a majestic work of performance art. Anyway, but I think it's fun to see things happening instead of just reading about them in a textbook, right? And it's just a computer, right? You probably won't screw it up that badly. I haven't managed to ruin it yet. Another kind of experimental thing I did was I tried to write an operating system. I realized rapidly that that was impossible, so I would take too much time. So I settled for just running a keyboard driver to figure out exactly what happens when you press J on your keyboard and it displays to the screen. And now I know. And I was really happy. So I have some rules of programming experiments that I apply, which are like, I don't really require any of them to be good, right? Because it's not important that it be good code and doesn't necessarily have to work that well. I wrote this TCP stack in Python and I could make a network request to Google through it and Google would reply to me and it would send me a response. And then after that, things would kind of go wrong and Google would be like, what are you doing? And then it would disconnect me. But it was fine, right? I managed to send my request to Google and get a response stack and then things went wrong. But I was still so happy with myself and I learned a lot. About how TCP worked. And I thought that was cool. Another thing I like to do is read books. Some books are really good. I like reading networking books. So there's a book I like called Networking for System Administrators. Even though I'm not a system administrator, I learned a lot from it. Another thing that I think is kind of useful is to actually read things that are kind of too hard where you read it and you're like, I don't understand what many of the words mean in this. There's this website called Jebson.io which does a lot of database analysis. So it looks at databases like MongoDB or Elasticsearch or a lot of others. And then it tries to break them. And then it's like, I did this and then the database broke and it didn't work properly. And I'm like, what? Why? What's happening? Should I be worried if I'm using that database? And so I think this is a really cool website. And I like reading the articles in part because they're kind of hard to read and I've learned a lot by reading them and looking at the terms. And of course, it's one way to learn about new concepts is to work with them in your job even if you're not quite sure what they are. So once I needed to work with it, we have an HTTP proxy that we used to send requests to the outside world. And I kind of thought I knew what an HTTP proxy was. But then someone was like, oh, Julia, can you add some logging to it? So I did. And then I found out that I actually didn't understand it that well and I learned much more about how an HTTP proxy works exactly. And I learned that it was harder than I thought. So that's kind of one of my favorite things actually is when I go to work with something that I thought I knew and then I'm like, oh wow, I didn't know this. That's exciting. So this happens to be constantly, right, is that I think I understand something and then I learn a new thing and I'm like, oh no, I don't understand this. And I feel like if you want to get really good at the systems you're working with, I think it's important to actually be like, no, why doesn't this work? I'm going to figure it out and actually dig into the problem. So for example, I had a machine recently which was swapping. So it was like writing memory to disk instead of, well, instead of not doing that, right, it started swapping memory to disk. But it was only using 12% if it's available memory. So I was like, why are you swapping, computer? That's stupid. You have three gigabytes of memory left. I don't understand. And I could have just left it at that because it didn't really seem to be causing that big of a problem. So I was like, this is probably fine, I guess. But then I decided to actually learn what was going on and I figured out that there's actually four different reasons that a computer can swap, which are that it could be actually out of RAM or it could just be mostly out of RAM and depending on the VM.swappiness setting, then it'll start swapping anyway. Or you could be using containers, which is the problem with me. Containers give you a lot of new classes of problems which didn't exist before, which is very exciting. So here my machine had lots of RAM, but the container was out of RAM and so it decided to swap, which was kind of dumb. Or you can have a bed. Anyway, you're probably not excited about all the different reasons a computer could swap, but I feel like the equivalent of this for whatever thing you actually are interested in can be kind of cool, right? Because you're like, oh, wait, there are so many more. There's this universe of things that I didn't know about. This is great. Oh yeah, specifically for swapping, there is a 600 page book about understanding the Linux Virtual Memory Manager that you want. And it has 200 pages of docs and then also 400 pages of code references where you can read the Linux kernel code for the Virtual Memory Manager. If that's something you want. I've been trying to learn more about Linux internals a little bit on and off the last four years. Before that, I used Linux for like 10 years. So I think trying to understand complicated systems takes a really long time because my mom bought me a computer in 2003 and then I decided to install a lot of Linux distributions on it because that was what I did when I was 15, apparently. And then I learned what a system call was in 2013. Anyway, okay, let's talk about asking questions. Asking questions is one of my favorite things. I feel like it's an important skill because I often have all these great people around me who know a lot of things that I don't know and they're not just going to tell me. Sometimes they are just like, Julia, did you know about how TLS works? And I'm like, I think so, but I'm not sure. Sometimes people at lunch just decide to tell me stuff, which is great, but usually I have to ask them first. So I think of the goal of asking questions. It's like, what is a good question, right? I feel like a good question is one where I ask you a question and then it's really easy for you to answer that question. You can answer me right away because you understood the question and you know the answer and then I learn really quickly. So you want to ask questions which are easy to answer. So how do you come up with a question which is easy to answer? So my favorite first thing to do to ask a question that's easy to answer is to say what I know. So let's say I'm asking a question with databases. I could say, oh, if you write a lot of data to your database, sometimes the hard drive can't handle it and then you have problems. So I'm going to be like, oh yeah, that's right, but maybe there are some edge cases there. And so this is cool. I like to do this because it helps me to organize my thoughts. Also because it helps me to reveal things that I don't understand because sometimes I'll be like, I know X, Y, Z and people will be like, not quite. X and Y are right, but Z is not super right. And then this is also good because it helps you avoid answers that are like too basic slash too advanced, right? Because it's kind of a waste of time if I'm like, tell me about like, I don't know, databases and someone's like, well, there's this language called SQL and I'm like, I know about SQL. That's not, I had a more complicated question or less complicated question. Another thing I like to do is to avoid asking like always the most experienced person because I think what happens a lot is sometimes you'll have someone who knows a lot of things and then everyone will just ask them all the questions and that can be kind of like tiring for them and like not a very efficient use of resources because you just have kind of have like too much resource contention on that one person. So what I like to do sometimes is like try to find like maybe a less wizardy person who I think will still probably know the answer to my question and then it gives them some like more experience answering questions and helps them like build up their knowledge a little bit even. And then I'll probably get the answer to the question I have anyway. And then it saves the other person who is constantly being bombarded with questions sometime. Yeah, that's why. Another thing which is good to do, which I think you hear a lot about, is to like do research right and be like, okay, I will Google the word like kinematic model first to try to understand what it means. I have not yet Google the word kinematic model. So I should do that before asking questions about it, I guess. Yeah. And then you can be like, well, when I Googled, I found out this and now I have a question and now that I've figured out a couple of things. Okay, so I was talking about kinematic models. I thought this talk was super cool about like calibrating robots and I like learned a lot from it and it was super good talk, right? But like one thing that happens I think to maybe most of us, certainly to me a lot, is I'll like find something I'm interested in and that I want to know more about. But sometimes I'm like deeply confused about the whole space. And I'll have like, let's say like with containers. When I started talking about containers, I was like, what's a container? What's happening? Can you have more than one process in a container? Or is it possible to have two processes in a container? Is a container even a real thing? Or is it a fake thing? Do you need to have Docker to use containers? And for me, I feel like asking, like trying to split my confusion into yes or no questions really helps me because you can be like, okay. Can you have more than one process in a container? Right? And the answer is yes. And then it's like, can you run more than one operating system on your computer at the same time? Like can you run like Ubuntu? And then also like Red Hat at the same time. And then the answer is also yes, which is weird. And then you're like, okay, cool. And then you like assemble. Then I can like assemble these facts about the world and I can be like, okay, how can I like build a model of the world, which makes all these like yes or no questions make sense. And I feel like that helps take me from being like, I am deeply confused to like, at least I am a little better oriented. Yeah. And also yes or no questions are easy to answer, right? Because the answer is either like yes or no, usually. Or sometimes like your question is malformed. Like that question didn't even make sense. But I actually feel like the skill of being able to like answer, like ask like a well-formed question, even when you're confused is like something you can develop. Like I attempted to ask a question about kinematic models in this talk, which I hope was well-formed. And then I got a really good answer. Thank you. It was so good. Hi. Okay. New question asking technique. Sometimes someone will do something that you don't understand. Right. Sometimes something will do something and you like won't understand how they did that. So what I like to do is just be like, as soon as someone does something that I want to know how they did it, I'll just be like, how did you do that? Can I watch? Can I see? Please explain. Yeah. Ideally not like a too intrusive way. But I find that that really helps me. And another thing I like to do, especially as I become more like senior, is ask questions in public, right? And to like not just like ask someone secretly while like hiding behind the couch, like how did you do that? What's happening? Like instead I'll just be like go into our public Slack channel and be like, I don't understand what this is. Can you help? Right. Or like, can you explain how you did that? And like, I think that especially like as someone who's like more senior or like more of a leader, it's really important to do that. Cause then it like opens up space for less senior people to ask questions too. Okay. We're done with asking questions. The next thing I wanted to talk about is reading the code and like especially like reading the code, cause we all have all this open source that we work with, right? And the delightful thing about open source is that you can read the code for all of it and then learn things. So this is cool because it lets you, one of my favorite reasons to read the code is just that I'll get an error message and that error message will be completely unhelpful. Like it'll be like the thing is broken and like, thanks. Like, but then you cannot correct that person's choice to make the error message it's broken in the past. So you can just like grab the code face for like it's broken and then try to figure out like what actually happened. So that's awesome. I think like reading the code has also helped me get around just like problem, general problems with documentation a lot because a lot of things, especially like when you're in some weird edge case are like really not well documented. But then if you just go read it, you can see how it works. One example of this actually is recently someone wrote a blog post about async.io in Python and they were asking for someone to review it and it had a bunch of like facts in it, right? Like it was like, oh, async.io does this or does that. And I wanted to like fact check some of them just because I was interested. But then when I tried to Google to fact check these like facts about async.io, I couldn't figure it out. So I was like, well, async.io has code, so I can just read it. And so like I went into async.io and tried to like start figuring out how it worked and it turned out to be totally readable, which I thought was really great. So like async.io has this event loop where like it'll like do the loop, do a thing and then come back to the beginning and you can interpret event loop in a pretty literal way, right? Which is that there's a loop and there is a loop. This is it. This is a while true. Do the thing and then if it's stopping and this goes on for quite a while, right? Like this is in a 1400 line file. But I thought it was cool that even with like, I mean async.io is like part of core Python. It's like not the simplest library. It's a few thousand lines of code. But I could just go in and start reading it and figure out what was going on. And like the more you do this and the more you practice like reading unfamiliar libraries, the better you get at it. And I feel like it's really like kind of a wizard skill to be able to be like this code is completely undocumented and I have a question and I need to know the answer to this question and I can just go figure it out. No problem. What do you need? And I think I really learned this from like my first boss where I was working with Drewful which is this like PHP web framework and I would ask him like, how does this work? I don't understand and he'd be like, Julia, you just have to go read the code. Like there's these docs you're looking for don't exist, right? It's not happening. Okay. So, we're done reading the code. Let's talk about debugging. Debugging is one of my favorite things. The reason, one reason I think debugging is so good especially like for getting better as a programmer is that when you're debugging I think you're often in a place where like especially if you're spending a long time debugging something you're in a place where there's something you don't understand, right? Because if you understood, you would have fixed the bug already. Probably. So, if you're really stuck, there's probably something like a learning opportunity there somewhere. So, I'm actually... Okay. I think I've gotten a lot better at debugging over time and I was thinking about how I got better at it. So, way one is to remember that your bug is happening for a logical reason because often I'll see a bug and I'm like that's not possible. That can't happen. I coded it. It was not supposed to do that. But, of course, there's always a reason, right? Like sometimes that reason... My friend has a really good debugging story. My friend Allison has a good debugging story about how like sometimes the reason is that your memory is being corrupted. And she would have... She was working with these like really old windows machines, like very, very, very old, which had really bad RAM and so their memory would get corrupted and then they would send back corrupted memory. But that's still a logical reason. If... Unlikely. And so, I think it's helpful to remember that there's always a reason because otherwise you start to think of those gremlins and it's like a very unhelpful train of thought. Another thing that helps me is to be confident that I can actually fix the bug. This is relevant because sometimes I try to fix bugs which will take me like days or weeks to fix. And so, I feel like it would be kind of embarrassing if I spent like two weeks trying to fix a bug and then I couldn't do it because then I would have to be like, oh, sorry. I just like wasted two weeks. But in practice I'm usually able to fix the bugs that I said I had to fix even if it sometimes takes me three weeks. Like, once I tried to fix this like Hadoop, MapReduce, like data processing job that was taking way too long to run. And then I just kind of like worked on it for a long time and then eventually I fixed it and it was super fast. While doing that, actually, this was kind of interesting. So this job was processing like a thousand records a second, right? And a thousand things a second. It seems like a lot. Another thing I found out is that floating point exponentiation is really slow. So you take like a floating point number and raise it to the power of a number. That's like a very slow operation, kind of shockingly. And like this job was actually spending like it's like 98% of its time just doing floating point exponentiation for no reason. Okay, so a thousand records a second, right? I kind of felt like that was slow. And then like some of my teammates disagreed with me, they were wrong. They were like, that's fast. That was a thousand is a lot. And I was like, I think it's not a lot. And it wasn't a lot. So I wanted to kind of like train my intuitions about like what is fast and what is slow. And we're going to play a game really quickly. Let's find the game. Okay, this is the game. I'm going to read this to you so you don't have to read it. So this is in Python. So let's say I have a dictionary, an empty dictionary. And then I just like put numbers in that dictionary. So like I add new entries. How many entries do you think you can add to this dictionary in one second? A million? Okay, let's see. 10, 11 million. That's so many, right? And people say Python is slow. You can put 11 million entries in a dictionary in a second. We could do another one. This is one where you parse an HTTP request. So I have some HTTP request here that I wrote. And the request for our thing is all in Python. So how many HTTP requests do you think you can parse in a second? You have to guess something. 100,000? Okay. You're right. It's 25,000, which is close enough. The goal is just not to be off by more than 10 times, right? Because you have one to one billion. And it's often, I think, really not obvious. Maybe we could do one more. Oh, we can download a web page. That one's kind of cheating. Oh, this one's kind of interesting. How many times can you run Python? How many times can you start the Python interpreter in a second? One. Okay, we'll see. Let's see. No, it's more. It's actually 77. Which is substantially more than one. I would argue. But also substantially less than, like, a million, right? So I feel like it's kind of useful to get intuitions for some of these things. And just to be like, okay, I think I could do... It's like more like 100 than a million, right? Like, yeah. And if you're just, like, adding two numbers and it's only doing it a thousand times, it's probably something wrong. Okay, so now we know about computers being fast. You can also play this game or, like, tell your coworkers. I think it's good to give it to people who are, like, arrogant, because, like, everyone gets it really wrong. I still make mistakes and I broke the game. So I think it's really fun. Another thing that's useful for debugging, of course, is to, like, know your debugging toolkit. And because when you're debugging, you often, like, ask a lot of questions of your computer, right? Like, you're like, is it, like, maybe a network request isn't working and you're like, okay, is the request getting to the server or not, right? Like, did it ever make it off my computer? And, like, having good debugging tools can help you answer those questions. I have a zine about debugging tools for Linux that I love. It has a cute penguin on the cover. It's on my website. And it talks about some useful debugging tools that you can use on Linux. And then, as I said before, the main thing I think about debugging is that I learned to like debugging. And I realized that I learned things and it's kind of interesting. Okay. How are we doing for time? I don't know. Does that mean that we're done? Great! Perfect! That means we can skip to the end. Oh, this is my talk title in Chinese. Thank you. This is the advantage of having unconnected sections in your talk as you can just be like, those ones won't happen. Do you have questions? Yes. Okay. This is a good question. It's like, how do you find people to ask your questions, too? I mostly ask questions with my coworkers, but I also ask questions of random people on the Internet. I think for that, it's good to ask, like, yes or no questions sometimes. Yeah, so this is a good question. It's like, you have someone who you think might have the answer to your questions, but you don't know. You don't know which one to choose. Well, I feel like you have to... You have to choose someone. But actually, I feel like asking exploratory questions is kind of an interesting skill, because you obviously don't want to quiz people, right? Sometimes I have an area of questions that I'm interested in, and I haven't found anyone to answer my question yet, so maybe I'll have some standard starter questions I start with, and then I just ask that question to a lot of people, and I see if anyone gives me an answer to it, and I'll like, yeah. Does that make sense? Oh, yeah, sometimes... Yeah, and I feel like it helps to have a really good question, too, like, or a question which is like, maybe kind of like, click-vady, or like, where someone is like, oh, I want to know that, too, right? You know? Like, because some questions you'll ask someone, and they'll be like, I don't care, and some questions you'll ask someone, and they're like, wait, like, and they'll like, join you on your mission. So I feel like it's good to have questions to make you people want to join you on your mission to find the answer. Yes. Yeah. It's... Yeah. Okay, so the question is like, how do you, like, do you sometimes like, get lost in a rabbit hole, or maybe also like, how do you choose your battles of like, which edge case you're going to, like... Yeah. This is... So I feel like for that, it helps to have like, an area of focus where you're like, this is like, the world of things that I'm interested in. So for example, like, I work on this thing, on this Kubernetes thing, and I work on trying to make sure that that system is like, really reliable. So if I run into like, a Kubernetes bug, I'll often, I'm more likely to be like, this is important. I'm going to investigate it. But if it's something like, outside of my like, box of things that I'm currently paying attention to, then I'm like, okay, we'll, we'll let that one go for now. Does that make sense? Or I don't know if that's helpful. Right. Yeah. Yeah. I feel like it's really hard when you're like, more of a generalist. Because I said, I feel like it's hard when you're being more of a generalist, and when you're like, touching a lot of things. Because it's like, it's impossible to learn about everything all at once. I think I'm just agreeing that this is hard. I have no wisdom. Right now I got to specialize a little bit, which is more, which makes it easier.