 Hello. Hi, welcome to the presentation on Netflix and FreeBSD. My name is Jonathan Looney. I am the manager of a software development group at Netflix. My group maintains and develops the operating system that runs on the content caches that serve Netflix's streaming video. And I'm going to talk to you a little bit about how we develop that software, and specifically how we use some open source software, and particularly FreeBSD, to deliver streaming video to happy consumers around the world. So to start with, I need to tell you about OpenConnect. OpenConnect is Netflix's CDN. OpenConnect is a global, efficient, and purpose-built CDN for distributing Netflix's content. It is global. It is large. It is very fun to work on the OpenConnect network. At our peak, we deliver over 100 terabits per second around the world. And that makes the engineering challenges very interesting. So 1% differences make a big deal. They're a big deal when you're dealing with these large scales. As I was preparing for the talk, and I was thinking about this, I was realizing that if we had a 1% change in the number of TCP segments, we retransmit. So for instance, if we were retransmitting 1% more of TCP segments on our network, that's an extra terabit per second at peak that we're sending out to the internet. And that's a lot of traffic. And that's a big change. And so we work in an environment where small changes are a big deal. And we also value small improvements. Obviously, we'd like big ones, but every little bit of efficiency matters. And so it's a really fun environment to work and do engineering work. The heart of our CDN is our content caches. You could almost say that the backbone of the Open Connect network, they are what delivers more than 100 terabits per second at peak. You see here a picture of one of our storage appliances from a few years ago. It's the 40 gigabit per second attached storage appliance. And it has 248 terabytes of storage and a 2RU form factor. We also have other varieties of caches. Our highest end production, one at the moment, delivers 100 gigabits per second attached. And it can deliver essentially full line rate in a 1RU form factor. The OCA runs almost exclusively open source software. And two of the more important things we run are the operating system is based on FreeBSD. It's a lightly modified version of that. And our web server is a slightly modified version of Nginx, or a slightly customized version of Nginx. This is our typical traffic profile. You'll see that we have lots of Netflix clients that are pulling data from our CDN. We have a group that manages the steering for that, so we can steer the Netflix clients to go to the right caches to serve their content. There are redundancies built into the system. And controlling the clients is very helpful, because it gives us the chance to tell them exactly how they should behave in various failure scenarios. So we can build redundancies into the system, such that if a single OCA fails, a single one of our content caches fails, the clients can seamlessly switch over to other content caches. And that means that if we are testing new software and that new software has some sort of fatal error, our clients can seamlessly switch over to something running more stable software and continue to be served, hopefully without even noticing any difference in the quality of their video. We also get a lot of data back from these clients. And so again, controlling the clients is very helpful, because the clients can tell us, or controlling these client applications is very helpful, because the client applications can tell us exactly what they think about the quality of the cache that they're talking to. They can tell us how fast they were able to get the video, whether they got it fast enough to satisfy the streaming needs of the client, whether it was very choppy or smooth, things like that. So we can get a lot of good information from our video applications. And that helps us to make informed decisions about whether we've made things better or worse in the software that we're developing. Our typical workload on one of our content caches looks like a typical web server. You've got a bunch of storage. You need to get data off the storage into memory. We then encrypt it. We often encrypt it for transmission over TLS session. And then we send it out. We send it out some NICs. Now this is not rocket science in a sense. This is something that many people have done before. And there are even people who have probably other people who have done it at the high speeds we are. What the secret sauce, if you will, or makes this special, is that we are trying to get high speeds, high efficiency, or high speeds out of our systems with high efficiency. So we want to not only deliver a great experience to our end users, we also want to control the costs. And so we actually work very hard to optimize our workload so that it can fit into relatively inexpensive parts that you could go buy off the street. There's almost nothing that we use that you couldn't go buy from someone if you wanted to. We don't have custom ASICs and things like that floating around. And so the secret sauce, if you will, is simply being able to get things through the system as fast as possible, as cheaply as possible. And you'll see that there's a couple of resource constraints we have. The first one is the disk I.O. and the network I.O. All has to go over PCIe. And so we're constrained to the number of PCI lanes we have. We're also constrained in the bandwidth those PCI lanes have. And then all the data coming in and out of memory. Again, we're constrained by the number of memory controllers we have, the number of memory channels, as well as the speed of the memory. So those are the two main resource constraints that we fight against. And you can either buy your way out of that by buying bigger CPUs, or you can try to make things more efficient by utilizing the resources you have more efficiently, more intelligently. And we try to focus on the latter. So using FreeBSD and Commodity Parts, we get 90 gigabits per second steady state out of our latest 100 gig storage appliance. Actually, I think all of our 100 gig storage appliances at this point. The latest one, we're using 55% of the CPU on a 16 core 2.6 gigahertz CPU. For those that are interested, it's an Intel Xeon 6122 CPU. There's 16 terabytes of NVMe in this appliance and 96 gigabytes of RAM in a 1RU form factor. The reason I'm quoting 90 gigabits per second number is because that's what we shoot for in production. We aim to use 90% of the attachments. The reason we aim for 90% is we do have a steering system that tries the appliance to the correct caches. And if you target 90%, you can do that pretty reliably. If you go above that, it gets a lot harder to do the control loop correctly. On top of that, we need some headroom in case there are failures in the system. So if a content cache fails, the clients need someplace else to go to. And so we need a little bit of spare capacity in the system. And the 10% gives us just a little bit to a little bit of headroom to absorb clients that are switching away. So we would consider 90 gigabits per second to be our target for this platform. And we can do that with roughly 55% CPU usage. And here's some graphs that show that. You'll see that the bandwidth is hovering right around 90 gigabits per second. And the CPU utilization is right around 55. And this is measured in an actual system serving production traffic. This is not a synthetic benchmark. This is what we get in production. So let's talk about the operating system that runs on these caches. It's a, as I said, it's a lightly modified version of FreeBSD. For those of you that are not familiar with FreeBSD, here's the 32nd introduction. AT&T had a UNIX distribution. They, the University of California, Berkeley got a license to use that distribution. They created what they called the Berkeley software distribution. And there's a lot that I'm skipping over. But essentially FreeBSD and NetBSD are both descendants of that code base. And the other, there's lots of other BSD distributions. I'm not trying to cover them exhaustively. But this is where FreeBSD came from. It's not a Linux distribution. It's not really related to Linux, other than the fact that Linux and UNIX look similar in a lot of ways, more or less, depending on who you ask, probably. The other thing is that FreeBSD is a complete software system as opposed to Linux, which is a kernel, and then the software distributions turn it into a full operating system. FreeBSD is a full operating system. So the FreeBSD release cycle looks probably like your typical software release cycle for an open source project. There is a head branch, which is the development branch. That's where new features and bug fixes go first. And then there are these release branches that come off of that every once in a while. And the release branch is called the stable branch. So for instance, there was a stable 11 branch about two years ago, I believe, which is the branch that all of the 11.x releases come off of. All of the releases that are part of the 11 major release, all the minor releases come off of that release branch. Likewise, there recently was a stable 12 branch that was created. Over time, of course, the code diverges, as you'd expect, with a software development model like this, and the new features go into head, and not all of them are backported, depending on just how stable they are. So a while ago, when I first joined Netflix, it was right around the time that we were running stable 10, which was the major release before stable 11. So our operating system was based on stable 10, and we had a choice when stable 11 was released. We could stay on stable 10 and get farther behind, or we could switch to stable 11 or head, and we decided to rebase our code onto stable 11 or head. And then we had the choice. Well, it basically stable 11 and head are the same thing at this point, because it's just branched. So which do we wanna do? Do we wanna go to the next stable release? Or do we want to track head? And we decided to track the development branch, and so we rebased onto the development branch, and we began to track that. And we did that for about two years, and then stable 12 came along, and we had a choice. Follow stable 12 or stay on head. And there was essentially no discussion or debate. Everyone loved tracking head so much that we've continued to do that. So it's worked really well for us. This is sort of what the code inheritance looks like for those of you that need this mental model or appreciate the mental model. So FreeBSD heads code bases at the top. We have our internal master branch, and about every five weeks we take a sync from head to our internal master branch. We develop some more code and then we branch off our release branches. And we shoot to have a release just about once every five weeks. Our typical release cycle is five weeks of development, followed by five weeks of testing. The five weeks of, so there's two releases in flight at any given time. One in development, one in testing and deployment. We start off with a free, emerge from FreeBSD head close to the start of the development process. Then we also, throughout the five weeks we're doing feature development and integrating our features into the master branch and with the other features that have been developed. And of course we're doing continual testing throughout the five weeks. At right at the five week mark we make our release branch and then we begin the process of doing further testing, further intensive testing on that release before we deploy it more widely. So at this stage we would be running this release on approximately, what we'd be using approximately 45 caches to test this release with a mixture of representative hardware from the different kinds of hardware that we support. Half of those caches would be running the old operating system, half the new operating system and we would do a very close comparison of the performance of the old operating system versus the new one. And so we look at obvious things like panics and crashes and so forth. But we also look at the metrics of how the system is performing. It has the CPU utilization gone up or down. Are we able to push out the end width as fast? Are we able to push out bits as fast? Have TCP retransmissions changed noticeably? Has power usage changed noticeably? So we look at all of the metrics you would probably think that we look at and maybe even a few more regarding the system itself. We also look at the metrics the clients give us. So remember the clients can tell us whether or not they like this new software. And if the clients tell us that their experience getting data from the cache running the new software is as good as the experience using it from the cache running the old software, that's a pretty good indication that we haven't messed anything up. And if their experience is even better, that's great. After we do our dev testing and we fix whatever bugs we find, it moves on to a canary testing. So we've deployed at this point to a much larger set of caches. So many more clients are hitting this and we can again do the comparison to see if things have gotten better or worse or stayed about the same and make sure that matches our expectations. And then finally we go through about two weeks of deployment. In an emergency, which does happen periodically. So for example, a security vulnerability or something like that, we can speed up this process very greatly. But if we have the time to do this sort of process, we much prefer to be slow and methodical than rushed. But if we need to, we can rush out fixes in a hurry. So one of the questions you might be asking is, well just what kind of features do you need to develop? And the features that we develop are basically aimed at one of two things. The first one is making our customer experience better or making the OCAs run more efficiently thereby lowering our costs. And we want to do one of those two things. Ideally we want to do both at the same time. But we try very, very, very hard not to sacrifice customer performance in order to save money. So we try to do both at the same time. And there are things we can do as software developers to make the customer experience better. So for example, getting bytes off the disk faster is something that can help the customer experience because their video starts playing that much faster. And there are things we can do as software developers that will make that better or worse. But here's some examples of some features that either we've developed internally or sponsored in the community. The first one is NUMA enhancements. The previous NUMAs had NUMA support for many years but it hasn't worked for our particular workflow very well when we've tried it. So, or at least it did several years ago. So we've explored adding, or we've been adding some enhancements to it to make it work better for everyone but also particularly for our workflow. The next two asynchronous send file and kernel TLS are things that enable us to get higher speed, higher efficiency through the box by avoiding a bunch of context which is in copying data which in kernel and user space. I think other people are probably, some other people are probably using DPDK in this space. We aim to keep as much in the kernel as we can. And so kernel TLS and asynchronous send file are ways that you can, the user space can tell the kernel, please do this for me. And then the kernel takes care of it for the user space. A string for the send file, you tell the kernel, please send these bytes from this file over the socket and it will just do that. And it'll only really tell you if there's a problem. The kernel TLS, you give the kernel the right encryption keys then the kernel will go do the TLS encryption for you. And so you combine the two together and with one system call you can send a large chunk of a file to a user without the user space having to make further system calls or do more copies. The next two were Pbuff allocation enhancements and unmapped mBuffs. Those are both ways that we gain efficiency by taking bottlenecks out of the system. The Pbuff allocations, Pbuffs are the physical buffers that handle data or store data when it's coming off of the disk. So they would be buffers for disk operations. There was contention for a lock which was making the performance very bottlenecked. And so we did some work to rewrite that and be able to get more parallelism during the Pbuff allocation process. Unmapped mBuffs, again it was about taking bottlenecks out of the system. Someone on my team managed to find a way that we could take the same amount of data and instead of representing it in I think it's like 64 network buffers we can now represent it in one network buffer. IO scheduling, again we can really make a significant impact on users if we get data off the disk at the right time. It actually really matters when you read and write data. You don't want to read it too early, you don't want to write it too early, you don't want to read it or write it too late either. So it really matters and we've done some work to make sure that we are doing IO at the right time and doing the right amount of IO and so forth. And likewise the same thing applies to trims and we've also been doing some work to make sure we send down the right trims at the right time and also at the right pace because sometimes disk drives don't behave well when you send down a whole bunch of trims all at once and so we've done some work on scheduling those as well. And finally we do work in TCP regularly so there's a whole group that works just on TCP algorithms and so forth. We've done a lot of work in both research and development in algorithms to get TCP to transmit data better where better would mean again sending the right bits at the right time for that session. And we've also added some logging and debugging infrastructure so we can tell just what's going on when things don't work the way we think they should. And that's been very useful for us in developing these things and we've shared all of this with the, I think we've shared all of this with the community at this point. So I think that just about all of this is either upstream or is in the process of being upstreamed in some way. Tracking head makes it, lets us stay forward looking and stay focused on innovation. The problem that you can run into when you are tracking the stable branches is that you continually have to deal with the problem of my code doesn't match what everyone else is developing on upstream. And so when there are bug fixes that occur or new features that you want you have to figure out how to get them back into the code that you're actually running. Likewise, when you get the new merges from upstream sometimes you have to redevelop your features because the APIs have changed or data structures have changed and all of a sudden the code you had doesn't work anymore. But by tracking head we're able to stay very close to what's going on in the upstream development community and instead of continually redeveloping the stuff we've already done we can just work on developing new things that will benefit us and hopefully benefit other free VST users as well. My hypothesis is that downstream users of open source projects can get stuck in one of two cycles which I've labeled vicious and virtuous. And this is not entirely original thought but I think you can easily get stuck in this vicious cycle, what I call the vicious cycle which starts with infrequent merges. And so if you're a downstream user and I don't mean end user in this case, I mean a company like us that's trying to write an operating system or either write an operating system and use an upstream operating system and customize it for uses or use an upstream application and customize it for uses, that's the kind of user I'm talking about. And I think this cycle starts with infrequent merges. So when you start with infrequent merges and this is infrequent merges of the development code. So this could mean that you are tracking a stable branch and or a release branch and then every year or two you jump to the next release branch. That's in essence, that's an infrequent merge of the development code. And when you start off with that, what happens over time is you get farther and farther away from the upstream project. So when you do actually try to do a merge, you have a lot of conflicts. You have regressions, things don't work right. And it becomes very painful. And so every time you do one of those merges, it's very painful and disruptive to the organization. The developers who were developing code suddenly have to stop developing new features and go back and make all the old stuff work again with the new code. And that's not very fun to do as a developer, but it also means that's a whole bunch of time you're not writing new features. And all that means is that you are, you end up thinking these merges are so painful that you're going to do them even less frequently. And when you do them less frequently, you have even more merge conflicts. And you have more regressions, which means that you have to spend even more time fixing them all, which means your feature velocity gets even slower. And the whole thing is so painful that you want to do the merges even less. And at some point, maybe you even stop taking merges from upstream, but instead you just simply try to cherry pick back the bug fixes that you think you need. So you can compare that to what I call the virtuous cycle, which starts with frequent merges of the development code. So when you're frequently merging the development code, you're diverging from it less over time. And that means that each merge is much easier. There'll probably still be conflicts, there'll probably still be regressions, but there should be fewer of them. And each merge should be much easier to handle. Furthermore, because you're so close to the development branch, it's easier to get things upstreamed. It takes time, upstreaming things is not just about pushing your code into the upstream project. Rather to upstream things, first of all, you need to generalize the feature and make it generally applicable to all the users of the project. On top of that, you probably need to do some code cleanup because there's probably some rough edges. And then if you are, and that's just if you're running the development branch, if your code is based off of a stable branch, you need to do the extra step then rebasing your code onto the development branch. By running the development branch internally, we're able to skip that last step. So we still need to clean up our code, we still need to generalize it. And there's some that costs extra money for us, but we're willing to do it because we think it's the right thing to do. And all of that means that we can get our code upstreamed that much easier because we don't have to rebase it regularly onto the development branch to get it upstreamed. And having that code upstreamed means that our diff against the development branch of the upstream project is relatively small. That lets us focus more on getting our developing new features. So we're spending less and less time dealing with old features, we're spending more time dealing with new features. And on top of that, it lets us collaborate effectively with other people in the upstream community who are using this project. Because we're, most FreeBSD developers run FreeBSD head and because we also use head internally, we are able to directly share patches with them and they can apply the patches, they apply cleanly, they get the NUST patches, they apply cleanly, and so we can collaborate together when we're doing our work. And all that means is that it makes it easy for us to do more frequent merges. In fact, we've gotten to the point where my developers will sometimes just commit stuff upstream to the upstream project rather than to our repository and we'll just pull it back in the next upstream sync that we do. Now having said that, we do keep local diffs for a few reasons. The first one should be fairly obvious, it's information covered by NDA. We do occasionally get information from vendors, which helps us to do something we need to do. But it's covered under an NDA and they will not let us upstream the code based on that. I think we have something like 100 lines of code that fall in this category. So it's a very, very, very small diff. But this does happen occasionally. We also have features which are still in development or testing. We will sometimes commit features to our tree for testing purposes and we don't upstream them usually until we're sure they actually work. And so we'll keep that local diff until we're sure the feature works and we've had the chance to generalize it. And that's the third reason we keep local diffs. There are features we have which are too specific for us. They're things that help us with our workload but they're not general enough. We have a very specific workload. As I showed you, our whole business is getting data off of disks into memory, encrypting it in memory, sending it out on a NIC. That's what we do. And there are other people that use computers for other things. And what is good for us might not be good for them. On top of that, we have a rather homogenous environment. All of our caches are Intel 64-bit. And there are people that run other architectures. And again, what's good for us in Intel 64-bit might not be good for ARM 64-bit or ARM 32-bit or RISC processors and so forth. So there are features that need to be generalized and we will hang on to those internally until we've had the chance to generalize them. And once we have generalized them and made sure that they work for everyone or at least don't make things worse for anyone else, then we will go through the process of upstreaming them. But it's our intention to upstream any code that we can. I think that this is one of the great things about working at Netflix. I've worked at other companies that used open source software as the basis for their product. And they kept everything inside. So they had lots of local patches that it was very hard to share outside of the company. Netflix does allow and in fact encourage the use of open source software or it makes sense, but they also let us freely contribute our software back. And so the approval process to upstream a feature is essentially, it's automatically approved. And that's really nice. So we can get our software upstreamed rather easily from the Netflix perspective. And we're willing to pay our developers or give them some amount of time to go through the process of upstreaming those, cleaning up the code, generalizing it, making it work for the broader community because we think there's value in upstreaming what we've done. So we get a couple of benefits from tracking FreeBSD head. The first one is we get quicker feature iteration. As I mentioned, we're not, our developers aren't constantly stuck looking backwards, but rather we can, they can develop looking forwards and they can rapidly develop new features knowing that they're going to be able to get them, they're going to develop them once for us in our code base. And then that's the same code base they'll be upstreaming them into. And so they don't have to think about what's going to work in stable 10 and then what's going to work in head and sort of develop the features twice. They can just develop them once and just worry about the one environment. We also get quicker access to new FreeBSD features. And this is probably the right point to mention that we're very happy running FreeBSD. We get great performance out of FreeBSD. It's very stable. And we get access to lots of really cool features that people in the FreeBSD community are working on. And so it's a great benefit to us that we run, we track heads so that we can get access to the new features as soon as they're committed. And so someone can commit a new feature today and it can be running in our network somewhere between five and 15 weeks from now, typically. So we like the fact that we get quick access to new FreeBSD features and we can do cool things by getting access to those features. We also get quicker bug fixes. And this isn't just because the code gets committed first to head, that's true. So FreeBSD developers, they're fixing a bug, they'll put it in a FreeBSD head. And then once they're sure it works and it's had some soak time, they'll back port it to older releases. So we get access to the bug fixes a couple of weeks earlier, generally running FreeBSD head. But on top of that, we also get quicker bug fixes because our bug reports come much closer to when the code's committed. So let's say you're an upstream developer in the FreeBSD project and you commit code today. It may be two years before that ends up in a FreeBSD release used by the general community. Once that happens, two years from now, someone may find a bug and report it to you. Are you gonna remember why you wrote that code two years ago? Contrast that with what happens when we find bugs. You commit the code today. In the next one to five weeks, we get the code in a sync. We find a bug and we report it. And we report a bug to you on code you just wrote a few weeks ago. You're much more likely to have the context in your head and it's likely that you'll remember why you wrote the code the way you did and you can quickly help us troubleshoot and fix the bugs. On top of that, they're easier to fix generally because there's not two years of other stuff built up that depends on that bug being there. That sometimes happens over time. People, they rely on a bug being there because it's been there for two years. They think that's just the way things are supposed to work. And you try to fix a bug and you end up having to untangle this rat's nest of dependencies that it built up by finding the bugs sooner and reporting the bugs sooner. We're able to generally get fairly quick fixes and before lots of other dependent stuff has built up that makes it harder to fix. It also enables collaboration. As I mentioned, most freebies-d developers run freebies-d head and so we're able to directly share patches with them and that's good. We really like the fact we can collaborate with other people in the community. And I wanna be clear, it's not a one-way collaboration where we're trying to figure out what we can get from other people. We really do like collaborating with people and being able to share the patches we have, the things we're thinking about and also find out what they're thinking about in ways that we can work on projects together, perhaps. It also minimizes our merge conflicts. As I mentioned, sometimes our developers will even just commit code directly to the freebies-d project and we'll pull it back as part of the DexSync. In other cases, it may take us a little bit of time to upstream things, but we can upstream them much more easily because we don't have to go through that step of rebasing onto freebies-d head first. And so that really helps us keep our local def low and minimizes the merge conflicts that we have. It also amortizes the merge cost. So if we did, once every two years, we switched, rebased our code on to the latest release, that would be a very painful, long, difficult merge. Instead, we spread it out and we do it once every five weeks or so, we do a little bit of that merge. And so it's in much more manageable chunks. The diff is much smaller, it's much easier to spot bugs. If we absolutely needed to, it's even possible to go do a binary search through the code that's been committed to figure out where a regression was introduced. So we've got much more manageable diffs, it's much easier to deal with. But overall, I would say we probably spent about as much time doing our, in aggregate, we spent probably spent about as much time doing our once every five week merge, as we would on the big merge, at the end of the two years. But I think that spreading it out is much more pleasant for everyone. Freebies-d also, I think, gets some benefits from what we do. The first one is that they get wide deployment of their head branch very early. So their head branch is, usually a commit on the head branch is deployed in our network somewhere between five and 15 weeks after it's committed to the Freebies-d head. And I think it's a testament to how stable the Freebies-d head code is that we can do this. It's a very stable, very usable release. And we're able to successfully deploy that code very quickly on a large number of caches. And they get the benefit of having that early deployment and that early wide deployment of their code so that we can find bugs. Now, obviously we try to find them before it's deployed that widely, but just in case they slip through, their code is still seeing use on a large number of content caches around the world. They also get early intensive testing of that code. We test the code thoroughly before we deploy it and, again, it gets tested in production. And we also have an incentive to upstream our code. And so we, because we wanna minimize our diffs, we have a great incentive to upstream our code and it's easier for us to do that. And so I think more of our features end up in the upstream code base than might otherwise. And I think that's good for the FreeBSD community as a whole. Again, we're not trying to just get our code into upstream FreeBSD, but we want to benefit the upstream project by giving them access to the features we've developed internally. And we think that that's good for everyone. So you might think of some objections to running development code. So the first one is, it isn't stable, right? Well, actually it really is. The FreeBSD community is fanatical about keeping head stable. And in fact, it may even border on unhealthy. It's, they're that fanatical about keeping head stable. Which is great, I think is great. It means that you can download and use FreeBSD head and it generally just works really well. Why should you pay to find the bugs that others are gonna find while they're testing head? Well, the obvious problem here is if everyone has this mindset, who's actually gonna test head? Who's actually gonna test the code? And someone has to do that testing and I think that we're well positioned to do that testing for, at least for our use case. On top of that, we would probably still find some of these bugs anyway, two years down the road when we get the code. But it would be much harder for people to fix because they would have lost all that context about why they committed that code. Aren't there more security bugs in development code? Well, actually no, not in our experience. I think almost all, if not all of the security bugs that we've run across in the past two years have, most of them, have been in the stable branches as well. And so on the whole, we probably have the same security bugs in general that the stable branches have. But the people developing the fixes for those security bugs are developing them on the head branch. And we get access to those security fixes that they've developed for the head branch. No one does this. Well, that's not completely true. There are other companies that do similar things. There's one storage vendor that announced a couple of years ago that they were going to try to base their storage appliance operating system on FreeBSD head. You pay a monthly cost to do merges. That's true, we do, but as I said, in aggregate, I think that it's no more than if we waited and did this once every two years. And I think each one is actually much easier and much more pleasant for everyone involved. And you get new bugs each month. Yes, that's true. We do get new bugs each month. We also get new features. And the new features are really cool. And guess what? If we were running the stable branches and we did emerge once every two years, we'd get new bugs then, too. So, but we wouldn't be getting new features during that two-year time. So I think that on the whole, it's true, you get new bugs each month, but it's manageable. It's at a manageable pace. They're a little bit easier to debug and fix. And it's counterbalanced by the new features we get, as well. So in conclusion, running FreeBSD Head has been really, really good for Netflix. It lets us deliver very large amounts of data throughout the internet very efficiently while maintaining a high degree of feature velocity. We've found FreeBSD Head to be very stable. It's stable enough that we deploy their commits in our network, usually, as I said, about five to 15 weeks after they're committed. It lets us deliver a large amount of traffic very efficiently, and it lets us maintain our development agenda. So we're very happy using FreeBSD and, in fact, using their head branch as the basis for our operating system. It looks like I have about five minutes if anyone has questions they want to ask. Are there any questions people have? Raise your hands high. All right. I see one right there. Is that it again? Hello. How much of the code is not ported back to FreeBSD? How much is not? Like which features, for example. What's that? Which features, for example. Which features are not ported back? As I said, our aim is to port back all the features, except for the ones covered under an NDA. At this point, there are a few features, particularly in the TCP area, which are not yet upstreamed. But we're in the process of working through our backlog and upstreaming them as we can. And we're also usually happy to share diffs, if we can. And so when people write us and ask for diffs, we're happy to share them. And we give them the caveats of, this isn't quite ready yet. Any other questions? Why did you choose FreeBSD kernel instead of Linux kernel? Can you repeat the question, please? Why did you choose FreeBSD instead of Linux? Why FreeBSD instead of Linux? OK. Originally, from what I understood, I wasn't there when the decision was made. Originally, from what I understand, it was all based on the license. So it was based on the BSD license versus GPL. At this point, we stick with it because it has a combination of a very positive, collaborative development community, a good set of features that are at a nice, stable velocity, and good performance for us. We have tested Linux in the past several years, and it didn't give us as good performance as FreeBSD. But of course, it's not exactly a fair comparison because we didn't do all the engineering work on Linux that we've done in FreeBSD. So it's not exactly a fair comparison. But we clearly would have to do a little bit of engineering work to be able to switch to Linux at this point if we wanted to. But as I understand it, the original reason was the license. But we're very happy with FreeBSD. And I think we get a lot of benefits from using it at this point. Any other questions? Yeah, go ahead. How many developers are focused on this in Netflix? How many developers? How many in Netflix for this topic? How many developers is in Netflix for this topic? How many developers work on the operating system? So my team has six or eight people, depending on how you count. And the team that works on the TCP protocols in that, I think, has four people on it. And there's a couple of other people that also assist us. So all told, I would say probably 16 or so. So it's a nice, small, manageable team. I wanted to ask, which file system of FreeBSD do you use? Is it UFS or ZFS? We use UFS. And are you using SSDs or just hard drives? We use NVMe, SSD, and hard drives. We use them in different systems for different reasons. We have some systems that are meant to store large amounts of content, but serve it less efficiently. And those have hard drives in them. We also have our 100 gig appliances at the other end of that. We have 100 gig appliances, which are meant to serve a much smaller selection of data, but serve it very fast. And they would be NVMe or SSD. For your development testing and your canary deployments, what is the percentage of your user base you deploy to? For the canary testing, you mean? Yeah, so for the, we do several stages of testing. The first one is to about 45 caches or so. The next stage is to a larger number. I'm not sure exactly what it is. I'm sorry. But it's, yeah, I'm not sure what the number is. I'm sorry. It's another group that does that. At that point, we've handed it off to another group, and they do that testing. So I'm not sure. I don't know that answer. I'm sorry. OK. Thanks again, Jonathan. No problem.