 Hello, and welcome to EuroBSDCon 2021's Building and Running a BSD Home Lab session. So I guess we get started with introductions. Hi, I'm Alan Jude. I'm a CTO at Clara, and I'm a free BSD source and doc committer. I also do a number of podcasts, including BSD Now, one eighth of the work, apparently. Tom and I figured out earlier. And wrote the ZFS and Advanced ZFS books with Michael W. Lucas. So Michael, you want to introduce yourself? Hi, I'm Michael Lucas. I write books, some of them about you people. Okay, I guess I'll go. I'm Kyle Niesel, known as Dr. KK on some of the forums, and I come to free BSD by way of free NAS. Tom, do you feel you had to unmute yourself? Just below the slides, there's a little mute button. It seems like you did. There's a lot of latency in the big blue button now. Yeah, so I'm Tom Jones. I've been joining this panel 48 seconds ago, and I build research to do internet experiments and have it appear as a home. So talk a little bit about what a home lab is and why you might want one. If you don't already have one, but it's oftentimes just a place for you to experiment and learn and build things. Oftentimes, it's kind of like the dev environment that everybody has at work. And then there are those lucky few who also have a production environment. But it gives you something like that at home, and it lets you try things and play with things and so on. And some people also use them for development, like I do, or testing, or just exploration and learning. Oftentimes, a home lab kind of grows organically. You don't start out planning to build a home lab so much as you repurpose an old machine as a home server, and then there's another one, and then you add this and that. That experiment you're running turns into something you use frequently. And then, all of a sudden, you have this big home lab, and you're doing all these interesting things at home with what started out as just one spare machine. And a home lab can range from something big and rack mounted like I have some pictures later, or it could just be one spare machine that sits on the floor by your desk like what Kyle has. So here's a quick tour of Kyle's home lab. Kyle, do you want to walk us through this quick? I think it's mostly self-explanatory, but years ago, I started a free BSD home lab in that black box on the right. Today, as Alan was saying, today that has sort of become my backup for what is now my primary home lab server, which is the white box on the left. And so, you know, this is distributed throughout the house, of course. I have the office, but that black box is in a completely different room entirely. So this is one of the organic natures of the home lab, especially if you're trying to avoid problems with your wife. So in any case, that's kind of a tour of my stuff. It's as simple as that. I don't own a rack at all, and all of my three or four BSD systems are just like this scattered around. Back to you Alan. Yeah, that's a topic we could touch on at some point is that sometimes the other people you live with don't want wires running everywhere, and, you know, things kind of cobble together, and it has to look nice. So there are quite a few different factors that can factor into the design of your home lab. You know, one of the things we talked about later is noise as well. So we kind of repurpose a dual purpose slide. So here's a bit of a picture of my home lab. This is as much of the rack as I could fit in the frame because it's so close to the wall, I couldn't get any further away from it. But I have a bunch of large rack mounted servers. There's about 750 terabytes worth of disks in the rack and four dual processor and then four more single processor servers. And it uses about 1700 watts and is backed by a six kilovoltamp UPS. And then I have this amusing graphic from my electric company, which shows kind of the bell curve of what everybody else in the neighborhood uses the 646 other accounts in my postal code. And then you can see this kind of outlier way off to the side with the red arrow that that's my house. So I'm wondering if the people over here maybe also have a home lab. And you know, I think it's probably worth pointing out since it is Euro, BSD, come that this is very good North American pricing. You guys would be in especially in Germany, I think in much worse shape than Alan, if you were running that. Yes. And you know, the 1700 watts for everything that runs through the UPS in the rack doesn't account for the cooling. There is some cooling involved in, you know, one of the ways that this home lab is able to work at my house is that it's in a room in the basement that has a door. And when you close that door, you don't hear all the servers. And there's also an air conditioner in that room that keeps the servers from overheating that small basement bedroom. And yes, like Patrick in the chatroom says, you always pay twice for energy after you reach a certain threshold, because not only do you have to use the energy to make the computing, but then you have to deal with the heat that's left over from it. I saw someone raised their hand with a question. Please just type the question in the chatroom and we will answer it. So talking a bit about the hardware you might select for your home lab, it doesn't have to be servers. My home lab kind of started out as a pair of Lenovo X220s I bought on eBay for about $100 each. And it was just I need these separate machines so that I can test, you know, booting with an encrypted disk or working on the installer. And so just I'm just going to build this USB stick with the installer on it and I'm going to run it many times and do bad things with it and, you know, mount it, rewrite and modify the script and keep working on it. And it just I needed it to be a separate machine or, you know, for some of the bootloader stuff, having a physical machine worked a bit better than a VM because I managed to make virtual box crash itself when trying to run my bad code. But then when it came time to actually get bigger servers, I went to UnixSurplus.com and they have a lot of used Super Micro and other rack mount servers that you can get at good prices. They also list their stuff on eBay frequently with a or best offer and they will sometimes accept offers less than the list price. But if you are going that route, I would avoid any of the Super Micros of the X7 series or at this point probably even the X8 series. The older hardware uses a lot more power and it creates a lot more heat than the newer stuff. And the other thing to be aware with when you're buying rack mounted machines is that they tend to be very noisy. They have very loud fans, although I think as we talked about on another slide, you can replace the fans with quieter ones and it will reduce the noise significantly. But the other tip, kind of like I had here is that the noise generated by these servers is mostly high frequency noise, meaning it doesn't penetrate things very well. So sometimes something is as simple as a curtain or a sheet or just, you know, a thin, it's not even a wooden door. It's practically, the door is made out of cardboard basically. And when you close it, most of the noise just doesn't get through. And, you know, you can sit on the other side of the wall and not even know that there's all these machines howling away in the next room over. So Morgan asked, is there some kind of hardware compatibility list for cheap and BSD compatible parts? In general, most server grade stuff should just work. You know, I've never really run into compatibility problems with the super micro hardware that I bought. As you can see, my racks are full of it. And it's oftentimes what you run into if you're renting machines from dedicated server providers and so on as well. There are lists on as part of the previous two release notes, although they're sometimes hard to comb through. And yes, as Devil points out, most of the time the man page for the driver has the specific parts it supports, although you have to be at least somewhat familiar with this stuff to know what driver is going to end up being to know which man page to check to find out. You know, for example, the LSI disc controllers are in the MPS and MPR man pages. And, you know, you can easily find out which one supports which devices, but you'd have to know that. Most of my home lab gear is old laptops that I've worn out one way or another. And generally, any BSD I try just works. I have them stacked up in the basement and I plug monitor and keyboard into them. The network works, disk works. And that's really all that matters for what I'm doing. Yeah, oftentimes the advantage that your home lab has is that it's made up of used hardware, which means it's had lots of time to get support in all the BSDs. So, outside of a couple of, you know, specifically known parts that just never have any support on any BSD, the old stuff tends to have the highest chance of working. Right. Until you get too old. I think, I mean, check me if I'm wrong on this, but I think you'd be very unlikely with any kind of reasonable used hardware that you would try nowadays, with the possible exception of some of the HBAs and things like that. But in terms of the NICS and, you know, the other elements of the main hardware platform, I think it would be unlikely that you could even run into any trouble vis-à-vis BSD with that. Isn't that correct, Alan? I feel that way as well. You know, Wi-Fi is the only thing that you are likely to have trouble with. An older Wi-Fi tends to, while not being as fast or whatever, do actually be supported, but, you know, your home lab is usually involving some ethernet cables anyway. I could say tend to be more fixed installations or whatever. And so, yeah, I generally don't have many concerns. But again, the advantage of used hardware is you can do some Google searching and often find people who ran BSD on whatever it is you're looking at buying and even be sure of it. Like using the New York City BSD user group's D-message servers. You can so maybe even find the logs of other people having booted BSD on the hardware you're targeting. Especially, again, when you're looking at server-grade hardware, there tend to be not quite so many models and things are kept more specific than laptops because, you know, replacement parts are a long-term thing for servers versus laptops and so on. And so, I expect you to have fewer problems there. Alan, there's a question from DSL, which I think is a good one about non-X86 platforms and how that might fit into a home lab. And I know you know something about that. Yeah, there's a couple of different options. It depends what your goal is. The one we talked about, I think at my user group last or earlier this week, was the Honeycomb. Solid Run makes this ARM Dev workstation called the Honeycomb. It's an ARM64 mini ITX board with like 16 ARM A72 cores at 2 gigahertz, an ADEX PCIe slot for a disk controller or something, but it also has four onboard SATA ports so that you can power your disk that way and supports up to 64 gigs of RAM and even has four 10 gig ethernet ports. So you can do some pretty high-end stuff with it. Of course, that's, you know, a brand new board more designed for developers than necessarily sysadmins or whatever, you know, for a home lab. It runs about $1,000 US, but if you wanted, you know, basically your whole home lab as a server and not x86, you know, that's a lot of machine for that price. But you can certainly use smaller things. Lots of people building small NASs with things like the Rock Pro and the Beaglebone Black and small devices like that. Although, if it's going to be a NAS or something, you probably want something that has real SATA rather than trying to use USB or something to connect to your disks. But the Honeycomb is what was recommended to me when I was asking more along the lines of is there something modern you can get like that? There was a desktop workstation that was ARM. I'm trying to remember what it was called. It was a couple of years ago, and it doesn't really exist anymore. And it was, you know, is there some kind of replacement for that? And this is what was recommended to me. But I don't have that much experience with small non x86 stuff. I can tell you as someone that I see it on Torrent, the all of the free BSD distros, including the ARM stuff for all those architectures. And I've been noticing a real uptick in the past couple of months and the number of people getting those. I think we're going to get more of these questions moving forward. Yep. So talking about some lessons learned from having a homelab, you know, we already kind of mentioned the special approval factor or just, you know, it has to not be messy and have tripping hazards and so on. And obviously, noise can be an issue. But something that I ran into that I didn't really expect as my homelab grew was that suddenly I needed a production site for my lab as well. Because little things I had stood up and started using for things suddenly became production and other people would complain if they were down. One of the VMs that I happened to run on my dev box, which because it's being a dev box often gets rebooted, that VM is now important to other people and it has to keep working. So whoops. Or, you know, the biggest machine in my house was running a GitLab instance for some internal projects. But that means that that has to be kept up most of the time. Or then it got really bad with Plex, the video streaming app that we use to access our media archive on our TV. And it has a sharing feature and that means that my sister uses it. And which means her kids use it. And then now my parents use it. And so suddenly if my Plex is offline, I'm getting text messages from my family being like, you know, why is our TV not working anymore? What'd you do? And so suddenly it becomes a production service. And, you know, I have this subset of things in my lab that, you know, that needs to run on a box that's not going to reboot, you know, six times every weekend as I'm screwing around with things. And, you know, it can be, maybe some of those things shouldn't be in the lab and they should run somewhere else. Or just, you know, being cognizant of the fact that, you know, other people are using some of these services you're running now and you have to keep that in mind when you're suddenly your lab needs scheduled maintenance. And you probably want to avoid that because it really limits the usefulness of your lab if you can't work on that project because you need to reboot the thing, but other people are using it right now. Yeah, no, I couldn't agree more. I can't tell you how many times I've had to set the alarm for four in the morning, because that's the only time I can safely do some of the upgrades and changes to the configuration on the homelab stuff without creating a massive outcry here. So that's absolutely correct that you do have this creep from homelab into production, especially if you start running services like you're talking about. That's true. And it's very easy to think you can take advantage of systems you have in the house that are production to squeeze them into just a little bit of homelab roll. I have a free NAS box, a free NAS mini. And it would be very easy to just deploy some jails on it for this small thing that I need on for testing. And in theory, nothing would go wrong. I could just destroy the jails. I wouldn't have to reboot that critical host. But especially these days now that people are working from home, the home production environment is very vital. So you've got to keep that separation. Yeah, and this tends to be how your homelab keeps getting bigger is because now you need another machine to either put all the production stuff on or to actually be able to do your not production lab stuff on. And then even here, I noticed the other day I said, well, you know, some of my lab is a bit cooperative. I let other people use it for stuff as well. And so it's like, right, that machine is currently being used by somebody to work on this research. And I'm using that machine to run a demo of this. And I don't have a machine to do the work I want to do. And it's like, oh, you need another one or, you know, one of my machines in my lab is full of SSDs for doing a bunch of performance testing on ZFS and so on. And it's like, well, you know, we're also using it for this other day. It's like, ah, I want that machine for this. And it's like, I'm going to need to like make a schedule of who uses the machine when kind of thing, a sign up sheet or something to try to manage the resources of what work should happen on what machine. And it's suddenly getting very complicated. So, you know, sometimes you can get away with, I'll just have one big machine, and we'll run lots of VMs on it. But sometimes you're actually better off with two smaller machines so that you can kind of separate the production in the lab side of things. Oh, here's a question about why not just put this all under a hypervisor or a beehive or something like that. I don't know that I have a lot of thoughts about that that are cogent anyway. So I'm going to let you two answer that, I think. Does Michael have any thoughts? Well, for me, often the answer turns out to be lazy. If I need something with a network, I find it simpler and less error prone to just swap a couple cables. In theory, all of this could be done under beehive or VMM. If you want something complicated, then you're more stuck with beehive because the net graph. But my brain still understands cables. Perhaps your kids are better off. So for me, I run a lot of stuff on beehive in a data center machine. But I find that the small army of dedicated machines really helps for testing performance critical stuff and testing timing critical stuff. Because the big problem with virtual machines is time doesn't really exist inside a virtual machine. And so if you need to do any network emulation, which I do a lot of, you have to have physical hardware. And it's really difficult to get good results if you cannot just pull together a bunch of machines. Right. And the machines don't necessarily have to be very big. They just you need a bunch of them. No. And so my home lab last year that I was very happy to share pictures of on Twitter because of how horrific it was was for APU two boards and a Raspberry Pi, which was a head node, which acted as a NAT gateway. Because I didn't care about performance, the outside world, I just needed the physical hosts inside the network, inside the testbed to speak to each other. And so you don't need big machines. But when you need machines, virtual machines just don't work as a good substitute. Yeah. The other thing I was going to mention is if you do need to virtualize a network and you don't necessarily have a bunch of machines and some switches and some cables or whatever, Tom wrote a great article on virtualizing the network stack in FreeBSD using the VNetJails to actually be able to create a complex network inside a single machine. But like Lucas was saying, sometimes being able to actually see the cables and be sure that's plugged into that. And when I unplug it, I can see that it's unplugged and so on. It can be easier to think about how it's connected and make sure that it's not working. Is it because I did something wrong or is it not configured the way I think it's configured? Whereas you can see the cable going from that machine to that machine. You know it's either connected or it's not. Not a bunch of software that could be confusing you. But like Jen pointed out, having a hypervisor where you can snapshot things and undo things easily can be helpful. Although my stack of machines mostly run ZFS. Although a couple of the machines usually more on the lines of VMs that I do ZFS development on use UFS because it means I can then unload and reload the ZFS module without having to reboot. Meaning that I only reboot when I crash it. Which is usually when UFS makes me regret using it. So another thing that came up, a trend I noticed based on questions coming into the podcast and so on is that the pandemic kind of changed a lot of people's home networks. Suddenly at home wasn't just for media consumption or whatever. People were spending more time at home and having more people at home needing more internet. Especially when kids were attending school via video conferencing tools like this as well. It meant that now you have three or four people trying to use video conferencing at once in the house. And maybe one of them is also trying to watch Netflix or something. And you know it became we need something better than just the off the shelf router that came from our ISP or was you know that we bought at the big box store. We needed you know a BSD powered router of some kind that we could configure to say you know prioritize this traffic or you know make sure that one user isn't it's in hogging all of the internet. And so that led to some very interesting changes to people's networks and home labs. Suddenly you know maybe you had Nat Gateway or something that's put in your home lab but you didn't write everything through it. But suddenly you had to or you know the machine you were using had to get bigger just because the amount of stuff going through it went up. Yes your home production network. Another thing I think we'll spend a few minutes talking about is just you know when you are building a home lab and you're picking your network components it can be a bit daunting to decide what to do or you know find something that has the features you want to do interesting things in your home lab without breaking the bank. You know I spent a long time trying to find a switch that could do 10 gigabit between you know my NAS and my other machine to be able to do the the SSH benchmarking I did a couple years ago and so on without causing me too many problems. The one I bought originally was this net gear thing with like eight ports but it could only be configured with an adobe air app or something and all kinds of other terribleness and so I give it to Lucas when I managed to finally get something better which at the time was that the ubiquity edge switch 16 which I have the model number and stuff on the slide here for which give us 12 svp plus 10 gig ports and then four 10 gig cup reports which allowed me to network a bunch of the stuff and the reason I wanted the cup reports is the machine I'm sitting at right now is upstairs on the other end of the house from my basement data center and the cables in the wall happens to be cat six and so I can actually connect this machine to the NAS at 10 gig but it would have been very difficult to do a fiber run for that but there was copper ethernet there so I was able to use that but that switch is not loud but it's not quiet it's not the kind of thing if your homelab was everything in one room and you know I don't know if you see over here I have a stack of laptops for working stuff but if my switch had to sit here it would be too noisy and annoying to have that one you'd have to you know swap up the fan or somehow make it less noisy and you know $700 is a lot if you're mostly just doing it to learn and not doing higher end stuff and so I recommended Microtech makes a little switch that gives you four 10 gig ports and one ethernet port and it's only about $150 which you know if you just need to connect two or three machines together is is a much more attainable thing and then for the network card side of it I have bought a few too many of the Melanox connect x2's on ebay they're like usually under $25 I even got lucky and got one set of them that came with the DAC cables included in the price which made it very nice to get you know all the machines in that rack you saw pictured earlier connected to each other via the switch and see I just run Alan's cast-offs no matter how annoying the stupid app is well as long as you don't need to configure a bunch of VLANs or reconfigure the VLANs very often it's not really a problem I suppose as I said no matter how annoying the app is and a user in the chat points out that Microtech also has a small switch that's eight one gig ports with two 10 gig uplink ports which can be enough if you just want to link you know a 10 gig out of your NAS to the eight other machines in your house or to one other machine that also needs 10 gig kind of thing that is definitely an option as well yeah if you're if you're just getting into 10 gig switching and routing it's going to be very hard to even remotely approach Microtech's bank per dollar ratio so I think that is definitely a good place to look and a number of offerings for routers and switches under a hundred dollars that Microtech have that really do really do seem like you know four and five hundred dollar class equipment for your network as well so I would personally encourage everyone to really look at Microtech it's the kind of equipment that will be I think agreeable to people who are interested in something like BSD in the first place so that's a that's a real good real good place to start looking for some of this equipment yeah so Kyle's put the model numbers of two of the very nicely priced Microtech routers in the notes here as well and you see the second one there even has two and a half gig so if you already have cat five in the walls of your house one option is to upgrade the equipment to support two and a half gigs over those existing cables rather than having to rip them out and go to something like 10 gig and that router also has a 10 gig port as well yeah somebody also has mentioned the Microtech 4011 which is you know there's a series of routers I think those are the big red ones that those are those are also excellent but really if you haven't heard of it this Microtech hex is a very unassuming looking router about the size of a paperback book and for $60 its specifications are just really absolutely unbelievable and it comes with a level 4 router OS license and it is enough to run almost everything that I need run to the WAN anyway so that is one thing I would suggest people look at is that device there and this RB5009 is a new offering and as Alan pointed out it's got a number of you know very nice features like the 2.5 gig so and that is an extraordinary price as well Microtech has a promotional video I think on that if people are interested in that one thing I recommend to people more than once is to build your own packet filter firewall for your your home border it's it's an education that's hard to get any other way but after you've done that three or four times you're not going to learn anymore I also use Microtech at the home border because it's cheap I tend to buy two of them at a time and every time I change the config I keep my backup the power requirements are so trivial that and the cost is low so that if it dies I just unplug it put the other one in copy the config over and we're done so I'm unwillingly impressed yeah I agree with you completely now everybody who is in a conference like this is generally going to be you know pretty inclined to a lot of the more technical stuff but those that have not used Microtech in the past there is a there is a little bit of a ramp up there is a sort of an admission fee that you that you pay to really get it done because with Microtech it's kind of the C programmer kind of philosophy that you're really totally free to make a circular firing squad and do whatever else you want to do there are absolutely no protections against that when you're using Microtech hardware and software and the router OS that they come with so it's it's it's both appealing and dangerous for that for that reason and particularly for the price so there there is a little bit of a front load to this to the Microtech community but those of you that have not um maybe you use ubiquity or something like that you would you would find the the change to Microtech to be uh just very compelling probably in many cases when you've got to use it so especially on a per dollar basis and and some of these new offerings are exactly what a lot of us would want for our free psd home labs so yeah you know the typical Unix philosophy of don't stop you doing something stupid because that would also stop you doing something clever being paid by Microtech i'm just honest thoughts on it yep um and then for wi-fi uh you know ubiquity used to be pretty highly recommended but there's been quite a few shenanigans shenanigans lately uh like Patrick mentioned just the docs aren't great but um you know if you use their cloud management tool and suddenly there's advertisements in it now as well and you're like hmm i don't know that i really want to you know have the control panel for my you know important network here uh having ads in the control panel because it means you know does that mean it could leak information about my config or something it's just not great um and sometimes yes the new UI is just uh too candy coated for some of us um and not to mention too uh ubiquitous some of ubiquity's recent firmware updates they pushed out for some of their access points and other devices have been total bombs i mean they they they they didn't work on a majority of of people's devices so it was it really lowered the confidence in the past 18 months or so with some of the uh uh ubiquity firmware so that's that's another reason i'm a little skeptical about ubiquity in addition to all of the reasons that that you're talking about and i used to really like ubiquity but i'm off of it now yeah in general for wi-fi i look for something dumb for the access point you know it's like i either have something like uh microtik or in my case just a free bsd box running ipfw um that's going to do my routing for me so my access point is literally just a wi-fi to ethernet bridge and it doesn't need to do anything beyond that okay i i have to ask why are you running ipfw because i like it because i'm trying to force myself to improve the way you configure ipfw you you understand that uh they make hair shirts for people like you uh it's more along the lines of developers tend to fix the things that annoy them so by forcing myself to be annoyed by the way you configure NAT in ipfw maybe it'll get fixed i you know i've used pf as well i'm fine with it i have some uh pf config file that that NAT's my network and it works fine but uh i've always been an ipfw guy yeah me too all of my uh digital ocean droplets uh on free bsd they all use ipfw sorry michael well uh and that's not horrible michael it's it's not horrible just having to configure it all in one line instead of being able to have a config file or whatever yeah it could be better but it's not awful michael maybe you just not looked at it for a long time oh i i used it quite a bit in the 90s and the early 90s and i was thrilled to put it aside i mean it was completely rewritten since the the 90s michael i understand there was a new uh update for multi processors and such well it's like literally it's like ipfw two or three and there's like rewritten from scratch by luigi okay i'll take another look for me i think the the feature that really sold me on ipfw very early on was being able to have a count rule for each user on the system and use that to feed into our billing system and be like this user is abusing the bandwidth on the shell server uh and things like that or even to the point where i um wrote a uh a denial of service mitigation system that looked at incoming uh bandwidth on each of the 200 ip addresses bound to the machine uh and when it saw one of them having you know its average over a couple of minutes going too high meaning somebody was attacking it uh it would unbind the ip and do the right stuff upstream to cause traffic to that ip to get blocked uh further up the network so that the traffic would stop hitting my server and hopefully mean the machine stayed up uh for everybody except for the person who was being denial serviced off of irc so i don't know if you're watching the chat michael but there is a request for an ipfw book from you that's a lovely request um how does it feel to want well the other thing i was going to mention is that the other big feature that draws people to ipfw was dummy net but christoph recently committed support uh to split some of the dummy net stuff out into its own tool uh with an eye towards hooking it up to pf as well which sounds very interesting dummy net is fantastic i used it to move a uh a conference room in detroit to germany which saved a lot of trouble uh i used it during the experimentation with the ssh performance to simulate uh you know different latencies and making the thing further and further away so that uh you could see how that impacted the uh socket buffer and so on and got some interesting uh results there so maybe maybe you can just keep the book about dummy net and not make it specifically about it i don't know ipfw deserves some love and yes uh as trooper in the chat points out i was guilty of using w ipfw for a while uh when i needed dummy net for some video streaming stuff uh i got the version for windows somebody ported luigi's work to windows uh and i was able to use dummy net on windows to simulate slower connections or bad wi-fi for video streaming of course then my router became previous d and i was able to do it from there instead of having to do it on the machine that was i was fiddling with but and of course i've used dummy net to simulate a lot of satellite networks but i'm going to talk about that tomorrow yes i see uh daniel was mentioning similar things uh so yeah for wi-fi uh tp link uh i i like um on the two and a half admins podcast we just did a talked about jim's review of their point to point stuff and it was interesting uh and i remember a couple years ago when we did um a group buy uh for bsd can and bought 20 of these tp link routers little MIPS things that can actually run for bsd on them uh and mine i have one sitting in a box somewhere in my lab uh that has like previous d 11 or something on it because that's what current was when we did it it's been a while uh and then i guess the other bit of advice i was going to give was um fiberstore.com for optics for the 10 gig stuff but you're probably better off buying DACs which are basically copper cables with the optic connector on either end uh because they're less fragile and they're cheaper than buying two optics and the bit of plastic tubing to go between them uh and they're more durable you know you can you don't have to worry about bend radius quite as much uh with the DAC versus the uh the cables or uh fiber cables uh and then i guess just a quick word on disks uh don't buy smr disks uh shingle magnetic recording does not work for zfs or raid or anything other than an external usb hard drive for backups um these days i mostly prefer seagate uh you know i'd love to tachy but they're owned by weston digital now uh oh yeah and then ssds are great uh you probably want to avoid the really low endurance ones or the the qlc ones for uh heavy workloads because they will wear out and go read only on you and uh that can be pretty annoying and then i think as always if a disk is starting to act dodgy it's just not working properly uh i give up on it i i really don't like this idea of keeping a bunch of suspect disks around and using them you know you could use them for scratch and just playing with but you know if you care at all about the data put it on disks that you have some level of trust in like yeah i you never trust a disk uh the you know disks are plotting against you and zfs can sound the alarm for you but you know if a disk has has shown itself to be uh a bad actor you it's it's better to give it away and make it somebody else's problem i have a box full of bad disks specifically so that i can write about bad disks however throw those bastards out yeah uh it is useful to sometimes to actually have bad disks to see how they act and and how it causes problems but that's for the developers to have to deal with you know it is your home lab is your data uh hopefully you care about it enough uh but remember that raid is not a backup you have to have back real backups and you have to test that you can restore them uh because you know a backup is not what you're after a restore is what you need to work uh and so you need to test the restore any final comments i think for at the end of our time now we have a bit time for any more questions that come up uh to daniel's comment about fiberstore.com they can configure your DAC to pretend it's from one of like 30 different vendors uh and they also when you order them you can tell them what switch or device you're plugging it into and they have samples of a lot of those and will actually test it for you and make sure that the way they've uh you know programmed the DAC to pretend to be a sysco actually tricks your particular sysco uh someone asked about my mathematics background yeah my phd is in complex dynamics and uh fractals and stuff like that so like real math yeah yeah i think that's it thank you for hauling us together alan yes thank you for coming and thanks for time for filling in at the last minute thanks alan thanks