 Welcome everyone this is actionable lessons from the call for testing lab a tour of the hardware methodologies and results keep point there of the call for testing lab my name is Michael Dexter and you've probably bumped into me at an event or two question are you in the right room I guess is a parallel talk you may vote with your feet at any time you can interrupt me with questions and I'll be watching the chat and these slides to this are at call for testing dot org slash log and I will put that in the chat for right now unless someone wants to just beat me to it oh it doesn't let me type okay call for testing log and so this will move quickly so you'll probably want to jump into that if you want to follow along because there will be code examples etc so the menu this morning which it's morning over here it's evening over there you are maybe getting tired but hey hopefully this will wake you right up thank you euro bsd con a bit about motivations a brief blah blah blah about blah blah actionable hardware lessons actionable software lessons and going down a rabbit hole and again thank you euro bsd con big thank you because this was all very unexpected in about 2001 I moved to Latvia I was leaving Oregon burned by rpm hell when it comes to technology I discovered free bsd jails and in Latvia with the help of peter's junishkins I set up cvs.free.net and dot open bsd because the internet situation in Latvia was pretty much an island international traffic was extremely difficult to obtain and expensive whereas local traffic was literally a land across the big cities so come 2006 I attended euro bsd con Milan come Copenhagen and then Christops Johnson said hey why don't you give a talk and I'm like oh that's terrifying okay I'll try and so from there if I click the next button and probably as Christops probably as peter's peter and so I then gave a talk in marcen and then gave a talk in Warsaw and then a talk Stockholm and attended say open con fazdem linux tag system fe in various european events something I could never ever ever do from the united states I then hosted pike con it was for the pike programming language I hosted peter hanstein to give a pf talk at the university of Latvia the msql developers were coming through town so I hosted a meet-up with them and then eventually I talked in Vienna more or less in Vienna so a a a deep deep thank you I then returned to the us and I've been organizing the portland linux unix group for about a 150 plus speakers for over a decade and then additional things such as the bsd can and azure bsd con and oscon and you name it a number of things so thanks to ryan air 99 euro cent plus tax flights and essentially your bsd con and the influence i'm a wonderful wonderful people I have been set down a rather unique path and I hope some of you have benefited from that craziness all may you rest in peace I miss you dearly so on that very point as I wrap it up uh do embrace the formality please stop you are right uh yes writing a paper is good for the brain it's good for your career being forced to communicate an idea a project a something is indeed important it's good for your again brain your career you name it and I guarantee I was more shy than you I I promise I was believe it or not it's just a skill to do this and then once you get passionate it gets kind of wild so moving on uh can everyone hear me fine uh maybe put in the chat there I yes great thank you I don't know if I'm talking to myself because hey I do a lot of that okay a little just briefly briefly about inspirational motivations and uh I've always wanted a sabbatical and the pandemic gave this nice crappy sabbatical not quite what I pictured so a little context January of 99 91 let's see a few months before the announcement of Linux I sat down at a unix system in college literally across the forest here and coming from Hollywood it was like oh whoa that's the inside of the computer I actually totally understand behind the camera that makes great sense I discovered red hat linux it's a portland linux town there's torval it's like drives by uh discovered 52 and thought wow this is a great scrappy little unix clone but uh if you haven't heard of rpm hell go ahead and look that up I don't want to scar you further but it did so many things right but wow some things are wrong and then by that time this whole notion of open source was becoming clear to people at first in a unix environment you have like sources and stuff and compilers and very little notion of what's open and free software and what you can do but move on eventually I realized looking at the windows and t stack it's like wow you can have a web server for a thousand bucks and a mail server for a thousand bucks you can't fix them you can't have features responded to you especially on the desktop features are not welcome apple at one point said no please do not send feature requests we might uh face the risk of lawsuit if you sue us for implementing your idea so don't don't don't send the idea it's like whoa that's a really crappy model so and then of course the cloud came along and things started going cloudified so this is about a lab I don't have a lot of pictures in this but uh a lab can be simple it can be a virtual machine on a proprietary os so fundamentally open sources participatory it's going to conferences it's sharing code with people it's interacting and yeah ironically we're often shy by nature but we manage so a proprietary platform can work you can have a a VM on a totally proprietary os and do all your work there or have a complete stack like oh vs code and work on open source software great that works but whatever but as I'll touch on many times there is a fundamental need for isolation and in practicality like look at the scenario put there it's like honey what do you think of haiku os and your spouse responds I have a conference call in 10 minutes what the hell is this so it's good to separate those it's just like a different machine different room different room for the noisy stuff you know so going crazy full circle I dug up my slides from 2008 I'm like okay let's let's let's see what what's there and here that just a few of my I swear it's quick so looking at my motivations it was nonetheless still rpm hell and that absolutely emphasized thank you y'all I'll get to that emphasize the need for separation isolation increased stuff did some fantastic work on like a taxonomy of isolation and compartmentalization all before docker all before all these nifty things and I threw and get off my file system I think I was looking forward to very good file systems so uh yes yon I I don't know I used quite a few hypervisors there will be a few in this talk so so stay tuned uh cross platform development yes you can have that vm you could have multiple vms on a proprietary os and do nifty things and test your one little bit of code just like I don't know jorgen does with uh open that fs on windows and mac so after you got one keyboard I'm not good with the whole army of keyboards and kbm's and all that I just want to focus so there are a lot of ways to do that be at rdp you name it but it's there's so many flexible options right now it's awesome and keep going the consolidation of systems so looking back wow across the way there I I'm told in retrospect I sat down at a 33 megahertz sun 4 490 running bsd I think 4 3 and it was great you go you type w who there are hundreds of users doing chat doing news groups doing fdp doing all their things wow this is amazing this is like less power than a raspberry pi but um but it's all happening simultaneously and then you get this whole push the desktop and clients server and whatever and it's like wait one person's doing one thing and server is struggling so back then looking at the future it's like what if it was you know hundreds if not thousands of like instances or users or people having their own little workspace it's like okay fine maybe that's so slowly materialized and then ultimately I was like I remember this my ultimate motivation and all of this and I take this to my grave is that I just want my computer to work reliably it's as simple as that and open source is the strategy to really say okay I want great file systems I want applications I want them contained and all that so there you have it that hasn't changed for decades so going more focused it has to look back well I consider the authorized because now it's security is a major factor in all things including storage which was so nice when that was like someone else's problem the authorized and validated synchronous write within hardware and virtual machines is pretty much the highest priority I'm I'm a part of a SNEA committee the storage networking industry on association and I was like well if we were to distill all this stuff fiber channel all this stuff to one thing it is data hitting a persistent storage device synchronously and reliably so fast forward and I didn't realize this way back 20 what are our years ago that means hardware machines virtual machines opens NFS and NVME which is a nifty technology because I'll briefly beat up on SATA and SAS flash devices that is just an abomination why take like a whole new metaphor and dumb it down to let legacy compatibility unless you truly need that but if you're looking forward well I'm glad that sat in SAS flash is going away slowly anyway a brief blah blah blah about blah blah blah so I have I believe Alan Jude to thank for pointing out the 2080 rule of open source now this is not to be confused with the 8020 principle in which 20 percent of your inputs give rise to 80 percent of your outputs and fire 80 percent of your clients and focus on the 20 blah blah blah this is not that so I'll describe it on the next one but in really short terms like oh my god your hypervisor booted and that's literally me when I got beehive working my daughter was oh gosh small she's now 16 that was some time ago so users cheer and bemoan your exciting new project when it's like 20 percent complete it's like whoa the future is here oh my gosh just work on that work on that and then they bemoan when you're at like 80 percent complete because it's like well wait you're missing this thing and the you know the proprietary one has whatever VMware has the thing and why doesn't your hyper the hypervisor any hypervisor that's open source have it blah blah blah so I get it I totally get it and and that said if if if you are so fearful of like this this 20 percent or the 50 percent I'm thinking like okay if you're worried about a midlife crisis you will hate a full life crisis and I can think of oh oh right here there are some great examples of 100 software that's not 20 percent not 80 percent 100 percent file maker pro 4 and 4 1 I have never been more powerful with but just pushing data around yes totally proprietary but oh my gosh I could generate like rtf files with that long ago in conjunction with freehand wow I could make rtf import it into a from a database to a presentation document for printing and have an amazing workflow and you can't have it for any fee right now and oh look at this big bad boy I've got my unix wear box I've I've never quite got it to work right I even have the SDK here so uh that's what 100 software looks like it is dead and look at what happened to openoffice.org it was shifted over to Apache like great good license great but like too late so it is like shelved there so that uh I guess a few contracts can be fulfilled and everyone moved to Libre office like well the right thing happened too late it's at like 90 something percent and the pike programming language I couldn't find many references but I think they're still alive but anyway uh python 2 that is absolutely the most timely example where it's like it's mature it's feature complete it's great it's the bane of ports right now because oh no sorry end of life get rid of that and you can't have chromium I missed the last talk I was like getting ready for this but that all said oh yeah dumping CCBs oh yeah so I've given talks on vaporware complete vaporware at MeetBSD 2010 everyone talked about like we need a hypervisor and like okay so nothing came out of that until it did and I gave a talk in Tokyo about disk control and it was absolute out of my behind vaporware and in that conversation was like what if we had a like a utility to dump CCBs cam control blocks and so these are just ideas and nothing more but hey there will be a talk tomorrow the same time about smart eight man page eight which was the evolution of uh disk control thank you Chuck for making that real the hypervisor thing I think they've made some progress from just an idea and so looking back I'd say the first one percent of software development on a project is just as important as the last hundred percent and every step in between because hey if we weren't like one and zero we like wouldn't have conferences it wouldn't have like decades of pf talks and you name it I'm going to quickly look over the chat for a sec yes so correct Ellen so for those sort of paying attention uh WarnerLosh has produced a camcorder for recording uh CCBs out of a storage stack a bit like usb dump or any tcp dump any kind of dumps you just sort of cap into the flood and hey it was just an idea in a conference maybe they independently got to that same idea but hey I recall it coming up in the talk so similarly I've got a very very gentle Lewis Carroll theme going on here which is like hey why sometimes I believed as many as six impossible things before breakfast so hey you know be creative don't feel bad don't hold back just just say it do it trust your instincts so that's a little introductory stuff in the context of being accused of giving talks on vaporware and at times being a know-op and being gee but just a noisy wheel brace yourselves for actionable hardware lessons followed by actionable software lessons I'm pretty sure every one of you will learn something in the next 20 minutes I've failed if you don't so back to the lab so there was I believe a homelab discussion I trust it went really well unless it's tomorrow and I might be able to catch it there is a log and various info called for testing.org yes the site's neglected I love Twitter it allows for interactions in instantly around the world unlike uploading blog pages but whatever I'll the log is more updated and hopefully this is totally dedicated with all that so hopefully you'll learn something number one dream big imagine your dream lab on paper from day one even if you don't have a budget maybe you'll have a job that includes that lab and you'll need to know to recognize it when you see it so learn hardware platforms vendors you name it the more you learn the more you will know which is very very helpful and then if you say wow this old switch was fantastic with this machine this old think pad whatever once you learn to recognize that you learn to recognize deals your local crankless type site SSLV whatever or eBay especially and so keep in mind hardware pricing defies logic I'll touch on that a little later from their placeholders are just fine if you want to serve a rack a bunch of free servers from colleagues stacked up and thrown together will teach you something it may teach you that you never want that damn model again ever because it needs firmware that's unobtainable and it's expensive and it's loud and your power bow went nuts but you'll learn something so please learn number three recycling centers are your friends I don't know if there's such a thing in Europe but over here in this state of Oregon there is a budget every time you buy a major appliance or a computer technology a tiny bit of money goes into a fund and that fund goes into recycling centers so you can confidently bring something in and they appear to have fair free game to sell those such that they often recycle probably 80 percent of it use 10 percent perhaps with programs that you'll see at free geek free geek.org check them out they are like a shining example of what can be done and then they sell some either in a local thrift store or online that is invaluable because often it's very recent very good stuff and then they get rid of the junk so it's kind of win-win you want to befriend those you want to meet them you want to have a relationship where they have a little watch list for you which is awesome because it's like oh you're looking for Jbots yes I'm looking for Jbots so and then hey once you're done with all that stuff you can give it back to them and I'll take it because that's what they do so uh Daniel you in there I hope you're in there okay so a funny thing about UPS we think of them as computer parts the rest of the world just considers them necessary evils so your your phone system has a UPS backing it up for reasons whatever so electricians are often tasked with bringing them in cycling them out every three years so they don't worry about batteries and model numbers and all that they just wholesale in wholesale out find some local like electricians to befriend because I'd hope you never pay for high end UPS as you only pay for battery replacement when necessary and so if you make friends with an electrical consulting contracting firm they'll be like sure make them go away they're off to the recycling center and you may have to take all of whatever's in their stack but you take the good ones you take some of the recycling or to your friends or whatever you rebattery a few a high end UPS is great for power conditioning alone and even if you get five minutes of power backup when it was originally certified for 30 minutes that's still useful and if it's free of charge great so make friends with it you'll thank yourself and Daniel you're in I believe Denmark looking for UPS and as we chatted about yesterday this is what I'm referring to so uh there are those same points about hey even if you get a pile of different ones you're like oh I'm I like this vendor well you will learn something from each one if you provide say consulting support you can learn how the UPS utility interacts with each one so you will learn something that's a theme of all of this so when it comes to a lab yeah stay consistent as your budget progresses by try to buy two of each especially for like server-ish stuff or you name it because sometimes you need diversity sometimes you need one model of everything if you're a storage developer or something or packaging software you name it but often you want two identicals like I instinctively started doing this years ago as I could afford it and often that meant for me putting one co-located big old tower system co-located at an ISP and then having one locally to like test an update you name it because uh you don't want to do that in production so as I touched on earlier lot pricing can defy logic if you want to get an amp and all fancy name brand sata cable it's often 16 euro for one if you want 25 of them it's like 16 euro for the same so it's a bit like a convenience store you've probably heard that term convenience store pricing moving on about that consistency I've been this partly from memory partly first from some random stuff I found I had two Pentium twos and big old towers and in Europe in the ego I had little one gigahertz mini itx via machines back when via padlock were a thing that was cool I see I have a an estimate which I think I bought for two tower machine Pentium fours there they were and then following that in the 2010s I had some random free service when I returned to the US and had no budget whatsoever then random pre-sandy bridge think pads then some sandy bridge onward when things really got interesting because of say virtualization support and moving forward I am more or less happy with the HBZ workstations they are targeted to engineers and people don't know what to make of them so it's like hey here's a i5 desktop for $150 I'll give an example in dollars and then the e3 is like $80 because people don't really know in that those circles like one's a fancier model than the other okay fine I won't argue with you know I've been I think I have about 11 of the Z220s that I've slowly acquired when I saw a good deal because hey I learned that hey I can have a stack of like ZFS simple reference machines by doing that the higher NZs are okay but I found that they brick when you remove and transplant ram I don't know what's up with that and that's about so I was lucky to find two compelling rebranded R720s at the local recycling center those have proven useful but they're getting a bit old the one of the first pieces of hardware that I've actually put new money into for new hardware was an HP epic for builds oh my gosh it's fast it's really fast and that's changed my whole workflow and as we get to little software bits later in this you'll see some results of that and then looking forward during a pandemic wow recertified Dell R730xd's are really handy because with a global chip shortage and supplies shortage you can have them in like a week instead of like no four months so moving on lesson six noise and power draw huge progress has been made since almost everything from like 2010ish maybe 2012 definitely onward like a Dell R710 is remarkably quiet whereas all predecessors should pretty much be recycled and put into retro computing that's some pinion there's a whole green movement switch vendors are realizing you might not want to screaming switch and there's a whole fanless movement so you know I do know quite a few people who had servers under their beds literally it's like watch for dust but bless your heart you couldn't do that before you certainly couldn't afford that before but on that point one last one there there are not a lot of secondhand fanless devices strangely moving on lesson seven oh you will thank yourself consistent cables are generally a very affordable investment and you will so so ridiculously thank yourself I was given a pile of about 20 yay long ethernet cables and I use them to this day however I later realized hey you can get a consistent batch of these little catsick thin ones for not a lot of money when it comes to that sort of cable chaotic mess please do it even with power especially you got those 10 machines from 10 different sources lined up each with a random power cord you go with a perhaps a workbench power strip and a short one foot cord to each one you will so so thank yourself and I've got power devices from my original desktop in 1989 my own college high school machine it's I've got the little power switch thing from that it's it those last you'll thank yourself uh is there a way to just make a big old raised hand or some noisy thing if there's a question because hey I'm totally happy to answer questions but it'll probably get more question oriented a little later moving on oh my gosh if you can't afford the cables yet get the screws you will so so so thank yourself especially in storage to get just 50 to 100 little identical screws some caddies have little longer ones some shorter but the cool black ones cost the same as the others and silver and black are both as easy to lose on a floor they they do hurt to step on but uh yeah get some screws you will thank yourself touching on power as you kind of grow your lab that there's that bench power strip that uh they are not computer components they're very affordable and go with that go with some short power cords you can have a really clean lab for very little money uh cyber power oh my gosh I hate the name does some rather affordable pdus like there's one that has a little number with uh the current draw and in the middle is a little checker there are various forms in the u.s or ones with like three lights and some fancy ones like this you really want to check each port before you deploy it because uh yeah I've had some that actually don't ask it's just please check your ports moving on there are lots of full height cabinets out there just just say no send another recycling center half heights maybe I found that these little like folding nifty little micro racks designed for like the tiny telco thing on the wall are really useful for like a switch or two a kvm and a few little goodies and a pv u and there's one in the garage there's one upstairs in the lab and yay uh a two shelf rolling cart super useful I've used the same metro tall shelf stainless cost 100 bucks back then they're now like 500 timeless priceless go with it it's not a computer component but it's fantastic at that and of course hey the like you lack rack makes a great story you probably go search it maybe put in the chat there the lack rack which is made on this sort of lack brand furniture oh for a server but it doesn't have a lot of meat to tie into with screws and all that but it makes a great story anyway moving on I don't know what changed there uh yeah so storage devices in the good old days I would have said yes invest in some oh did I lose video and oh I lost video when I did sorry I'll poke some push some buttons here sorry you need the hand gestures of course I think your last hand just show you in the good old days or something what's that yeah I turn to the lack rack I'm so excited here here we go um I was lucky to get some cheapo bx 500 drives they were like 20 bucks a pop and now they're like they were 40 and then they were unattainable or less my microphone too so now with the pandemic and supply shortages and and mining operations good luck out there it's it's brutal but uh all hard drives are terrible in some way so you'll learn something from all of them and as I touched on earlier u.2 storage is the future in my opinion wow it does it does away with satin sass interfaces and is like a native PCI interface yay and the support in the operating systems one might discuss at this conference it's become has become very good I previously beat it beat up on why sass and sada shouldn't have happened it's about an abomination so as for Ikea those tubs are amazingly useful I should have brought a sample but they they go from like this big for all the little parts and maybe cpus and ram and screws and stuff and a little cpu paste to big ones to you name it those are ridiculously useful and don't forget the lids that either sell it with the lid or without I don't know it's don't screw up and leave without the lid because you may not get it next round especially during a pandemic anyway moving on to lesson 13 network infrastructure yes network um that's a whole different sea of ideas and categories and products and not my specialty however I found that going with first whatever switches your colleagues will give you it's like the best switches in the world because they're yours and they work and you can experiment learn something from them maybe learn that you hate them maybe learn that you love them but I found that some just big old ports all across the front 48 port dumb switches are always useful even if you're getting the fancy switches have the dumb ones just because it's consistent there's no like patch two's a forum great in the lab you'll thank yourself and I'd watch for prices to drop on those for under a hundred bucks you can generally get a decent 48 port switch that's my opinion so document as you go you will so thank yourself serial numbers license keys when you bought stuff IP addresses for lab I mean I've learned to be consistent and it's really helpful like one password for certain lab stuff and sequences and the IP MI IP below the the machine that it controls firmware revisions all that and even things like with those little clunky z220s note the RAM and such and when you're shopping for more and you see a great price and it's the exact model you want move quick so the more you document the more you'll thank yourself moving on ultimately know where your data is this talk has forced me to bring together previous presentations and I'm sure like all of you I had like presentation that PDF which is great that week and terrible eight years later so name things well that's in fact you know in the context of ransomware and data protection it's really good to know where stuff is but let's just say you'll thank yourself for every single bit of organization you do there it's good to know where those isos are sometimes you need an old iso of really old stuff just hey try to stay organized and looking back to like the 2008 or thereabouts none of us had space for many isos this were tiny and slow and expensive and so now hey knock yourself out it's easy so as you grow your lab this is the foundation of growth it's like oh here's the model I want I need some of these and back full circle to the first slide have on paper a design of your dream lab and build it up as you go it's worked for me hey and it works in a small apartment I did it in a small apartment within confines of that and a budget and now I've got stuff in the garage you name it so I've found that collaborative docs are super useful for that especially if you say give someone vpn access to that lab and say hey okay here's where to go find that thing please install for this machine you name it there really needs to be a open alternative to google docs be it called collaborate you name it but it's not quite here in my experience maybe it's here maybe it's not I gosh this this presentation is in google docs I feel somewhat bad about that but my gosh it's a useful tool then on to the consistency you get out of consistent devices well test that they're consistent if you get two little machines that are in that whole palette of them whatever make sure they all perform similarly hey a drive might be failing on one or for whatever reason one has some very inappropriate ram that makes the performance much worse so it's actually good to just bench stuff have a nice baseline to compare everything to and there was a this guy here in town who built cables in the 90s back when cables were like a totally fancy thing and he's like okay check every pin the person who made this cable has never used a computer I'm like oh gosh okay so I have definitely had issues with those power sockets less so with network devices believe it or not but yes and eventually they will give you trouble I think in the last homelab session at bsd can I talk about power draw that's a whole different thing all itself I have a lab doc linked from call for testing that does go through everything there with a whole bunch of model numbers and you name it I should post that at some point please post it if you find it there uh check for noise noise can be a warning of bad things it can be a warning of oh you brought the wrong the wrong model you name it such test test test you will learn whenever there are identical systems yeah verify that because you never know anyway ideally you log things cradle to gray from the moment it arrives and you document the warranty from day one and you have a plan for it years later at that recycling so yeah lesson 16 anything can fail it's funny anything literally anything can fail the simple double is the dumbest power cord can fail never rule out anything new and expensive does not always mean it's not going to be do a dead on arrival be nimble when say giving a talk and having two more computers here handy my nifty new and this could should have been a slide the ebay not so cheap but not so expensive think pad t480 is quite promising i have my wife's air there i've got machines ready to quickly nimbly jump between them do the same in a lab because yeah stuff can fail ergonomics matter uh yeah personal protective equipment as we've especially learned during the silly pandemic is not that expensive and you kind of like designer masks like where the euro bsd con masks i guess it's not in person but it might be nice bsd can mask maybe maybe so uh yeah i'll i'll give this an example in a slide or two but it just protect your ears the treatments are expensive and crazy and like disposable gloves are handy and i've i've hated every keyboard of my entire career so i cannot say what what to get no idea not a clue uh sitting technologies at the very moment i am in this bomb box saddle chair i picked up in the nineties i mean not unplug myself but you probably haven't seen one of these and you probably won't see another one so it's like literally a saddle but with a back optional back you might see them at say european cold checkout line or in a medical facility where it's like whoa you mean you don't force your workers to stand all day that's weird but uh such as life invest in all that you will thank yourself and a chair might last a decade whereas a certain raspberry pie is those uh weeks until you burn it up so hearing especially i live in these especially with like kids running around and uh flying to say asia yeah they might think it's the fancy beats whatever no one's canceling no it's just like it's probably what the ground crew has outside the plane but uh combined with now these uh uh wired bone conduction headphones and fatties on twitter blue collar mage suggested them and oh my gosh he needed them in a machine shop so you can hear people and like for safety but like hear music so they're a bit teeny but oh my gosh they're great and being somewhat flush and these kind of squishy you can actually combine them which is cool so i i live in those and doing support calls i have two of those that i've been picking up second hand because the wired ones are now discontinued uh for what it's worth i like the think chairs i saw mention of a chair but i i literally have a second one they are usb battery powered that's a there's no power little indicator of like how much left but hey oh they've changed my workflow and i was so nervous about in your earbuds nuking my ears for conference call after conference call so moving on learn learn learn learn learn everything will teach you something even that big old thing you tear apart because it's dead and you just see how things are laid out and what screws they use go for it colleagues are fantastic the hallway track is invaluable folks like alan who just chatted is like oh well i use this kvm and this whatever just milk your colleagues for information they're brilliant and they're sharing and share what you've learned it's great it's two-way street uh i keep saying it hey knowing what not to buy is just as important as knowing what to buy that's very key when someone's like hey you can pick up this whole palette of cool switches that are all power draining they're screaming loud you name it so again there's some great stuff on reddit there's a homelab session here pay it forward hey you outgrow a laptop give it to another developer just there's i guarantee there's someone local or at the conference or otherwise yes i've seen quite a few handoffs at conferences like oh right here's the new thing go for it enjoy and you can see more notes on the lab documents there and i think that links to the homelab google doc and then there's the log which has some goodies and finally my absolute secret weapon is what's called over here painter's tape it's easy to apply easy to remove and uh what's over here a sharpie pen and often i just have like okay what the disc is how good the battery is and what os is on there because often you know those things how'd they look alike so us bikis especially like oh my gosh they they'll have this like scribbly like free nests whatever version you name it installer or fruit device all those flash media used to have a tiny little field we could write something once like no no no no you you have to iterate quickly this is priceless any questions okay buckle up software lessons absolutely preaching to the converted own the stack the bsds and alumos are the way to have a complete operating system stack where you can poke at any layer of it talk to colleagues at any layer and fix stuff improve stuff and make a career of stuff and make a conference talk of this one little thing the tiny little thing down there so they are they're awesome and that is critical that is just absolutely critical to standing by it delivering it delivering a service a a reliable internal service whatever you you all know that but just just hey let's all be thankful for that there's a consistency of these systems the documentation it's unix behind the camera that's how the system works that's just how computers work and there have been countless unix replacements written in unix and though they will continue to replace unix for decades to come with unix uh control t super useful opens out of s oh my gosh detrace beehive jail zones i can suppose i suppose throw zen on there we'll get to that later uh the bsds bless your hearts upstream of open ssh pf mandoc open rsync countless heavy lifting technologies that are like well documented reliable with cool people awesome i love you all um that is a unrivaled highly flexible repairable toolkit i like this toolkit and i guess you wouldn't be here you totally disagreed i'll do a quick one on everyone knows that because these are little things that oh my gosh blew my mind and here they are because you should you probably know this but there's no shame in not knowing this because everyone learned something at some point so control t the progress of the command line utility it was at the uh solaris day oh zfs day hosted by i think joint back 2012 ish before it became the open zfs developer summit uh someone was flashing a usb drive into control t and like what the hell was that what whoa whoa cool wait i didn't know that it was on mac what what what the hell anyway um so you install a previous package uh type rehash if nothing's like working it's like oh okay um i that that's really helpful if you uh didn't know that uh if you've got some later think pads like this old t 480 uh function control k gives you the scroll back because on the hardware console that's still really useful on phibia so you can panic the system at will which might be useful for useful for testing debugging you name it and it's like why don't they tell you that and uh if you hop into a single user mode it's not always intuitively you just type exit that's it and thank you alan and claire or someone at claire uh i did not know you simply do shut down now and it kicks you into a single user mode exit again kick right back what's up wait what it's sad but hey you learn uh with it enabled you can do control out escape on phibia see you know it kicks you in the debugger which as i'm racing through early beehive is like i panicked it's like oh i didn't and what the hell so that's kind of cool oh from i think uh a little chat earlier of maybe late last night uh that's the update extract before you uh as you're doing an upgrade because hey you actually want to do a three-way merge and you don't want to lose the opportunity as you go a little further down the road there someone suggested i think yon of like hey uh you can't read a man page you don't know about so it's actually very helpful to go rummage through them look for a name because they might not be found in apropos and then you can pump them into mandoc directly or z-grep or all sorts of things so that's your friend that was a great tip moving on a few zfs ones that i've seen it's funny you can type like zfs get written and it shows you the written property for all your data sets and like uh wait i saw a client do that in belgium ironically in a hotel i'm like what the hell did you just do i i didn't know you could do that you don't want to specify each one and there you go very useful uh z pool import dash n without mounting anything fantastic when you're replicating to it i love it i love it do it all the time uh as yon pointed out hey zfs diff it's it's powerful it looks at two snapshots compares the content tell you what's modified deleted you name it very cool i will touch on that a little later uh z pool import dash n a dry run i almost say that you do the fault my gosh especially when doing something fragile it's like this should work this shouldn't work look out you name it oh zfs set read only equals on you can have immutability free of charge anytime any place instead of like unmount file system remount all that stuff we used to do it's free it's awesome i'll touch on that later and of course show the snapshot directory is visible don't aim rsync at it you will be unhappy but what it gives you is like a virtual representation of every point in time it's awesome okay moving on so you got that thing pad or similar you name it well acpi comp dash i012 however many batteries you have it will tell you the design capacity of that battery and the last full capacity when you receive a new laptop such as this one off of ebay it's good to know if that's like 80 percent good or five percent good but like new ps batteries your mileage may vary very very very helpful i get to see broken stuff all the time and these mca errors are terrifying that's it on free here free bsd if you have say a failing memory module it's not always as simple as the half of band management saying hey this module is broken go replace it with a clear guidance of guide of where it is because hey it's like eight modules are they all alike in the cpu one or zero or one so on some platforms that warning is giving you an address however with dmi decode you can go look at your modules and it gives you a start and stop address this is definitely on super micro not sure about del might not be but key point that failed address is an offset go find the module it's within you'll thank yourself moving on so there are a lot of varied leads out there in this community because we all kind of focus on our thing and our thing so over the years some before has sprouted the ability to be a somewhat useful active directory server and it's funny using like say free bsc 13 and opens edfs and some before you can have one that could be a secondary or do a number of things i have taken a little old proof of concept i have slammed it into github this is not an endorsement of github but there it is i welcome your pounding on it i simply configure it on please a new separate lab system and give a whole bunch of commands you can use to add a user and poke at it and check it and make it break so i i would love to hopefully find someone here oh who comes to mind patrick hello patrick maybe let's take a look at that about more varied leads free bsc has become a pretty good zen host that i touched on that earlier so it's pretty solid it does things be hive can't do it is lacking zen domains but i have a review up and i threw it into this repo so here is a script called xenomorph to convert a working zen uh free bsc 13 install into a zen host and then an example zen domains which is the rc script that controls starting stopping you name it uh roger is a busy person he's doing his best i believe in parent main on free bsd it supports uefi boot traditionally for years it's been only on legacy boot it's a whole like os within an os and it didn't even make the release note so hey take a look here's an actionable item that you can go spend an afternoon with moving on smart control cuddle whichever term you prefer finely sprouted json output and there will be a talk tomorrow about smart utility which does a whole bunch of things different and is permissively licensed could be used in a base system or instead of fs however uh here is how you parse basic smart control output with json it's only relatively recent versions that support it but there it is if i want to find out the model number the serial number of kind of as you can see fudged any uh data here in humorous ways and form factor and speed you name it block size reported block size go for it yaml don't know take a look uh come to our talk tomorrow so that's useful actionable detrace for the rest of us it is awesome when brendan gregg produces a flame graph that like saves the day or yells at a desk array or you name it however uh yeah i just need some really simple stuff uh git and v trace have a funny habit of like formatting not quite machine parsable not quite human readable like a flame graph it's just something in between so i'm like okay i will figure this out so i sat down i used dash q to shut up a lot of the like headers and goodies and then very carefully chose the uh very much c print style or even shell print f style of what i want to see so here is some detrace code to to show the user id 0 epoch time stamp but in like nanoseconds so you can either chop it off or do whatever because hey of course if you're chasing on performance you need fine grain time and of course what was run so i just ran it and sat back and then whenever chron said yo you want to run at run okay i ran at run if you have some piped commands they will show up as two commands right after one another which is fine so you learn to interpret that but uh i can parse the heck out of that that's useful to me flame graphs have their place and they're awesome in conference presentations but for me like this is useful so hey go ahead copy paste so yeah uh this sucker has a windows key and um someone paid good money for it and i technically technically paid good money for it so it's good to like find out what that is and they funny they don't have a big old sticker anymore which is now like they got smaller and smaller and then they slammed it in a bios here is how to obtain the windows key from a probably ivy bridge and later laptop possibly desktop from you whatever ufi place for acpi where it's hidden away uh here's a syntax to grab that uh it takes the acpi ca tools and that is not a typo and i checked it three times no that's the name and no that's not really a oh t but it's got hints of a real key so you might find that useful and slow slam it into your documentation system please you'll thank yourself especially i'll go back to say especially when your kids are like thrown into homeschooling and suddenly all your lab previous older lab think pads become homeschooling machines which they didn't so now down the rabbit hole in the context of open owning the stack here is something that free bsd can do extremely well this has been a journey since like 2003 i'll get to that i was hoping for maybe some rabbit hole imagery but i didn't want to violate the copper copyrights i thought this medieval images somewhat appropriate for the battle i've been fighting so let's take a moment to reflect uh free bsd 13 is total milestone in obvious and not so obvious ways oh whoops i should have thrown zen host on there it's like it does it it's real it seems reliable um that that was not working for a very long time it draws upon upstream opens at fs it includes reproducible builds it is pulled from git it has working source dot com build options and i repeat it has working build options and some of you have heard me just talk over and over about this and some of you are like what the hell is the build option so let's get into that briefly briefly i promise so going back to 2003 i discovered jail and second oh sweet great um so you're saying i could have like a binary for send mail a few dependencies throw it in a jail and bang i have a mail server that does one thing well cool great i like it thank you i'll take it and you could there are tricks to like subtract stuff until it breaks or use ldd to find dependencies and build it till it works and there are a lot of ways to do this however for decades pre bsd has had build options in which you say include something don't include something sometimes that means including an experimental feature sometimes that means removing stuff and removing stuff was broken for a very very long time i'll touch on how to detect those things what to do about it you name it but for now hey from a like high perspective like you want a faster build build less you want to contain software wow maybe we call them containers i don't know if you look at man the manual page for stars dot com you will see a whole bunch of say on 13 230 ish options without accounting without a cpi to start at the top there and some of them are with where you can add a feature with us and mail yes young um and so i thought oh how about like firewall block all add what you need how about i just turn everything off and add what i want till it works um that's been horribly painful for years until the 13 release and i want to i will thank everyone who helped make that happen here's a random example so you go into like source and do show config and it throws out this this list of where it's at so mkvi let's make vi and in this example it's an okay example uh when you're building the rescue crunch gen items include vi it's like okay fine and then yeah alias at ex fine so you can get to control on off on off flip the switch so uh for for a very very long time uh paul hannon kemp has had the build option survey sitting in user source tools tools and i cannot imagine how long it used to take to run but it steps through every build option it looks at how much space it saves if it does in fact work and some of you may recall me running the build option survey as best i could on just hardware and just waiting and waiting and saying oh this is broken and talking to various folks and say hey should this work that's the first key question is this known to work did it once work etc so i took the the build option survey and fixed brought it a little bit in the modern era so you can only say choose test test the ones you want focus on the hard ones rather than everything because i'm pretty sure without vi is working it's i haven't had a problem and i've thrown in some tools like find faults which when you get a a build failure and you get a whole bunch of garbage and if you've got say parallel builds you've got like failed here here here here here and because they're all the parallel things that that of course died with the one thing that really failed here is a grep syntax to look at that output and hop ahead roughly to where it says stopped because that's when it actually failed and then a little syntax so that's been super useful for uh for seeing how and when a build failed on you so yeah i did my nudging my grumbling my reporting and thank you kyle evans thank you michael horn thank you beyond z who got network without networking to work oh my gosh and ed mass who's been really kind of championing a bunch of this thank you thank you uh there are a few oddballs that either need to be enabled for it to function at all or need to go and source and whatever thank you this is huge progress and the foundation for a whole bunch of coming stuff so that said you may have caught my talk at nice bug about outcome bsc which is like well what if we just streamline the creation of a a bare minimum virtual machine or jail leveraging the heck out of a minimum source dot com actually a maximum source dot com which is tons of stuff you don't want a kernel configuration file that gives a bare minimum and currently it spits out to be high zen and jail zen isn't very tested so that's like literally fresh jail is kicking into the root directory in the jail which doesn't seem like it's correct but i will look at that as soon as possible and thank you anthony for taking a peek at it so i kept finding myself doing that now that they work so i threw them in a library and the library can be found in outcome bsd it can be found in the next tool you may find that useful so as we get there if you build free bsd it's handy to do it at arm's length so you like have a separate object directory and you're specified in your script you have a source dot cons which is very important when doing this you specify where to find that source tree you maybe set some parallel builds and when it comes to install time you set a destination directory and uh two kids have just walked in and walked out so many of you have probably seen death steer for like jails and such for installation but here's all the other stuff for the other part and i'm one it's really useful two i will leverage the heck out of that later in a few slides so yeah let's talk git for a sec because hey uh git's here it's a thing cool and i keep hearing this like false dichotomy of like developers versus ordinary users well yeah okay there are yes there are committers there are countless developers all 90 of those 99 percent are users and then there are yes independent users we all consume the source tree in some way so here is some copy paste useful git goodies that took hours and hours to figure out that aren't super well documented hey it's actionable it's free to use go for it so here's how to truly mirror the repo so and please submit fixes uh in the software that draws on this or in chat wherever you name it but let's mirror it and keep that mirror up to date let's pull a branch from it here's the syntax to do that and notice that's like the source directory will sometimes be generated for you so keep track of that but i i verified that all these are correct you can copy and paste change the urls to your liking and verify again a lot like the make-c independent position independent syntax in this directory run this command yay okay i'm in the right branch good okay thank you so here's how to check out a specific uh uh hash commit cool well funny i want a source tree and a specific one great i don't care if it's a tarball i don't care if it's from svn i don't care if it fell off the moon yeah cool it gets does that well i guess but here's where it gets interesting so i'll run through this git-c go in this directory hip okay fine log log is very interesting i went with reverse order uh what did i just format for the syntax oh for detrace i formatted the output syntax so you've got your like unix timestamp a unix date ooh i know unix the hash and the summary okay well let's clean that up so we jump into this branch we start at the beginning of that branch such as create the stable 13 branch as our first entry thank you edmas for nl no that's not netherland that is like just make a list out of it okay start at zero okay you said to chop off a leading space or two and throw in some tabs which are these funny characters here and slam them into a source log what you get is a numbered list with an unix epoch date stamp totally parsable and translatable and malleable that got awful hash maybe you could use a shorter one type g are they truly global i don't know but fine that's a git thing and then the summary this is super human readable this is human and machine readable well human in so far as i recognize it and i know what to do with it this is really human readable that puts git on my terms cool this is suddenly really useful and these log operations are fast that i like about git so that i hope makes up for some sins that git commits and yeah go check the you can put in the link the dummy man page generator command generator it's funny because somewhat true so jumping forward for years fortunately there's not like a 13-0 thing there's been meta mode and here's how you do it and i i knew about it for a long time and i finally started using it because i had a really pressing kit use case you load a kernel module you throw this flag into your build and off you go which means you create you do a build world a build kernel you name it the next time you do that it looks for what's changed in the source tree and just skips all the stuff that hasn't changed and funny when you're following a branch not much changes with one commit so that is your friend and i will use the heck out of it shortly so some of you may have bumped into my crappy sabbatical project which was up to bsd lv during this pandemic it's like okay fine free bsd update let's let's track a any branch a stable branch a main head current branch you name it so thank you connor bay for helping pound through this and yes i figured out how to use colin's previous the update figured out how to put everything in place to build releases to update to make it make it available it really didn't help that the project switch to get in the middle of it okay so all the time that was going to go to improving it went to like just accommodating get fine but that was really educational and gave us those few slides a second ago uh out of that came a tool you're welcome to try this absolute beast of a build script that takes in like architecture with a flag the exact architecture branch you name it and does everything to pull it in build it barf it out you can tinker with it but the components of that have been more useful and they will show up shortly too long didn't read free bsd update is not free bsd upgrade yes how we've always done it however it was just designed for taking patch levels and building them and applying them and great but when it came time to upgrading a system it is really not the way to go try interrupting and upgrading you'll find out exactly how this really that is a statement so a touch more reflection free bsd update did its thing does its thing okay fine but however when colin made it it came prior to metamode builds it came prior to reproducible builds it come it came prior to opens out of fs it came prior to get and the logging i just mentioned it came prior to like oh my gosh powerful epic cpus and it came prior to the ransomware epidemic which will actually play into this uh colin i am mildly disappointed you didn't produce a proof of concept on at least three of these because you are an amazing developer who have produced some amazing things but hey you're forgiven other kind people stepped up and made them so in the big picture there's package space at one point i heard of six different possible implementations one of which is a joke i look forward to it and for all that build option housekeeping and hygiene i mentioned earlier i'm sure it serves that purpose so hey it's all good so let's reflect a little what if when you use like previously update for an upgrade you you start with a dvd iso it's like what and then you you pulled apart you build stuff a few times you run through a bunch of steps you security sign it you do all this stuff i'm like oh you okay as alan would put it noob it from orbit which is an alien's reference plus your heart and so whoa whoa okay so what if builds are cheap like really cheap like crazy cheap thanks to metamode you name it uh reproducible build so it's like oh we're we're following a branch and we want to see what's different well if if everything is as binary identical as possible except for the thing you changed in that one commit well embrace that and then opens fs is not the top subject of this talk but wow it's powerful it has things like oh snapshots it has diff it has all kinds of good stuff which makes builds really cheap so what if you take metamode you take reproducible build you take opens fs and you just funnel it all together and you slam it in epic and you use say ockham bsd so you can build world in like one minute 30 seconds you can build a kernel in like seven seconds and you can iterate for testing really quickly in ways that were not possible when i sat down a free busy first people spent like every night building and then installing and wow it's like that can't be the most efficient way so well that was then this is now get navigation makes branch navigation really fast opens that a fast makes immutability really cheap and again dynamic you don't have to like unmount your root file system or uh u mount dash u w root and stuff like from single user mode and all that that's all gone that's then and things like chaflags and crsg it's like wait oh you want these few files to be immutable how about you just make the entire data set immutable it's it's free it's great it's awesome and of course ransomware if you look at ransomware mitigation the number one suggestions are stay up to date and like make things immutable because hey it's hard doing group stuff that you can't read so that's been a good motivator you can catch my snea talk this month on that topic and zfs but looking at all this stepping back snapshot every commit on a branch the fs don't care it'll handle that snapshot metamode as you go gfs don't care diff and clone each of those builds tools don't care rsync binary diff is actually kind of useful and probably call in binary diff so let's see what if you had a read only kernel data set and a binary data set like bns bin you name it libraries that are all from upstream zfs don't care it'll do it and uh what if you were even to slam the binary results into git get don't care throw it into svn svn don't care and wow there's a tool for checking out svn in base what do you know well so that all said i've put these together as a workbench for building a golden image based on a branch and stepping through every commit how however far you want to go you've got the immutability when you need it you've got deltas and diffs where you need them in line with my terrible project names i've called it petri bsd because it's like a petri dish where you kick that off you create a golden image you create a virtual machine image you do a zfs send from one to the other i still have some boot issues to work out about how exactly you hand off and there are countless ways in free bsd which is actually really cool you hand off from a kernel booting to say what is hopefully a mounted data set but if it's missing certain components it's not mountable so that there are advantages to be had there but actionable on github i uploaded it last night actually technically this morning have at it you i it's a bit rough but once you see what's going on there uh go crazy as for listing snapshots i've seen systems with 20 000 empty snapshots the fs don't care it might take time but it'll do it correctly it's not like it'll tip over so got it uh yes you could follow the entire free bsd or operating system of choice history from day one it would be inefficient doing it on branches is wickedly quick it's all good it's all good so seriously thank you uh that's a little of what i've been up to and this talk has technically been in in production since 2003 when i first sat down at jail and discovered build options uh i gear i miss all the languages of everyone speaking but hey you can you help pick pick your thanks there but seriously without your bsd con i wouldn't have gone down this road and this i hope you find is absolutely all actionable and i hope you learn something so thank you so much i'm totally happy to answer questions both here and now or possibly in the social contraption or offline thank you i okay uh organizers tell me what's next i'm happy to stick around it's morning here hey you're probably all tired now i got to kick the coffee when my daughter's bob ross cup is finally kicked in i'm like you'll have to rock and roll and for the social event i've got my art bag anoa got my aja bsd con cup man maybe i should have a little small tiny celebration even this morning here that sorry to miss the social event sorry to miss all the things you're welcome tom i don't know if you can hear me i can't hear you but hey um i hope you're grabbing the chat because i think there were some links here there's a spatial who is in that space um can't hear us because we're all staying muted oh jeez well i'm happy to talk and you are welcome to join the same space tomorrow when chuck and i will present on smart a permissively licensed smart data utility for bsds instead of that's another fun thing oh domagoy hello yan hello tom i want to see all of you in person ah rick i saw hanning did a great intro aw seriously any questions you're all just saying hey thanks uh michael check the shared notes tab ooh a new thing so my my test thing was like a minute long somehow because time zones and thank you rene for doing that was better than nothing and thank you chuck for explaining oh here we go oh i didn't know that i'm so sorry which hypervisor yeah um there's zen there's behive of course and uh if you need nesting i say just use uh kvm on even like umbuntu on zfs go for or proxmox go for it you can nest uh techniques for isolating a dying fan unplug it replace it um check your temperatures uh if someone mentioned i think the question was how to figure out which fan is the one that's making the squealing noise yeah hopefully that is um well if you're lucky obvious going low tech remember a ups is not always a computer component automotive folks have a little ear thingy on a hose that has a a little tube you can you can fashion one of those out of just a piece of paper roll up a piece of paper in a cone shape and try to use it literally as a a sound funnel and it's amazing how even to figure out which hard drive is clicking in an array of hard drives it can really help narrow it down i was thinking of maybe ordering a stethoscope or something but yeah that can help our mechanics use that's a good idea sometimes it's obvious uh if a fan blade is missing or something like that no it's i've got a bearing that's it's a bearing that's a good chance you one of the rpm readings will be uh by a hundred rpm off not yet but i can sure hear it i've answered the third question directly why not um so yeah you want to use it m.2 or or u.2 nvme device on an older system there are pc i cards that adapt them and you the bios might not support it but your os generally will so you just use it as fog use it as your high-speed pm storage go for it and it they they worked well for me i'll leave it at that uh builds faster yeah nested virtualization uh beehive by consensus probably will never have nesting because it's a whole lot of extremely difficult complicated stuff that doesn't necessarily get much value it lets you have say uh under development environment but most people don't need that so i say as i mentioned use kvm why not do it use i think beehive was mostly developed on vmware fusion on a mac in a cafe so bless your heart peter that was awesome um zen has absolutely rudimentary nested virtualization to host zen on zen and i think uh hyper v can support hyper v on hyper v to a point um i don't know use jails i mean look at podrie has now has image and i think it's daniel or someone suggested hey why not put huckum bsd as a provider there well i see they just grab a source.com so generated externally slam it in uh go with the podrie image build and off you go so jails and but i'm um nested jails or even jails and virtual machines are absolutely your friend um i didn't want to pound home the point that my talk on on isolated build environments is very much where this has all been headed over the years and yeah that was vaporware but now the components are in place to at the moment build aggressively like crazy but you probably need to build on a previous version of the os which might work in a jail or it might be a virtual machine we're building for the purpose of another virtual machine and oh oh oh so there is a blog post or two on how to build previous d under say linux and mac os there is free bsd come linux compatibility has anyone built previous d under linux compatibility and the build compatibility i would be very curious uh package base i tried a snapshot it was it installed and someone is has an unofficial repo which is kind of cool where you can apparently track it i i don't have a url handy but it might not be hard to find uh i've been asking for years at conferences and hey i i hope that's the future oh and on on package base of all the weirdness of red hat five two which at least match the documentation i love the fact that everything's accounted for through packages for the base system so literally from the framework of files on up it's all in a package i'm like that's great so yeah maybe we can have that too i've only tried it once boom uh public chat oh so chat and shared notes are separate so you have to hop between them okay yes but i've seen people successfully do the build free bsd from linux using the the stuff that's out there now under free these linux compat oh no but on actual linux uh it's just having the the packages or whatever the the python stuff to to replace to build enough of the previous details to build free bsd with the tools it expects that is something that could use some documentation love because there might be some use cases that really aren't obvious so hey uh i think that's a nifty cool possibility so it's yes it can twist it and counterintuitive but at least it validates all the components and validates that the build is working and in the course of all this i think uh auditing the base system is a lovely future talk for somebody uh is when are in this capable call this week uh what is supported for um static building of base yeah okay maybe pam needs his modules and can't be built but what can be and what should be but is broken so yeah there's rescue and we know that works but there are other components that have been throwing me some really weird errors such as a libc thing in zfs and and some dump utility some bizarre thing so nfs root is important i realized if much of the pain from nfs is from all these like uh csh or g whatever flags and just flags if you're using zfs and you can do immutability for like free uh maybe we can just turn off all that legacy that's how we used to do it functionality with special flags oh and quick tip ls dash oh i believe lo was we'll show you the flags and i that's a nice grab bag one i should have thrown in there i learned that this week thank you all and so just wow conferences are awesome these have if it's not obvious been a little side projects proof of concept you name it that have been sitting spinning on a drive for years and this forced me to dust them off polish them up they some are more rough than others i will admit that but get them out there and github isn't perfect but it it's what people are using and i look forward to moving on to the next thing whatever it might be other questions you've got me it's morning it's saturday the chat shared notes boom and how are we on time i have never once confession i've never once looked at the time for practicing a talk or planning a talk or anything and yes i've gone a bit late on some your bsc con at age of bsc con ones my apologies but often there's a break and often there's time until the social event so no one will complain i'm going to look at the list of your time was uh up 22 minutes ago oh man i'm so sorry you can get out of here but i'm happy to talk there's nothing the social event starting in a couple minutes and so i think we'll move over to this special chat or whatever is it purely spatial or is part of it bbb um i don't know for sure spatial only as far as i can tell cool okay that's a neat tool um that that's clever it's a good uh attempt to solve the problem and oh it's a problem i'm stuck you're far away from all of you and can't like buy you anyone a beverage but hopefully come what bsc cans of consensus maybe we'll be in person possibly it's uh wait and see right now of course was that adam tomson you've been that was how are you i miss you i too would much rather be doing this in person hopefully we'll be able to do that next year hopefully what's funny is that it meant my preparation was well longer than ever and deeper than ever so it's a whole bunch of actionable code that you're welcome to go poke at rather than some ideas so hey i hope question for the chat did anyone learn something in this was i successful yeah so i get stuff was interesting uh different than the one i wrote for you which was you know who was the last one to touch each file in this directory that's next yes totally that is that that once i realized the power of log it's like oh okay that yes thank you alan i'm yeah specifically when it has that dash dash format where you can choose what to output and be like oh i want a csv that i can stuff into a thing it's like i want to show me a list of of all the commits between these like you can also give it date ranges say every commit after this date and before this date and then be like you know uh only the ones that contain these words like um you can do git log with i think dash dash grep and you say only ones that contain this keyword in it somewhere so you can basically grep the commit log but in a format that you can parse like you were doing and there's all kinds of other filters you can add so looking back git is a source code database and happens to have source code management bolted on and does some stuff but hey uh like i said i i want a tree it i don't care how it got there however those tools are really useful i guess if you haven't seen it are you familiar with the git work tree sub-command oh no okay so if you have a git clone of the free bsd source code or whatever sometimes you want to work on more than one thing at once so if you do git work tree and add some name and then the name of a branch it will check that out as a sub directory in your current checkout oh okay and so you can have four different projects going at once on different branches um and but they all share the same dot get directory so you don't you know waste the space of having four whole git checkouts or you know when you pull in newer code it's available on all of them and you don't have to so like my my checkout of the zfs one has like nine work directories with different projects going on in them uh or it's even just handy to be able to have a checkout of you know uh main as well as stable 13 and stable 12 or whatever uh i will share one git thing that burns me twice this morning and i mean like late night morning so uh don't do this don't create a new github repo and commit to it probably any repo but i have to be using github make your initial commit realize oh i meant to rename that file so you just like revert and then you go into rename a file and everything's gone like oh thank you git so i get a red flag uh we'll let you get back to before you reverted and so on i'm glad they kept that and the the i thought intuitively i thought the revert would be like go back in time nuke the current one well no no no it didn't do that and everything i was working on vanished until i realized yet still stuck in there so i retrieve it out and out of spite nuke the repo but yeah um uh that is worthy of a blog post of just don't do this you'll you'll you'll cry especially hit like 2 a.m about to give a talk on that very subject i want to get some code that's actionable after your friends and colleagues okay the as with every mobile interface the interface for that uh social chat is good but not great i found that like closing it logged me out but i'll try it i'll jump on social um well doing it with the computer seems to work better and then you can drag your head around and um it's it works really nice as a hallway track because there's different areas and the people you're standing beside are louder and you can move away and talk to other people and it's uh instead of having you know four breakout rooms you kind of get this more organic field of being able to wander around and listen to what people are talking about without you know randomly jumping in a room or whatever i'll try it i admit i'm a little saddle sore from this thing but the downside to the uh saddle seat i suppose anyway oh here we go oh what's no i did try a desktop what's with it saying here's your plugin for your browser it's like i don't want to know i didn't step one step two continue like i don't know i don't want that what browser are you using this thing called chrome and in fact it's an iridium pork so the plugin might not even work i'm just using plain chrome windows isn't it uh let me in without any questions about plugins okay well let's see if it lets me get past it try again okay so oh it shows me into the browser store whatever not i'd no need to for you with that anyway it's been a pleasure and i'll see you bright and early tomorrow morning at the same time and evening to you um uh it's still morning for me just not as early as you did you have any other west coast speakers um i don't know i had my talk and then enough time to eat and then uh moderating for your talk well how did the home lab stop go if it hasn't quite good uh no it was this morning um mickey couldn't make it but we got tom jones to fill in cool uh a couple of bits of overlap about the you know the concept of careful your stuff in your home lab doesn't mix in with the production stuff you know turns out there are services you run in your home lab that suddenly people depend on if you break the app that makes the tv work then uh people will complain at you this is true cool very cool i guess it's clever i mean it's clever i like that absolutely that's clever i'll go with mobile so i can also move around the house and i was like helping my wife load some wax into a thing like during that talk and that's why i'm like good idea young um awesome i miss you i can't implies that enough see you over there cool take care bye everyone thank you for coming i hope you learned something