 Tom here from Learn Systems and TrueNAS 12.0 U1.1 hotfix has been released and I do recommend you upgrade So if you don't want to watch for us this video and you're just here to wonder if it's a good upgrade I would say yes. Now it is a data corruption problem in ZFS Which is sounds really serious and I would say it is the good news is though it has been fixed and also it doesn't really Happen that often it's a pretty obscure case. It's a pretty rare condition It takes a perfect set of circumstances in order for this to happen. So, you know in any of my testing I was even Unable to produce it and I did a little bit of load testing on some basic hardware But i'm not denying that it happens because they were able to finally reproduce the problem and I want to talk about some of that methodology and process that went into this and a quick clarification This appears to only affect the open ZFS 2.0 that was released with the TrueNAS 12 series So if you are running an older version, um, this is not something that's previously a problem in ZFS This is a problem that was introduced very specifically with a synchronous copy on right So ZFS is a copy on right file system and enhancements were made to make it a synchronous copy on right that would allow for better performance and This was done all the way back in march of 2020 and submitted to the beta code base and then into release base and Very very few people found this problem. Uh, it took like I said a special set of circumstances But it has been addressed and is still being addressed because they turned it off as the short-term solution Uh, the long-term solutions can be some fine-tuning under certain circumstances that do cause this but Before we dive into all those details, let's first feel like to learn more about me and my company Head over to laurance systems.com if you like to hire short project There's a hires button right at the top if you like to help keep this channel Sponsor-free and thank you to everyone who already has there is a join button here for youtube and a patreon page Your support is greatly appreciated if you're looking for deals or discounts on products and services We offer on this channel check out the affiliate links down below They're in the description of all of our videos including a link to our shirt store We have a wide variety of shirts that we sell and new designs come out Well, randomly so check back frequently and finally our forums forums.laurancesystems.com is where you can have a more in-depth Discussion about this video and other tech topics you've seen on this channel now back to our content Now here's the release page from junas with the update and all the details of it And links to the different, you know side notes and of what they actually changed Now one of the things they said right here. It just be very careful This is a minor version release to work around a critical zfs issue that could impact systems with a rare But serious issue with data integrity under very uncommon access paradigms It took a lot to find this problem And that's one of the things that's important and I still think you should patch But it may not affect you and but if you're worried about it and you're already running new one There's not a reason not to patch because there's not other changes involved That's just a minor change that turns off the copy on right system And now that's listed out here. This is the march 13th when this code was added And this is the january 14th two days ago from right now because it's january 16th when the code was reverted back But the thing I want to talk about is The way these tickets are created and the way you want to get community support Is not by saying things like free nas now true nas is no longer stable That doesn't sound like a silent zfs corruption Of course that is the details that were in there and people like this over on reddit Where this also blew up and people were messaging me as well And that's why I'm addressing this person said christ this could be the death knell for ag systems You know telling people to be nice on the internet. I know it's not an easy thing And people seem to really pile on when there's doom and gloom if a company has a flaw the number of people that will jump on to Reveal in the commiserate together on a flaw from a company and predict the doom and gloom of everything related to that company is amazing But it doesn't help you get tickets solved. I suggest when people want something solved You don't start out with saying how your product sucks Then people kind of don't really want to help you if you start out with a Really solid Here's all the things I did. Here's all the steps I tried I'm experiencing this problem under these circumstances technical people Especially people who take a lot of pride in your product go how can we help you because we didn't see this in our system But clearly your system is different than ours. So let's dive into helping and you know That's not how this started, but it is how it did work out in the end I just want to bring this up because it comes up in my forums and things like that too I get told my videos were completely wrong because someone will miss a step and they always start with I did everything you said and it doesn't work. You're just a terrible person on youtube That's not the people that I jump and help as much But I'm not going to rant about that. We will talk about what happened and is Pages and pages of details here. This is something impressive to me about ix systems Their team does care greatly about the quality of their product And even though this issue is clearly it took very specific circumstances to recreate They did work very heavily. I mean, there's a lot of comments in this ticket To get it resolved and people setting up and there's more than one person in here that chimed in that Seen the issue as well and they started reproducing those circumstances and saying all right, I can reproduce it here They everyone started turning debugging on to really dive into all the details and essentially what happens is When you go through the system and get silent data corruption, it's the worst type zfs protects Really well against this type of corruption But what silent data corruption is is one of the underlying system that being zfs doesn't tell you there's any corruption It's when you go to get the files and because these vms were stored So you have a hypervisor such as vmware and it's using free nas to nas as your storage server And it seems to work over nfs or ice cuz either Corruption problem did not seem to be transport dependent And you take those and those virtual machines represent the hard drives that are stored over there The silent data corruption is all of a sudden you boot up your machine and it wants to check the integrity of the disc But you look over at true nas and it doesn't say there's any problems You're like, oh, maybe it was shut down or properly or maybe something did happen you know file systems have integrity checking because things can happen and This started happening more and more and it's not like it happened every time It appears to be based on all the reading in here when there are certain conditions met for High loads and doing a lot of coalescing with vmware At least someone had mentioned that was when they were able to consistently do it And like I said, it took a long time for the testing to his first week It has to be become a reproducible problem because you don't want to make any changes to the underlying zfs system Until you can reproduce the problem Then they can start doing it and eventually that's where he came to the conclusion That these and this was all the way at yesterday. This is the original ticket poster who is happy at the outcome Thank you for the feedback everyone for the assistance matter We greatly appreciate it the work around to disable a synchronous copy on right has been committed and will release in 12.0 u1.1 Which is going to be pushing tonight for those watching closely this process may take six hours and you know There's some of how long it takes to push a commit through the system before Everything gets there an update. So by the time you're watching this video You'll probably have an update notice on your true nasty system and they fixed it This is what the good thing about this whole process is and read through there And you can see once we got past the fact that they just called it an unstable system and got into the details That's when problems get solved and this is you know an issue It's certainly not an issue. I was able to necessarily see on my systems, but I don't have vmware to test this with I didn't see it I'm me running a bunch of workloads. I bring it up because I was recently running a bunch of foronics benchmarks For some future videos. I have to test out some storage servers Using different hypervisor testing and I did not see any corruption doing that and I do at least create a handful Of snapshots that have to call us back together because well It's something I was testing to get some iops numbers out of there is what happens when you do this or what happens when you do that No data corruption, but I'm not going to be dismissive this if someone says they do have data corruption You know, I just need to know the parameters in a way to reproduce it because I didn't see it here And that's all I have to say about this. Uh, go ahead and patch your systems be kinder on the internet I know someone will say something not nice down below because It's the internet and sometimes it happens and uh, it doesn't mean I'll stop I'm still very positive because if you look at the big picture of things everything does get better over time Um, there's always those naysayers who think it's the end of a company because there's a minor fix But they do take it serious over at ag systems So I do applaud their solid work in pushing a fast update out once the problem was identified and articulated well Thanks And thank you for making it to the end of the video If you like this video, please give it a thumbs up If you'd like to see more content from the channel hit the subscribe button and hit the bell icon If you like youtube to notify you when new videos come out If you'd like to hire us head over to laurance systems.com fill out our contact page And let us know what we can help you with and what projects you'd like us to work together on If you want to carry on the discussion head over to forums.laurancesystems.com Or we can carry on the discussion about this video other videos or other tech topics in general Even suggestions for new videos. They're accepted right there on our forums, which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you and once again, thanks for watching and see you next time