 So the majority of computer systems are running gigabit over Ethernet. It's really popular. It's what's out there It's been around for a while but so has 10 gigabit and 10 gigabit in the standard specifically that we're using and I pulled up the Wikipedia page because there's even more standards and I was even aware of The one we're going to be talking about today is SFP plus direct connect copper And this is actually standard spend around for 2006 now the good news about something that's been around since 2006 There's a lot of it on ebay There's a lot of used cards that are 10 gigabit that are well very affordable to build Your connection like we did between our citric sen server and our free nas box and we wanted you know 10 gigabits now There's other ways to achieve high speed connections between free nas such as bonding together several network interfaces but the 10 gigabit is a really affordable way to do it and I like even labeled in the Wikipedia article here. It says cheap low latency and low power which definitely defines it. So it's both affordable Easy and really low latency to do this. So I'm going to show you first what the parts we bought are This is the twin x cabling Uh type sfp plus direct attach copper And I'll leave all the links below to this. We can look at the wikipedia page But where you can get this so we chose the uh chelsea net app dual port sfp plus 10 gig Connector this card is all of $22 on amazon and I ordered a couple of them It is both supported in zen server and free nas It's been actually specifically which you look these up as free nas based on free bsd So you look up the free bsd support this card spent in there a while Someone made a comment before that this wasn't supported you go ahead and look it up I actually bought a little while ago But I've been busy and haven't had a chance to install it and set things up There's a whole series of things that had to occur before we got to this point So we ordered a pair of these now It has two which is really nice because that means if I wanted to connect this to two separate free nas boxes I could or have the free nas box connect to something else but we're only going to use one of the ports for this demonstration and Kind of give you an idea of how we set it up now. This is the cable. We bought this is what they look like They have little tiny connectors on them Uh that slide in a clip in really easy and you pull this little piece here to Pull the clip out. I don't have it physically in my hand because I got it plugged in and working I didn't feel like shutting down my virtual machines just so I can show you this But these are common you see in between switches and things like that as well But they will work without a switch. That was something I wasn't real clear about before but after doing some testing That's part of the reason we bought this and now I'm doing the video to let people know you can just buy a pair of these You can set them up and Direct connect them between there now. Let me show you how the direct connection is set up I have a diagram I drew so I have my free NAS box and I have my citrix n server There is a series of network ports on this multiple cards multiple cards in the zen server Some of it goes over here to a this network with some vlands And then this network which is for some other the virtual machines that I want a completely physically separate network Not just vland off. So that's all that but No switch direct cable sfv plus the cable I showed you the twin x sfv plus cable Is plugged directly one end into the free nas and one end to the zen server Now because pf sense is providing all the ip ranges and routing for everything that's going through the switch parts of this We have to then assign ip addresses. So let me just kind of walk you through some of the setup for that So we go over here to the free nas box and he's sure wondering because this will come up This is an intel i5 processor the i5 45 70 at 3.2 gigahertz with 16 gigs of ram And we're going to talk about the performance you get with this and where some of the bottlenecks still are Now the network setup interfaces right here is the Interface uh, that's the 10 gigabit one active and what we had to do is sign it a static ip address Which was 192 168 10 dot 10 i figure 10 gigabit put it in the 10 network Here's the two other networks i mentioned that are standard ones in there now There's no gateway on this one other thing we did when you're going to direct connections So you can get the most speed lowest latency is enable the jumbo frames Uh, I tried a couple different settings the default 1500 4500 and then 9 000 and the 9 000 works quite well For giving me the least amount of latency what a jumbo frame is allowing per each frame more data within it Essentially, i'm not going to get into all the details on that but i when you're tuning these You have to have everything matching on both ends So we both put the mtu 9 000 setting on this one and then we put this on the zen server So this one's a sign 10 dot 10 It's going to The i scuzzy is where it's bonded and it's running on a raid z2 Which also will come up because people don't ask about performance. So let's switch over to look at how it's done in the zen server So here's the zen server Here's the networks Oops, i'm sorry networking And it's referred to as a storage now, you notice the lack of a gateway. So this is 192 168 10 dot 15 I know it's a little small. I don't really have an easy way to zoom that but this is dot 10 at 15 So it's directly connected now. I made the net mask Uh slash 24 in case I ever want to add more to it. I could have narrowed it down to just two ip's It's not overly relevant because there's nothing else on that network Matter of fact because they're directly attached with no switching between like I showed you in the diagram It is a direct absolute connection to it. So it doesn't really matter what other ip's are in there But not having a gateway also means nothing tries to route over that network So it's by itself and nothing's ever going to go there other than things calling for that And the only thing tied to that is the storage. So when you're setting up your storage repositories And we'll jump over to one of them here. We have this free nas one here storage And you can see that the iSCSI LUN is that there so when you're adding the storage just like I showed you in the other video You put in the iSCSI but you just put in the ip address of it routed over that now back to how that goes in here Services And under portals you say which ip address is it's bind to I have it binded to the 10.10 and the 2.7 The 2.7 is so I can actually do some iSCSI targets over standard gigabit while I was doing some testing I don't really using it at the moment, but then you bind it to here 10.10 As I talked about in my other video for setting up citric send server with an iSCSI back end with free nas Is you need to identify and implicitly list in the portals which ip's it's going to bind to it Doesn't just automatically like some of the other services and free nas bind to all of them at once So when you're going through the iSCSI setup you go to and set up which ip's is going to be attached to so we attach it to that one and Then when we set up the citric send server We've attached it to that same ip address which automatically because that's the only thing that can route over that network automatically goes across the Gigabit because it's a non-routed and it realizes that's where it can go and the dot 10 network doesn't exist anywhere else So it can't try to guess the wrong things all right Now this is where we're going to talk a little bit about the performance and reason i mentioned that it's raid z2 This is an amazing guide. Someone did it's uh zfs raid performance capacity integrity comparing space speed Uh and safety per raid type and they basically took the same drives and benchmarked them in amazing different configurations So they took 24 drives and gave you each read write performance So I have four drives raid z2 and that's where my performance level is now bring this up because 10 gigabit is faster than The drives themselves Now there are ways if you're really in this for performance getting some drives and setting them up in this configuration guide Will give you the most effective way to set these up On there for the most speed. So that's why I'll leave a link here So if you're trying to design it and you have a performance concern This will show you the fastest performance you can do With the right amount of drives and how you can get to that So that being said this is with our free nas box Let me show you what the benchmarks look like and the load on free nas That's why I talked about the specs a lot here. So let's jump over to here And this is the disc performance we're getting It's actually pretty impressive. So this is the pass mark Uh software i'm running windows in here and windows runs quite well. So actually, uh, I haven't really had any problems with it It's windows 10. So it's you know, pretty grindy for her drives as far as i'm concerned windows 10 doesn't feel the most efficient to me But uh launching applications Not really a big deal. This actually pauses on the preferences for some of the plugins Um, but it seems to launch reasonably fast. I personally purposely opened this something completely uncashed Because I've been running the disc mark utilities quite a few times. So I could uh get that and give us a second Yeah pauses on some of the plugins I have and then it opens up So and this is over an rdp connection So you can see that i'm reasonable, uh disc performance I'm getting out of this across the time gigabit over to there because once you raid them together Their the drives are actually a little faster and this scores and we pull up the pass mark scores and bring them over to screen here I know they're a little small. Let me zoom in here. So this is at, uh 4,445 is the disc mark using the Passmark test and we'll zoom in here and that actually puts me up here with some of the single ssd performance So not horrible Uh pretty reasonable and of course that's enough for me I know you can get even better disc performance if I were to put this for example on an ssd array These are all spinning 7200 p.m drives in my free nasbox They're the hitachi series ones the i'll find the model and leave it in the link in the description But yeah, they're not anything super high performance. These are not like your 15 000 k drives or anything like that or they're not ssds So i'm actually getting some pretty reasonable performance using the gigabit using a raid z2 And the machine the virtual machine is running on eric will completely very usable and without really any issues here So let's talk about though what this looks like when we run this. So let's go back to reporting Here's where I ran the disc tests on here and you can see that we almost hit 60 cpu usage on running them Almost 60 on an i5 so pretty much not much now. This is also everything else running on here at the same time I still have all my Primary virtual machines running on here. I still have my nvrs running on this the well using the drive for it Which is another virtual machine that's tied over to here and it's working quite well It barely taxes the system and that's at full disc read doing something real intensive Just for the purpose of benchmarking and testing And you can see like here's the ice cozy tests where we ran them here you can see where it peaks out and seeing like 600 Megs and not not bad performance. So it's definitely Taxing it and it works quite well And system processes Let me look at network performance You can see where the network spikes came through because it's jumping across that network the gigabit network So there's our little spikes for some of the tests we did It doesn't really have much effect on the memory on this We're seeing just a little bit of change in here Most of the memory in free nas in the system gets put over to arc or you know some of the caching for free nas And there's all the disc ios and the spikes you've seen for the rewrite tests that we're doing with that So it's really not that hard on the system But let's get back to some of the design considerations when you're using ice cozy in free nas Now this is part of the free nas 11 documentation It says for performance reasons to avoid excessive fragmentation Keep the use space below pool below 50% when using ice cozy the capacity Of an existing extent can be increased by growing your lungs And this is just part of the documentation. This is a little bit of a design Problem so to speak with the way zfs works now This is where those are located. This is the zen storage and because the way ice cozy works On top of zfs it set up as just an individual file that file is One contiguous file as far as the system is concerned as in citrix But zfs is a copy on right file system So every time he right is committed An entirely new copies moved and it kind of shifts that over There's some technical details you can read about exactly how that works But you can see immediately where there might be a problem with that where you don't have enough storage To create another copy. Well, then it won't do that And then you start running into fragmentation problems because it's got to write parts of it different into the file system So you can run into Overall slower performance because there's not enough space to for zfs to do its magic in the copy on right but The thing that's interesting to me about this is zen doesn't support thin provisioning But that means you can't thinly provision or Compress the drives and only allocate what's needed. So if you have a one terabyte Allocation it wants to allocate one terabyte even though you may only be using inside the virtual machine physically like, you know 10 gigs of storage. So this is where zfs works some magic The files are bigger as far as they're reported to in zen zen thinks of this as a three terabyte Uh load and we now have two of them on here Because zfs works with compression. It only is using 198 gigs or because these drives I have 1.7 terabytes here. It's only using at a 4% usage right here So we're not using a whole lot of the 1.7 terabyte available space I'm sorry not 1.7 4.3 terabyte space. So we have 4.3 terabytes, but only on the zen storage We only have This small amount used and let's look inside on how zen looks at this is where it gets kind of cool So you see the disperformance and yeah, we'll be disconnected and when we look at the storage I moved over to The zen orchestra because it gives me a better view of the storage. We look at the storage. We have a 2.3 terabyte 2.47 terabyte today 546 gigs used But this is only one piece of it that windows isn't even on this one This is just some of the other drives I have and as you can see it's showing based on the entire extent of the drive So let's go to the other storage pools And this is a second one I have called free nas lab And it's a 1 terabyte with another 285 gig you so I have 700 to 285 plus the 546 That's 831 gigs used yet free nas Only sees 198 gig you so this is where the whole magic happens with free nas compressions EFS If you're not then provisioning, you're just filling the drive with zeros So free nas goes no problem. I see a bunch of zeros. That's really easily compressed and then brings it all down So both of these extents for the two I have and like I said the one via you can notice I have one called lab and one called Just free nas so lab is where I'm doing some of my testing with a bunch of other random virtual machines But you can see that there's so much data used According to zen, but then free nas goes I got this and compresses all of it So it's actually not using as much So this makes storage that much trickier so to speak and thinking about it because it's not just allocation Because this is going to work some magic and compress a lot of it. You actually don't go Oh, no, I'm only going to be able to create a storage extent. That's this small You're going to be able to create something bigger not bigger than the entire drive But you're going to use it more efficiently because free nas will take care of The compression that and still not compromise performance. So yes, the 50% rule does apply You do not want it to exceed that But do that knowledge based on what you see in free nas not what you see So speak in zen orchestra or in zen server in general When you're looking at the size of the drives And this is like I said important when when you're looking at the design of it going, okay I do have enough room to build out these virtual machines with larger hard drives Even those are empty and not thin provision. You still have the space to do it And that's kind of what I wanted to talk about as far as this we first we can build the 10 gigabit really cheap We can use it across ice guzzy and before people start slamming me and go You can't use it on free nas because of the 50% problem and it wastes a bunch of storage Yes, you can and free nas will take care of that even though it's not thin provision So I just wanted to share that with you. This is how we're we have things set up on our virtual machines with our Citrix Zen servers is more or less kind of production kind of lab It's a little bit of both for me for all the stuff I do But it's working wonderfully and also I'll show you real quick here This is the performance reports of what most of the stuff looks like for us for let's go to a one day view So it's actually way overkill for our machine. So We barely use any cpu on this unless I'm doing testing. This is my testing. I was doing this morning You can see that it's very very low usage So it's nice because when I do want to do some testing these things open up really fast matter of fact Even creating snapshots and jumping around for the virtual servers It's nice how fast this works and this is really affordable. I mean you look at these parts on amazon and you know 30 under 30 dollars for a 10 gigabit card along with the twin x cable Being 17 dollars. So basically for under a hundred dollars You can get 10 gigabit connectivity between two adjacent servers or you could even do this for your own computer And I've seen people do this. You want really high speed In the awesomeness of freenias you can put this in your Computer plug it in directly and have your freenias box next to your computer and enjoy The raid z3 even on a windows box chair and it'll work over that as well So that's about it. I hand for this The only thing I did realize and I kind of found a problem and I I got to work on this to make it a bug report I found out that if you both use this for ice guzzy And nfs at the same time on the same network and you try to do fully saturated read rights from both that Freenascos no and just turns off the network interface. I've tried this on a couple different network interfaces both gigabit and a tank gigabit And then the problem occurred on both I'm trying to create a test scenario because it was bad when I did it because I was trying to back up virtual machines across Both at the same time and created a problem So I got to create a little temple for this and talk to the freenias people to say I can make this problem repeat and it's kind of interesting. So If you're doing it for ice guzzy, you probably want to dedicate it for ice guzzy if you decide to go with nfs dedicated to nfs and I'd mentioned before just so you're wondering ice guzzy with zen works wonderful With freenias freenias does not work wonderful with zen over nfs There's apparently some sinking issues and the performance isn't quite there Much better performance over ice guzzy in case that question comes up. All right Once again, hopefully this was helpful and taught you a little bit about how 10 gigabit works and how Zen can perform on it and how you can put together a 10 gigabit network for under 100 dollars For the upgrade and pop these cards in oh in case you wanted this is only an 8x slot so they're not really Most computers have this like I said this card is old as well It's been around for it's been around for a little while and you may even find it cheaper somewhere else I'll just throw in the amazon links because they're in stock right now. If not, you can use the part numbers to You know find one for yourself. All right If you like to kind of hear like and subscribe and if you have more questions about 10 gigabit Or if you think you think I didn't cover something so I need to do a follow-up video. Let me know