 So I did my rack tour the other day and people had asked the book cards I was using for all my 10 gigabit connectivity because I'm using in both Zen server and free NAS So these are Chelsea Chelsea, oh, I'm not sure exactly how to say that Chell soves 10 gig e3 20 cards. This is an older model card and it works great I've actually bought a handful of these off of eBay one in each of my servers now. I bought dual SFP plus cards now SFP and SFP plus are very similar as in they physically look the same SFP being the one gig SFP module standard and SFP plus is a 10 gig standard There is some interoperability because you can slow this down to one gig on the connection as opposed to 10 gig But all of the equipment we have including the unify 16 g 10 gig switch is going to be all connected at 10 I will note because someone had asked about our unify 24 port switch and that only has SFP not SFP plus on it the one that we have in particular and I think all the unified 24s. You don't get SFP plus on the unify 24 switches, but you do get it on the 48 switches. So if you're going to cover that real quick now We've chose deck tables as an option for connecting these these are direct attached copper twin acts They make cable management fairly easy They're kind of designed for either in rack or you can go if you have an adjacent rack. They're running them over They're different than the fiber I mean the fibers popular if you want to do long runs and you can get a fiber transceiver that fits into SFP and we'll work and I have tested this generic one here and unify sells their own Fiber intercontact so if you need to do a longer run fibers a little bit harder to work with it Doesn't quite bend as well. It does bend really well. Actually, it just will break easier and it can be a little bit more Troublesome to try to set all this up for cable management. So we're going to stick with the deck Now buying dual cards the reason I buy dual cards is if you have in case of Zen servers of em if you have your storage network, which we do plugged in and you then want to have a Virtual interface that also has a 10 gig connection. You could assign the storage here and the virtual interface for your Virtual machines over here that way you don't have any conflicts with the storage When that example would be if you're pulling a heavy I o on your storage And then you would like to serve that over SMB at 10 gig via the virtual machine Let's say a Windows server that's running on here You'd want to have dedicated Nick if not they would share the bandwidth now for our use case We only have a single one in use I don't really have a need right now for my VMs themselves to have a 10 gig interface, but it's there and these cards Like I said are reasonably priced Now the direct-attach copper is pretty easy to install. Whoops. It's just like USB. You'll get it wrong the first time It snaps in and clicks Now a couple options here if you are setting this up in your home lab And you would say you know I just want to have two servers talking to each other No problem. Go ahead and snap two together. You can have two cards Direct-attached to each other and it works perfectly fine You just statically assign the addresses and they can talk to each other without having a switch in between This is a really convenient way to do it, especially when you're doing testing and when I build my test stations here on the desk I do Usually set them up this way It's just really convenient and it's usually the expense of a switch But in our case because we have two free NAS servers and two Zen servers and we have a storage network We built which I'm going to show you here in a second We like to keep them all plugged into a switch because that gives us really fast transfers between anything that we want to do Across that network and isolated. Alright, so let's take a look at the physical layer now on our network And then we'll get into the software. Okay, we'll start here at the front. So here's our unify 16 port 10 gig switch and here is the SFP cables that we have connected there And these go to the back of the server. Now, this is the SFP ports on here But please note there's no plus next to them. So we've connected the RG45 style from port 19 to port 13 and that's only going to connect at 1 gigabit Which is why I don't show some of the camera. That's green So that only shows we have a 1 gig connection and all the white lines So we have a 10 gig connection here. So those lines all come back this way And down and now they're plugged in over here. So here is the 10 gig interconnects that we have on each of these And like I said, we have the dual cards those Chelsea O E320s And it carries on down to each server and like I said, we have a spare if we need to so pretty straightforward And you can see the active light when they're plugged in I know this will be the light for this side over here Now the nice thing about these and we'll get a wider view of my rack here for cable management They do have a decent bend radius So when the door closes it touches these a little bit, but it doesn't exceed the bend radius on these It would be a worry if I had a fiber module and a fiber sticking out and then a bend down All right, we're gonna start here within the unify part. So with unify I love the way that their software makes the layout look exactly like the switch It's just convenient and pretty so we can see that all these are connected at 10 gig via SFP Just like we seen when we're looking at the rack here is the air rectangle case you're wondering what this one is That is also an RJ 45 10 gig and here is that one Purple cable that connects from the unify 24 port to here for the uplink that can bridges this into the rest of the Network and it's only a gigabit because well the 24 ports only a gigabit switch now these being the storage network they're on their own VLAN and by doing that that means we have Done this and a locked down Setup essentially so if we look at it and let's actually go jump over here to the settings networks this over We have our storage VLAN only and It's set at VLAN ID 20 and then we go over here to any port that we edit and we've all Chose this as a profile to be 20 now This network is not defined in my router and the reason why is because I want to keep the Storage network isolated from the rest of the network that way anything in there is locked down to only those servers There's no other interference. There's no routing going back and forth or in and out of that network It is essentially a flat network now by doing that and not having even a DHCP server in there And it does mean we have to statically assign these which is pretty easy to do now the one config change that I did make in the switch itself is under services Enable jumbo frames jumbo frames You're not familiar jumbo frames You can look them up and get way more detail that I'll give you right now But the short of it is the ethernet frames are set to 1500 by default a jumbo frame carries 9,000 bytes of payload But you do have to have a switch that supports it and it does add some speed to there because for each packet You're able to get more data squeezed into it. So if you look at the frame type here's a standard at 1500 here's nine and Overall, you can see the payload side total transmitted efficiency is a little bit more in real world performance Yes, you get a little bit of gain on here and it varies by protocol But generally speaking if you're on a network that supports it, it will give you a little bit of enhancement on there There's plenty of debate that I'm not going to get into about how much enhancement in the real world testing for it It's easy to do. It doesn't have a cost to it There's not really a downside as long as your switch supports it So we have any enabled jumbo frames is the only real change that we've done on the switch outside of defaults So once that's enabled now, we're going to go to how we have these all statically assigned now By the way, because these are assigned only that VLAN I don't have to tag anything within the network interface itself So you don't when you're configuring these in free nas or in Zen server You don't have to make any changes or add any VLAN tags to these because it looks as if it's one switch when you do that together So those are all focused only on that VLAN. No other VLANs Information is passed across. So we have here 192.168.1010 on this free nas server And let's go ahead and look at some of the interface Exactly and we're going to go ahead and edit 192.168.1010 I set them to be slash 24 I've covered this before and someone says well, you're wasting IP space Why I don't know I troll say the things they say anyways You can set it to whatever you want. The only thing on here are these four servers So you could make it the net mass smaller Habit is slash 24 in case I add a couple more devices to the storage network It's easy enough to add them to you just got to remember. I'm statically assigning all of these Then MTU 9000. This is how you pass that option on to get the jumbo frame So I set this to MTU 9000 inside of free nas And we'll look at the new free nas Same thing 10 gig. We're gonna go here and edit 10 gig. Here's a static assignment slash 24 They do all have to match in that mass So whatever net mass you go with just make them all the same across all the machines you're statically assigning and MTU 9000 Now Zen Orchestra is a way to do this when you're looking at the Zen server. There's just a couple options here So we're gonna look at network And I've got it passed over as storage network MTU 9000 but I'm gonna show you a different way because the way I find it a little bit easier I love Zen Orchestra when it comes to networking if you're not clear on how to use it There's another way you can assign these so if you open up the XCPNG center you can see all your networks and then you go down here to configure And this is how you can configure your storage network as an extra network So it's nice and separated here if you're doing it in the XCPNC And I want to go back to Zen Orchestra so I can show you how you do that But if you notice when I'm assigning these this is 10 dot 15 But has no gateway because like I said, we don't want any routing going in there Now on our side note when you're doing this once you've done this if you want to and we're gonna head and Properties of this You notice how the MTU is grayed out the MTU is grayed out because I have it assigned down here This question has come up before You want to set the MTU before it because once this has been assigned to the storage network and is Using it as a storage network Then it doesn't allow you to edit the MTU just kind of a side note But if you're wondering about that that's something that's important to note and the same thing we're going down here as an effort to Networking same thing storage network. This one's 10 dot 20 So 10 dot 15 10 dot 20 and it allows you then to do the same things configure No gateway and all these of course can now see each other and when you're connecting to things like the Free NAS storage You do it by IP you just type into 192.168 10 dot and then go from there Now back over to Zen Orchestra. So if you go over to your hosts and go to network You see all the networks and you can add a network But where you want to go when you're setting up the networks is you go to pools pool network This is where you can create like the bonded networks and you'll see the options are very slightly different This is what this is where you can set the MTU's and configure Some of the things so you can add This you can kind of get the idea that they're a little bit different I guess at the Zen Center seems to be a little bit easier to read when you're building a storage dedicated network Versus in here because there's two spots you can go for networking just wanted to leave that note in there So you have two options in whichever way you choose to configure it. You can Easily change it in either one after effect inside of here But only when the storage networks disconnected can you change it in terms of the MTU like I said easy enough to pass the MTU setting Now one of the cool things is once it's on here, and I grab this little demo And we have the disc is Dozer right here, and that is our free nest this one here over here to the pools to show you what it looks like storage pools These are just a bunch of spinning drives a bunch of older To two terabyte sass drives, but they're reasonably fast And I'm bringing this up and showing you how they're connected. So here it is NVMs These are connected via NFS on the 10 gig network. So what kind of speed do you get out of this? We'll just run this quick speed test Not the most accurate I'm not getting real in-depth here, but over the 10 gig network it's reasonably fast I mean we're getting a right speed of about 586 megs now what command that I would do to run this speed test SSH. It's this right here. It's Time SH-C writes a bunch of zeros test file 1.9 gigs count 1000 and it sinks the file system and then it removes the file and that gives us like I said a speed of 586 I've done some of the benchmarks on this. It's it's reasonably fast. It's not quite as fast as a Samsung evil SSD, but the speeds are pretty reasonable on it over the 10 gig network It's it's not bad and of course it goes faster if I forget some Money to throw at a storage array that is all flashed would of course increase that much because we do have plenty of speed going between these One other thing that's kind of neat is when you do a migration using Zen Orchestra. Zen Orchestra is aware So we're gonna choose it's on Xenoper and we're gonna find Xenoper 2 the other one was sent over to local storage It sees not only the pool wide area ETH 1 so it says hey these servers talk to each other in ETH 1 It realizes they also can talk across a link that I just named 10 gig storage So it's the storage network, but instead of storage this because the Zen servers are on that network They can talk to each other and this will allow me to migrate VMs It only takes a couple of minutes to migrate even a live one because they're talking to each other at 10 gig a bit through The switch so that's another advantage you get is not only that you have fast access to the storage Repositories that are stored on a 10 gig network the 10 gig interconnects between the Zen servers allow me to easily pass them right over At full speed now last thing I'll show you is you know people want to know is the speed test between the servers And we'll cover that real quick here So we'll go ahead and log into one of the free NAS boxes And we're gonna run iperf dash s for server dash B capital B 192 168 10 dot 8 and what this does is binds it to that IP address So the servers do talk on other networks. We want to make sure they're only talking across the 10 gig network This is T-Mux for those wondering That's how I split the screen now we logged into Xenifer and we're gonna do iperf 3 client 192 1 6 8 10 Dot 8 and we can see that they're able to transfer at Quite fast at 9.9 gigs a second So we're saturating the network Between these two and each send server can talk to the other Zen server at this speed and each send server can talk to the Free NAS at this speed or each of the free NASs because we have two free NASs and the two Zen servers on there So I'm able to move data around quite fast through this switch at 10 gig And of course if I because these are all dual cards if I ever wanted to have the VMs Have an extra network interface Go back over here to hosts Networking network. I labeled them all as not in use, but this is the e4 is storage 10g This not in use is actually the not plugged in right now Other network interface that is available on there So if I ever wanted to have the VMs assigned to it, it's pretty easy to do so hopefully it clears up Or gives some insight into how the 10 gigabit network is connected for my network And how to set it up because it's pretty straightforward. These DAC cables are reasonably priced I'll leave links where you can buy these from you can get these cards I'll put the model of the card in there that way it stays relevant I can't really link directly to an eBay link because well, they're only relevant for short periods of time But as long as you know the model of the card I've had no compatibility problems with running these in free NAS or Zen server and that includes in the free NAS 11 One and in free NAS 11-2. They're all the same card in all of these now Well, you can mix and match cards and I have tested it with a couple other cards that I have and I didn't see any Difference in speed or performance on there when I was doing kind of some lab setup. These cards happen to be both low profile and Full height so you can get them either one with different brackets So please note when you're getting them get what matches your server, but the nice thing is the card itself does come in different formats Thanks for watching if you liked this video go ahead and click the thumbs up Leave us some feedback below to let us know any details what you like and didn't like as well because we love hearing the feedback Or if you just want to say thanks leave a comment If you wanted to be notified of new videos as they come out go ahead and subscribe and the bell icon that lets YouTube know That you're interested in notifications. Hopefully they send them as we've learned with YouTube Anyways, if you want to contract us for consulting services You go ahead and hit launch systems calm and you can reach out to us for all the projects that we can do and help you We work with a lot of small businesses IT companies even some large companies and you can farm different work out to us Or just hire us as a consultant to help design your network Also, if you want to help the channel in other ways we have a patreon We have affiliate links. You'll find them in the description You'll also find recommendations to other affiliate links and things you can sign up for on Lawrence systems calm once again Thanks for watching and I'll see you in the next video