@suthngin You need a 10Gbps ethernet switch, I do not think it is different firmware, maybe just an update. We have an Infiniband switch, but from the SM website(for the blade module we use):
In addition to this outstanding InfiniBand capability, the AOC-IBH-XQS can be configured alternatively as a 10-Gigabit Ethernet NIC when used with the Supermicro SBM-XEM-002M 10-Gigabit Pass-Through module or the SBM-XEM-X10SM 10Gbps Ethernet switch.
If you are using Linux to do the RAID what are you using for your SAN software? Are you just presenting NFS shares to VMware? Sorry for all the questions but this is really intresting stuff as we are starting to build our own datacenter.
@suthngin We are currently using XenServer. Started of as NFS shares, but XenServer could not handle high NFS load, or when the SAN was under high load, NFS on the XenServer's started disconnecting VMs. Now we migrated to iSCSI on half the SAN and will change all to iSCSI in January. We might also change all to CentOS6 KVM for our virtual pool.
Wow ok so that is something I need to research is how the JOBD works with the existing attachement. If you are not using the RAID controller then you are using a Linux software RAID. When the JOBD attaches to the SBB it is an additional 16 drives for a new raid volume or does the Linux box just see it as an additional 16 drives and you all handel the RAID in Linux? This is where I get conflicting answers.
@suthngin I haven't used that model myself yet, but depending on the model, you can configured it either as exposing all drives to the SAS port, or do the RAID itself and expose only 1 (or more) drives.
The SBB has an external SAS 6Gbps port on each motherboard. The JBOD also has 2 SAS ports, so you connect each MB to the JBOD and it can be as if you just added 48 1TB drives, or 1x 24TB (RAID1) drive. But download that JBOD manual and have a look.
@suthngin Yes we use Linux RAID, it is just much more flexible and manageable, the only drawback is that (AFAIK) you cannot use software RAID in HA active/active, and it complicates even active/passive because both nodes RAID (mdadm) is not 'cluster-aware'.
@suthngin Yes, the SuperMicro SBB has built-in raid controller that share all hotswop drives to both motherboards, like a cluster with shared storage in 1 box.
I love it! If you want to load more VMs, I suggest you rather get the JBOD add-on (also shared storage with 2x6Gbps SAS, 1 to each node, up to 48x2.5" HDDs) with many more drives. 16 is not enough. (we use Linux software RAID 10 and not the controller's RAID)
@suthngin Btw, in the end we did not use the SSDs and filled up all slots with 2TB drives. The SSDs only give marginal better performance on large block writes, it is super-fast on small writes, but in using VMs, the blocks tend to be larger and we wanted to maximize storge.
It is 2TB Seagate SAS drives. At the moment it is running about 30 VMs but I think it has reached it limit as heavy work will slow the notably SAN down. We tried Nexenta software, mainly due to the HA active/active plugin but had some issues with Infiniband so now we run CentOS 5.7 in active/passive as GFS2 filesystem needed for active/active is slower than EXT4. We can get about 450MB/s (3.6Gbps) but again some issues with Infiniband cause it to be effectively about 2.5Gbps to Virtual hosts.
Your video said that you were using 2TB drives. What SAN software did you use and what sort of performance are you getting with SATA drives? How many VMs are you able to run? GREAT Video!!!
@suthngin Stay away from Infiniband! Rather get 10Gbps Ethernet as you cannot use Infiniband in VMs (it cannot be added to a Linux bridge without expensive BridgeX hardware). 10Gbps ethernet would be much better and you can then use SR-IOV to give your VMs 10Gbps as well. With Infiniband without BridgeX, they can only get the standard 2Gbps virtual adapter to a Gbps ethernet network. Only the Hosts can use Infiniband to e.g. the SAN, but 10Gbps is enough in any case.
I was talking with SuperMicro today and they said you should be able to flash your Infiniband to 10GbE
suthngin 2 months ago
@suthngin You need a 10Gbps ethernet switch, I do not think it is different firmware, maybe just an update. We have an Infiniband switch, but from the SM website(for the blade module we use):
In addition to this outstanding InfiniBand capability, the AOC-IBH-XQS can be configured alternatively as a 10-Gigabit Ethernet NIC when used with the Supermicro SBM-XEM-002M 10-Gigabit Pass-Through module or the SBM-XEM-X10SM 10Gbps Ethernet switch.
TheEkkas 2 months ago
If you are using Linux to do the RAID what are you using for your SAN software? Are you just presenting NFS shares to VMware? Sorry for all the questions but this is really intresting stuff as we are starting to build our own datacenter.
suthngin 2 months ago
@suthngin We are currently using XenServer. Started of as NFS shares, but XenServer could not handle high NFS load, or when the SAN was under high load, NFS on the XenServer's started disconnecting VMs. Now we migrated to iSCSI on half the SAN and will change all to iSCSI in January. We might also change all to CentOS6 KVM for our virtual pool.
TheEkkas 2 months ago
Wow ok so that is something I need to research is how the JOBD works with the existing attachement. If you are not using the RAID controller then you are using a Linux software RAID. When the JOBD attaches to the SBB it is an additional 16 drives for a new raid volume or does the Linux box just see it as an additional 16 drives and you all handel the RAID in Linux? This is where I get conflicting answers.
suthngin 2 months ago
@suthngin I haven't used that model myself yet, but depending on the model, you can configured it either as exposing all drives to the SAS port, or do the RAID itself and expose only 1 (or more) drives.
The SBB has an external SAS 6Gbps port on each motherboard. The JBOD also has 2 SAS ports, so you connect each MB to the JBOD and it can be as if you just added 48 1TB drives, or 1x 24TB (RAID1) drive. But download that JBOD manual and have a look.
TheEkkas 2 months ago
@suthngin Yes we use Linux RAID, it is just much more flexible and manageable, the only drawback is that (AFAIK) you cannot use software RAID in HA active/active, and it complicates even active/passive because both nodes RAID (mdadm) is not 'cluster-aware'.
TheEkkas 2 months ago
Thanks for that! That is what i really wanted to know. How would you rate the hardware? Are you using a supermicro RAID controller?
suthngin 2 months ago
@suthngin Yes, the SuperMicro SBB has built-in raid controller that share all hotswop drives to both motherboards, like a cluster with shared storage in 1 box.
I love it! If you want to load more VMs, I suggest you rather get the JBOD add-on (also shared storage with 2x6Gbps SAS, 1 to each node, up to 48x2.5" HDDs) with many more drives. 16 is not enough. (we use Linux software RAID 10 and not the controller's RAID)
TheEkkas 2 months ago
@suthngin Btw, in the end we did not use the SSDs and filled up all slots with 2TB drives. The SSDs only give marginal better performance on large block writes, it is super-fast on small writes, but in using VMs, the blocks tend to be larger and we wanted to maximize storge.
TheEkkas 2 months ago
It is 2TB Seagate SAS drives. At the moment it is running about 30 VMs but I think it has reached it limit as heavy work will slow the notably SAN down. We tried Nexenta software, mainly due to the HA active/active plugin but had some issues with Infiniband so now we run CentOS 5.7 in active/passive as GFS2 filesystem needed for active/active is slower than EXT4. We can get about 450MB/s (3.6Gbps) but again some issues with Infiniband cause it to be effectively about 2.5Gbps to Virtual hosts.
TheEkkas 2 months ago
Your video said that you were using 2TB drives. What SAN software did you use and what sort of performance are you getting with SATA drives? How many VMs are you able to run? GREAT Video!!!
suthngin 2 months ago
@suthngin Stay away from Infiniband! Rather get 10Gbps Ethernet as you cannot use Infiniband in VMs (it cannot be added to a Linux bridge without expensive BridgeX hardware). 10Gbps ethernet would be much better and you can then use SR-IOV to give your VMs 10Gbps as well. With Infiniband without BridgeX, they can only get the standard 2Gbps virtual adapter to a Gbps ethernet network. Only the Hosts can use Infiniband to e.g. the SAN, but 10Gbps is enough in any case.
TheEkkas 2 months ago