Super Blade & SAN infiniband

High-availability SAN
High density, high performance twinblades

Category:

People & Blogs

Tags:

• SBB
• supermicro
• infiniband
• twinblade

[suthngin]

I was talking with SuperMicro today and they said you should be able to flash your Infiniband to 10GbE

[TheEkkas]

@suthngin You need a 10Gbps ethernet switch, I do not think it is different firmware, maybe just an update. We have an Infiniband switch, but from the SM website(for the blade module we use):

In addition to this outstanding InfiniBand capability, the AOC-IBH-XQS can be configured alternatively as a 10-Gigabit Ethernet NIC when used with the Supermicro SBM-XEM-002M 10-Gigabit Pass-Through module or the SBM-XEM-X10SM 10Gbps Ethernet switch.

[suthngin]

If you are using Linux to do the RAID what are you using for your SAN software? Are you just presenting NFS shares to VMware? Sorry for all the questions but this is really intresting stuff as we are starting to build our own datacenter.

[TheEkkas]

@suthngin We are currently using XenServer. Started of as NFS shares, but XenServer could not handle high NFS load, or when the SAN was under high load, NFS on the XenServer's started disconnecting VMs. Now we migrated to iSCSI on half the SAN and will change all to iSCSI in January. We might also change all to CentOS6 KVM for our virtual pool.

[suthngin]

Wow ok so that is something I need to research is how the JOBD works with the existing attachement. If you are not using the RAID controller then you are using a Linux software RAID. When the JOBD attaches to the SBB it is an additional 16 drives for a new raid volume or does the Linux box just see it as an additional 16 drives and you all handel the RAID in Linux? This is where I get conflicting answers.

[TheEkkas]

@suthngin I haven't used that model myself yet, but depending on the model, you can configured it either as exposing all drives to the SAS port, or do the RAID itself and expose only 1 (or more) drives.

The SBB has an external SAS 6Gbps port on each motherboard. The JBOD also has 2 SAS ports, so you connect each MB to the JBOD and it can be as if you just added 48 1TB drives, or 1x 24TB (RAID1) drive. But download that JBOD manual and have a look.

[TheEkkas]

@suthngin Yes we use Linux RAID, it is just much more flexible and manageable, the only drawback is that (AFAIK) you cannot use software RAID in HA active/active, and it complicates even active/passive because both nodes RAID (mdadm) is not 'cluster-aware'.

[TheEkkas]

@suthngin Btw, in the end we did not use the SSDs and filled up all slots with 2TB drives. The SSDs only give marginal better performance on large block writes, it is super-fast on small writes, but in using VMs, the blocks tend to be larger and we wanted to maximize storge.

[TheEkkas]

@suthngin Yes, the SuperMicro SBB has built-in raid controller that share all hotswop drives to both motherboards, like a cluster with shared storage in 1 box.

I love it! If you want to load more VMs, I suggest you rather get the JBOD add-on (also shared storage with 2x6Gbps SAS, 1 to each node, up to 48x2.5" HDDs) with many more drives. 16 is not enough. (we use Linux software RAID 10 and not the controller's RAID)

[suthngin]

Thanks for that! That is what i really wanted to know. How would you rate the hardware? Are you using a supermicro RAID controller?