Static drive mapping using Open Compute Windmill with CentOS 6.4 and the Open Compute Open Vault (Knox Unit) JBOD

In my previous post about Installing CentOS on the Open Compute Windmill servers, all of the testing was done and completed without using the OCP Knox Unit. Once connected, it routinely caused drive mapping issues. For instance, /dev/sda would become /dev/sdb, /dev/sdo or /dev/sdp at reboot. Causing the server to hang at boot since it could not find the appropriate filesystem.

The problem being, that the megaraid_sas driver was being loaded prior to the libsas driver, causing the Knox Unit drives to come online prior to the internal drives. Unfortunately, it was not consistent enough to just use /dev/sdb, /dev/sdo or /dev/sdp as the boot drive, since it would rotate depending on what server I was connected to.

After a plethora of testing, the working solution I was able to come up with is:

1. Blacklist megaraid_sas in the PXE menu as a kernel parameter using: rdblacklist megaraid_sas

2. Blacklist megaraid_sas in the kickstart file as a kernel parameter using: rdblacklist megaraid_sas

3. Blacklist megaraid_sas in /etc/modprobe.d/blacklist.conf with blacklist megaraid_sas

4. Load megaraid_sas in /etc/rc.modules with modprobe megaraid_sas

My updated kickstart:



Open Compute Windmill + Open Compute Open Vault hangs at booting from local disk fix

Working through installing CentOS 6.4, ESXi and others, I started running into issues where the systems would run their PXE installations just fine, then end up hanging at booting from local disk afterwards… As it turns out, the systems we’re having issues trying to boot to /dev/sda when /dev/sda was not always where the OS was getting installed… as it turns out, sometimes the local SSD would be /dev/sda, /dev/sdo, etc. This is due to the mpt_sas driver getting loaded after the megaraid_sas driver.

Continue reading “Open Compute Windmill + Open Compute Open Vault hangs at booting from local disk fix”

Open Compute Open Vault Knox Unit Drive Identification

Trying to find a failed drive that does not have a red light on it can be quite challenging with the Open Vault Knox Unit(s) since the drives/slots are not labeled.

After looking around the interweb’s and reading the Open Compute Project Open Vault Storage Specification I found the diagram indicating which drive is which.

Open Vault Knox Unit Drive Layout
Open Vault Knox Unit Drive Layout

That being said, you should be able to use MegaCLI to identify the drives using the following command:

Using MegaCLI, you can also list those drives, and their status using:

And you can see the results here, after the drive was replaced.

MegaCLI working examples cheat sheet

MegaCLI is a pain… a real pain. I’m not a fan of the syntax, it’s not easily readable, did I say it was a pain?

Well, our OCP Knox Units are connected via the LSI MEGARAID SAS 9286CV-8E (SGL) controller, so I’ve been using MegaCLI quite a bit lately. Here is a collection of my working examples:

List All Devices

Create raid 1 volume with disk 0 and 1

Create raid 5 volume with disk 0-14

Create individual raid 0 volumes on drives 2-14

List  all volumes

Create raid 50 volume

Bring physical drive 14 online

Show physical drive status

Enable JBOD support – this is necessary for vSphere vSAN technology

Set global hot spare

Delete all volumes

Clear foreign configs

List drive state

Enable drive location LED