CentOS 6.4 Mellanox 10GB Driver Install on the Open Compute Windmill V3 server

Our build out of the Open Compute Windmill V3 server utilizes the Mellanox MCX342A XCAN 10GbE, 2-p SFP+ mezzanine card. By default, the Mellanox card is not natively supported by CentOS 6.4.

[root@server /]# lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

That being said, I had to download the appropriate driver from Mellanox here. That being said, I also needed quite a few RPM’s to do the driver install, I did:

yum -y install kernel-source kernel headers gcc rpm-build  gcc make patch redhat-rpm-config

Even then, I was getting an error:

No kernel sources/headers found for 2.6.32-358.el6.x86_64 kernel.

As it turns out, the symbolic link of /lib/modules/2.6.32-358.el6.x86_64/build to ../../../usr/src/kernels/xxxxxxx was incorrect. I had to do:

ln -s ../../../usr/src/kernels/2.6.32-358.23.2.el6.x86_64 build

within /lib/modules/2.6.32-358.el6.x86_64/

Only then was I able to correctly install the Mellanox driver.

Automating the Cisco UCS build using Cisco PowerTool

After installing Cisco UCS and performing the initial setup of the UCS Fabric Interconnects; there is a lot of work that needs to be completed prior to configuring the blade Service Profiles.

This can be done a multitude of ways, manually through UCSM, scripted through SSH or now through PowerShell using Cisco PowerTool. This allows us to use variables at the top of the script; these variables allow us to make the script portable for various customer deployments.

There are a couple features that are not working yet:

  1. Setting the Global Power Allocation Policy – due to the enormous lack of detail in the get-help sections of Cisco’s PowerTool, I have been unable to find which function controls the Global Power Allocation Policy.  I believe it to be the set-ucspowergroup cmd-let but I have been unable to make it work correctly.
  2. Adding custom roles with customer permissions – the add-ucsuserrole cmd-let is not very well documented either, like others; I cannot seem to figure out how to get this cmd-let to create a new role with custom permissions.
$fabavsan = "VSAN_4"
$fabavsanid = "4"
$fabbvsan = "VSAN_5"
$fabbvsanid = "5"
$customerportgroup = "Guest_VLAN"
$mgmt_ippoolstart = "10.0.0.2"
$mgmt_ippoolfinish = "10.0.0.26"
$mgmt_ippoolgw = "10.0.0.1"
$ntp1 = "ntp1.domain.com"
$ntp2 = "ntp2.domain.com"
$snmpcomm = "readonlycommunity"
$snmplocation = "Datacenter Customer Location"
$traphost1 = "10.0.0.100"
$traphost2 = "10.0.0.101"

#User and Role commented out due to role functionality not working correctly with PowerTool (yet)
#Create Additional User
#Add-UcsLocalUser -Name test_user -Pwd Passw0rd! -FirstName Test -Lastname User

#Create Additional Role
#NOTWORKING add-ucsuserrole -name "Helpdesk" add-ucsuserrole does not modify this... cannot find within PowerTool

#Set Chassis Discovery Policy
Get-UcsChassisDiscoveryPolicy | Set-UcsChassisDiscoveryPolicy -Action 4-link -LinkAggregationPref port-channel -Rebalance immediate -Force

#Set Power Control Policy
Get-UcsPowerControlPolicy | Set-UcsPowerControlPolicy -Redundancy grid -Force

#Set MAC Aging Policy
get-ucslancloud | set-ucslancloud -macaging mode-default -force 

#Set Global Power Allocation Policy
#NOTWORKING -  set-ucspowergroup does not modify this... cannot find within PowerTool

#Add UCS FI Uplinks on FIA and FIB
add-ucsuplinkport -filancloud A -portid 17 -slotid 1
add-ucsuplinkport -filancloud A -portid 18 -slotid 1
add-ucsuplinkport -filancloud B -portid 17 -slotid 1
add-ucsuplinkport -filancloud B -portid 18 -slotid 1

#Add UCS FI Server Uplinks on FIA and FIB
add-ucsserverport -fabricservercloud A -portid 1 -slotid 1
add-ucsserverport -fabricservercloud A -portid 2 -slotid 1
add-ucsserverport -fabricservercloud A -portid 3 -slotid 1
add-ucsserverport -fabricservercloud A -portid 4 -slotid 1
add-ucsserverport -fabricservercloud B -portid 1 -slotid 1
add-ucsserverport -fabricservercloud B -portid 2 -slotid 1
add-ucsserverport -fabricservercloud B -portid 3 -slotid 1
add-ucsserverport -fabricservercloud B -portid 4 -slotid 1

#Configre Unified Ports to all be FC
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 1
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 2
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 3
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 4
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 5
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 6
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 7
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 8
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 9
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 10
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 11
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 12
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 13
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 14
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 15
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 16
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 1
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 2
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 3
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 4
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 5
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 6
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 7
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 8
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 9
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 10
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 11
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 12
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 13
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 14
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 15
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 16

#CREATE VLANS
Get-UcsLanCloud | Add-UcsVlan -Name ESX_MGMT -Id 102
Get-UcsLanCloud | Add-UcsVlan -Name ESX_VMKernel -Id 104
Get-UcsLanCloud | Add-UcsVlan -Name Utility -Id 108
Get-UcsLanCloud | Add-UcsVlan -Name VC_SQL -Id 110
Get-UcsLanCloud | Add-UcsVlan -Name $customerportgroup -Id 299

#CREATE VSANS
Get-UcsFiSanCloud -Id A | Add-UcsVsan -Name $fabavsan -Id $fabavsanid -fcoevlan $fabavsanid -zoningstate disabled
Get-UcsFiSanCloud -Id B | Add-UcsVsan -Name $fabbvsan -Id $fabbvsanid -fcoevlan $fabbvsanid -zoningstate disabled

#CONFIGURE QOS
get-ucsqosclass bronze | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass gold | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass platinum | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass silver | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass best-effort | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled

#CONFIGURE SAN PORTS TO VSAN
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true

#ADD Managment IP Pool Block
add-ucsippoolblock -IpPool "ext-mgmt" -from $mgmt_ipoolstart -to $mgmt_ipoolfinish -defgw $mgmt_ipoolgw -modifypresent:$true

#Configure NTP
add-ucsntpserver -name $ntp1
add-ucsntpserver -name $ntp2

#Configure TimeZone
set-ucstimezone -timezone "America/New_York (Eastern Time)" -Force

#Configure SNMP Community
set-ucssnmp -community $snmpcomm -syscontact ENOC -syslocation $snmplocation -adminstate enabled -force

#Configure SNMP Traps
add-ucssnmptrap -hostname $traphost1 -community $snmpcomm -notificationtype traps -port 162 -version v2c
add-ucssnmptrap -hostname $traphost2 -community $snmpcomm -notificationtype traps -port 162 -version v2c

#Create QOS Policies
Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name BE
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "best-effort" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Bronze
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "bronze" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Gold
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "gold" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Platinum
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "platinum" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Silver
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "silver" -Rate line-rate
Complete-UcsTransaction

#create local disk policy
Add-UcsLocalDiskConfigPolicy -name Local_Raid1 -descr Raid1_LocalDisk -mode raid-mirrored -protectconfig:$true

#create scrub policy
add-ucsscrubpolicy -org root -name Format_Disk -Desc Format_the_disk -DiskScrub yes -BiosSettingsScrub no

#create default mac pool to silence any alarms
add-ucsmacmemberblock -macpool default -from "00:25:B5:00:00:00" -to "00:25:B5:00:00:0F"

#create iscsi pool block to silence any alarms
add-ucsippoolblock -IpPool "iscsi-initiator-pool" -from 0.0.0.1 -to 0.0.0.1 -modifypresent:$true

#create default wwn node pool block to silence any alarms
add-ucswwnmemberblock -wwnpool node-default -from  20:00:00:25:B5:00:00:00 -to 20:00:00:25:B5:00:00:07

Assigning VSAN’s to Cisco UCS VSAN Uplinks using Cisco PowerTool

When building out your UCS deployment, you create a series of Fibre Channel (FC) uplinks. Then you have to go back and add those FC uplinks to the VSAN’s that you created. This can be tedious if you have a lot of FC uplinks as you would need to modify each one. With Cisco’s Powertool, you can do this quite quickly with no mouse movement :)

$fabavsan = "VSAN_01"
$fabbvsan = "VSAN_02"
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true

As you can see:

$fabavsan = "VSAN_01"
$fabbvsan = "VSAN_02"

are variable used so that you only need to modify the VSAN ID once. Otherwise, you would need to specify on each and every line.

This took me a while to figure out as Cisco’s get-help files (man pages for Linux folks) leave a lot to be desired.

if you run:

get-help add-ucsvsanmemberfcport -full

You get:

NAME
Add-UcsVsanMemberFcPort

SYNOPSIS
Adds Managed Object of type FabricFcVsanPortEp.

SYNTAX
Add-UcsVsanMemberFcPort -Vsan <FabricVsan> [-PortId] <uint> [-SlotId] <uint> [-SwitchId] <string> [-AdminState <string>] [-Name <string>] [-ModifyPresent] [-XtraProperty <Hashtable>] [-Ucs <UcsHandle[]>] [
<CommonParameters>]

DESCRIPTION
Adds Managed Object of type FabricFcVsanPortEp. FC Port that attaches to the parent VSAN. Indicates the parent VSAN will be configured on the specified FC port (FC storage port or FC uplink port). This is
configured explicitly by the user.

PARAMETERS
-Vsan <FabricVsan>
Specifies a Vsan object. The Vsan must already exist.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-PortId <uint>
Specifies the port identifier.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-SlotId <uint>
Specifies the slot identifier.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-SwitchId <string>
Specifies the switch id. Valid values are: A, B, NONE

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-AdminState <string>
The administrative (desired) state of this end-point. Valid values are: disabled, enabled

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-Name <string>
NO COMMENTS

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-ModifyPresent
The ModifyPresent option ensures that the add-cmdlets modify the MO, if it already exists, instead of returning an error.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-XtraProperty <Hashtable>
The XtraProperty parameter ensures that unknown Managed Object properties can also be used.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-Ucs <UcsHandle[]>
Specifies a list of ucs connection object(s)

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

<CommonParameters>
This cmdlet supports the common parameters: Verbose, Debug,
ErrorAction, ErrorVariable, WarningAction, WarningVariable,
OutBuffer and OutVariable. For more information, type,
"get-help about_commonparameters".

Which would lead you to believe that this command would work just fine:

add-UcsVsanMemberFcPort -Vsan VSAN_01 -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true

But it does not, you have to pipeline through the get-ucsvsan output first.

Introducing Vmware Horizon, where did ThinApp go?

VMware’s recent announcement of Horizon Suite, Horizon Workspace, Horizon View and Horizon Mirage seems to be lacking mention of ThinApp! So where did it go?

VMware Horizon Menu
VMware Horizon Menu

As we both know, VMware Horizon Mirage (formally Wanova) and Horizon View are the same products that they have always been. Horizon Workspace and Horizon Suite are the new kids on the block. With Horizon Suite being just a bundle of View, Mirage and Workspace. Continue reading “Introducing Vmware Horizon, where did ThinApp go?”