Hyper-Converged Solutions

For the past few years, the big boys (HP, Cisco, Dell) have been trying to convince us that converged infrastructure is the way to go. Buy the stack of compute, storage, and network from us and off you go. One throat to choke, the best experience because all the components are engineered to work optimally together.

But is converged infrastructure already on the way out and now hyper-converged infrastructure the way to go with virtualized workloads.

My first experience with hyper-converged was with Nutanix. To be honest I didn’t get and didn’t trust it. I needed to get my hands dirty in order to understand the new approach to virtualized workloads that Nutanix offers.

It finally clicked and then eventually I find there are others out there that compete in this space (Simplivity, Scale Computing, Pivot3). Great! I am all for competition and choice.

When it comes to hyper-converged solutions in general, some of my concerns are:

  • Will the solution start cracking apart at a certain scale? The file system starts to crack at certain node count.
  • And since I am a desktop virtualization guy, does the solution support GPU’s?
  • Do you support only a single virtualization platform? Will there be additional options in the future?

From an administrator point of view, hyper-converged solutions are a great solution because of the brick by brick ability to scale out approach.

What’s been your experience with hyper-converged solutions? Are you using it for a certain use-case?

Automating the Cisco UCS build using Cisco PowerTool

After installing Cisco UCS and performing the initial setup of the UCS Fabric Interconnects; there is a lot of work that needs to be completed prior to configuring the blade Service Profiles.

This can be done a multitude of ways, manually through UCSM, scripted through SSH or now through PowerShell using Cisco PowerTool. This allows us to use variables at the top of the script; these variables allow us to make the script portable for various customer deployments.

There are a couple features that are not working yet:

  1. Setting the Global Power Allocation Policy – due to the enormous lack of detail in the get-help sections of Cisco’s PowerTool, I have been unable to find which function controls the Global Power Allocation Policy.  I believe it to be the set-ucspowergroup cmd-let but I have been unable to make it work correctly.
  2. Adding custom roles with customer permissions – the add-ucsuserrole cmd-let is not very well documented either, like others; I cannot seem to figure out how to get this cmd-let to create a new role with custom permissions.
$fabavsan = "VSAN_4"
$fabavsanid = "4"
$fabbvsan = "VSAN_5"
$fabbvsanid = "5"
$customerportgroup = "Guest_VLAN"
$mgmt_ippoolstart = "10.0.0.2"
$mgmt_ippoolfinish = "10.0.0.26"
$mgmt_ippoolgw = "10.0.0.1"
$ntp1 = "ntp1.domain.com"
$ntp2 = "ntp2.domain.com"
$snmpcomm = "readonlycommunity"
$snmplocation = "Datacenter Customer Location"
$traphost1 = "10.0.0.100"
$traphost2 = "10.0.0.101"

#User and Role commented out due to role functionality not working correctly with PowerTool (yet)
#Create Additional User
#Add-UcsLocalUser -Name test_user -Pwd Passw0rd! -FirstName Test -Lastname User

#Create Additional Role
#NOTWORKING add-ucsuserrole -name "Helpdesk" add-ucsuserrole does not modify this... cannot find within PowerTool

#Set Chassis Discovery Policy
Get-UcsChassisDiscoveryPolicy | Set-UcsChassisDiscoveryPolicy -Action 4-link -LinkAggregationPref port-channel -Rebalance immediate -Force

#Set Power Control Policy
Get-UcsPowerControlPolicy | Set-UcsPowerControlPolicy -Redundancy grid -Force

#Set MAC Aging Policy
get-ucslancloud | set-ucslancloud -macaging mode-default -force 

#Set Global Power Allocation Policy
#NOTWORKING -  set-ucspowergroup does not modify this... cannot find within PowerTool

#Add UCS FI Uplinks on FIA and FIB
add-ucsuplinkport -filancloud A -portid 17 -slotid 1
add-ucsuplinkport -filancloud A -portid 18 -slotid 1
add-ucsuplinkport -filancloud B -portid 17 -slotid 1
add-ucsuplinkport -filancloud B -portid 18 -slotid 1

#Add UCS FI Server Uplinks on FIA and FIB
add-ucsserverport -fabricservercloud A -portid 1 -slotid 1
add-ucsserverport -fabricservercloud A -portid 2 -slotid 1
add-ucsserverport -fabricservercloud A -portid 3 -slotid 1
add-ucsserverport -fabricservercloud A -portid 4 -slotid 1
add-ucsserverport -fabricservercloud B -portid 1 -slotid 1
add-ucsserverport -fabricservercloud B -portid 2 -slotid 1
add-ucsserverport -fabricservercloud B -portid 3 -slotid 1
add-ucsserverport -fabricservercloud B -portid 4 -slotid 1

#Configre Unified Ports to all be FC
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 1
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 2
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 3
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 4
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 5
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 6
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 7
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 8
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 9
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 10
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 11
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 12
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 13
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 14
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 15
Get-UcsFiSanCloud -Id “A” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 16
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 1
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 2
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 3
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 4
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 5
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 6
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 7
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 8
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 9
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 10
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 11
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 12
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 13
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 14
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 15
Get-UcsFiSanCloud -Id “B” | Add-UcsFcUplinkPort -ModifyPresent -AdminState “enabled” -SlotId 2 -PortId 16

#CREATE VLANS
Get-UcsLanCloud | Add-UcsVlan -Name ESX_MGMT -Id 102
Get-UcsLanCloud | Add-UcsVlan -Name ESX_VMKernel -Id 104
Get-UcsLanCloud | Add-UcsVlan -Name Utility -Id 108
Get-UcsLanCloud | Add-UcsVlan -Name VC_SQL -Id 110
Get-UcsLanCloud | Add-UcsVlan -Name $customerportgroup -Id 299

#CREATE VSANS
Get-UcsFiSanCloud -Id A | Add-UcsVsan -Name $fabavsan -Id $fabavsanid -fcoevlan $fabavsanid -zoningstate disabled
Get-UcsFiSanCloud -Id B | Add-UcsVsan -Name $fabbvsan -Id $fabbvsanid -fcoevlan $fabbvsanid -zoningstate disabled

#CONFIGURE QOS
get-ucsqosclass bronze | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass gold | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass platinum | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass silver | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled
get-ucsqosclass best-effort | set-ucsqosclass -mtu 9000 -Force -Adminstate enabled

#CONFIGURE SAN PORTS TO VSAN
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid B -modifypresent:$true

#ADD Managment IP Pool Block
add-ucsippoolblock -IpPool "ext-mgmt" -from $mgmt_ipoolstart -to $mgmt_ipoolfinish -defgw $mgmt_ipoolgw -modifypresent:$true

#Configure NTP
add-ucsntpserver -name $ntp1
add-ucsntpserver -name $ntp2

#Configure TimeZone
set-ucstimezone -timezone "America/New_York (Eastern Time)" -Force

#Configure SNMP Community
set-ucssnmp -community $snmpcomm -syscontact ENOC -syslocation $snmplocation -adminstate enabled -force

#Configure SNMP Traps
add-ucssnmptrap -hostname $traphost1 -community $snmpcomm -notificationtype traps -port 162 -version v2c
add-ucssnmptrap -hostname $traphost2 -community $snmpcomm -notificationtype traps -port 162 -version v2c

#Create QOS Policies
Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name BE
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "best-effort" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Bronze
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "bronze" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Gold
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "gold" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Platinum
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "platinum" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Silver
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "silver" -Rate line-rate
Complete-UcsTransaction

#create local disk policy
Add-UcsLocalDiskConfigPolicy -name Local_Raid1 -descr Raid1_LocalDisk -mode raid-mirrored -protectconfig:$true

#create scrub policy
add-ucsscrubpolicy -org root -name Format_Disk -Desc Format_the_disk -DiskScrub yes -BiosSettingsScrub no

#create default mac pool to silence any alarms
add-ucsmacmemberblock -macpool default -from "00:25:B5:00:00:00" -to "00:25:B5:00:00:0F"

#create iscsi pool block to silence any alarms
add-ucsippoolblock -IpPool "iscsi-initiator-pool" -from 0.0.0.1 -to 0.0.0.1 -modifypresent:$true

#create default wwn node pool block to silence any alarms
add-ucswwnmemberblock -wwnpool node-default -from  20:00:00:25:B5:00:00:00 -to 20:00:00:25:B5:00:00:07

Assigning VSAN’s to Cisco UCS VSAN Uplinks using Cisco PowerTool

When building out your UCS deployment, you create a series of Fibre Channel (FC) uplinks. Then you have to go back and add those FC uplinks to the VSAN’s that you created. This can be tedious if you have a lot of FC uplinks as you would need to modify each one. With Cisco’s Powertool, you can do this quite quickly with no mouse movement :)

$fabavsan = "VSAN_01"
$fabbvsan = "VSAN_02"
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabavsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 14 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 15 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true
get-ucsvsan $fabbvsan | add-UcsVsanMemberFcPort -portid 16 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true

As you can see:

$fabavsan = "VSAN_01"
$fabbvsan = "VSAN_02"

are variable used so that you only need to modify the VSAN ID once. Otherwise, you would need to specify on each and every line.

This took me a while to figure out as Cisco’s get-help files (man pages for Linux folks) leave a lot to be desired.

if you run:

get-help add-ucsvsanmemberfcport -full

You get:

NAME
Add-UcsVsanMemberFcPort

SYNOPSIS
Adds Managed Object of type FabricFcVsanPortEp.

SYNTAX
Add-UcsVsanMemberFcPort -Vsan <FabricVsan> [-PortId] <uint> [-SlotId] <uint> [-SwitchId] <string> [-AdminState <string>] [-Name <string>] [-ModifyPresent] [-XtraProperty <Hashtable>] [-Ucs <UcsHandle[]>] [
<CommonParameters>]

DESCRIPTION
Adds Managed Object of type FabricFcVsanPortEp. FC Port that attaches to the parent VSAN. Indicates the parent VSAN will be configured on the specified FC port (FC storage port or FC uplink port). This is
configured explicitly by the user.

PARAMETERS
-Vsan <FabricVsan>
Specifies a Vsan object. The Vsan must already exist.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-PortId <uint>
Specifies the port identifier.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-SlotId <uint>
Specifies the slot identifier.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-SwitchId <string>
Specifies the switch id. Valid values are: A, B, NONE

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-AdminState <string>
The administrative (desired) state of this end-point. Valid values are: disabled, enabled

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-Name <string>
NO COMMENTS

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-ModifyPresent
The ModifyPresent option ensures that the add-cmdlets modify the MO, if it already exists, instead of returning an error.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-XtraProperty <Hashtable>
The XtraProperty parameter ensures that unknown Managed Object properties can also be used.

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

-Ucs <UcsHandle[]>
Specifies a list of ucs connection object(s)

Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?

<CommonParameters>
This cmdlet supports the common parameters: Verbose, Debug,
ErrorAction, ErrorVariable, WarningAction, WarningVariable,
OutBuffer and OutVariable. For more information, type,
"get-help about_commonparameters".

Which would lead you to believe that this command would work just fine:

add-UcsVsanMemberFcPort -Vsan VSAN_01 -portid 13 -slotid 2 -adminstate enabled -switchid A -modifypresent:$true

But it does not, you have to pipeline through the get-ucsvsan output first.

Creating UCS QOS Policies using Cisco PowerTool

Creating QOS policies within UCS Manager can be somewhat cumbersome. In UCS Manager, on the Lan tab; you would right-click UCS policies and click create… one by one you would go through this process to define you UCS policies, if you have a lot of policies or a lot of UCS builds to do this can become quite monotonous.

Creating Cisco UCS QOS Policies
Creating Cisco UCS QOS Policies

Using Cisco PowerTool, this can be done quite quickly using the Add-UCSQosPolicy and Add-UCSvNicEgressPolicy command-lets. For example:

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name BE
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "best-effort" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Bronze
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "bronze" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Gold
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "gold" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Platinum
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "platinum" -Rate line-rate
Complete-UcsTransaction

Start-UcsTransaction
$mo = Get-UcsOrg -Level root  | Add-UcsQosPolicy -Name Silver
$mo_1 = $mo | Add-UcsVnicEgressPolicy -ModifyPresent -Burst 10240 -HostControl none -Prio "silver" -Rate line-rate
Complete-UcsTransaction

Would create the Best Effort, Bronze, Gold, Platinum, and Silver QOS Policies.