Category: PowerCli

Evening,

I got notified of this problem earlier today thanks to my awesome BCS engineer.  You can read VMware’s week old KB here.  Essentially certain versions of tools can cause a BSOD when a quiescing operation is done.   This is a big problem for API based backups since when possible they use this method.   There are three solutions provided by VMware at this time:

• Disable quiescing
• Do not select Quiescing guest file system when taking a snapshot
• Downgrade the VMware Tools to previous version not affected

Good news… the latest version of VMware Tools in 6 and 5.5 is affected.  I was notified that a few specific version were affected.  The KB does not specify them.   You can track versions to build numbers via this table: http://packages.vmware.com/tools/versions

I have been told the following are affected:

• 8399 – 8.6.15
• 9231 – 9.0.15
• 9355 – 9.4.11
• 9216 – 9.10.0

 

Here is a powershell snippet to locate the machine that are affected:

 $VMS = get-vm |get-view | where {$_.powerstate -ne "PoweredOff" } | where {$_.config.tools.toolsVersion -eq "8399" -or $_.config.tools.toolsVersion -eq "9231" -or $_.config.tools.toolsVersion -eq "9355" -or $_.config.tools.toolsVersion -eq "9216"}

 

I am not sure if there is a scripted way to downgrade the tools.  Here is VMware method to download older tools version.   As with all my articles you should open a VMware ticket to get specific production assistance.   Let me know if you know a scripted way to downgrade tools.

 

The issue with ESXi 5.5 U2 and 6 seems to be more wide-spread than just IBM storage.   One of the main problems with the current solution in the KB is that is requires that you login to each host individually.  A very smart co-worker provided this script with help from VMware’s BCS team.   He has given me permission to post it to assist others.   No promise that it works perfectly but it should help you on your path.

 

A few notes:

• It is set to test for the presence of the setting and do nothing if already set
• It is set to test if it’s 5.5 or higher
• It is set to test if it’s build 2068190 or higher

 

Param(

$VIServer = (Read-Host “Type the vCenter servers name and hit [ENTER]”)

)

if ((Get-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) -eq $null)

{Add-PsSnapin VMware.VimAutomation.Core}  <# Checks for the existence of and if not found,

loads the VMware snap-in#>

if (!($DefaultVIServer.Name -eq $VIServer -and $DefaultVIServer.IsConnected -eq “True”)) {

Connect-VIServer $VIServer -Credential (Get-Credential) -ErrorAction Stop | Out-Null

}

 

$vmhosts = Get-Cluster | Sort Name | Get-VMHost | Sort Name

foreach($vmhost in $vmhosts){

If($vmhost.ApiVersion -ge “5.5”){

If($vmhost.Build -ge “2068190”){

$esxcli = Get-EsxCli -vmhost $vmhost

If(($esxcli.system.settings.advanced.list($false, “/VMFS3/UseATSForHBOnVMFS5”)).IntValue -eq “0”){

Write-Host $vmhost.Name “already set to revert heartbeat” -ForegroundColor White

}

Else{

Write-host “Starting on” $vmHost.name -ForegroundColor Yellow

$esxcli = Get-EsxCli -vmhost $vmhost

$esxcli.system.settings.advanced.set($false, 1, “/VMFS3/UseATSForHBOnVMFS5”)

Write-Host “Finished with” $vmHost.name -ForegroundColor Green

}

}

Else{

Write-Host $vmhost.name “build is less than U2 (2068190), so skipping host” -ForegroundColor Red

}

}

Else{

Write-Host $vmhost.Name “ESXi version is less than 5.5, so skipping host” -ForegroundColor Red

}

}

 

I hope it helps you. Please don’t use it unless directed by VMware support.

I have been working with vSphere to get internally generated Alarms an SNMP Trap for ticket generation.   This process seemed simple on the surface but proved quite challenging.   The high level steps are as follows:

• Choose which vCenter Alarms need to be ticketed
• Configure the Alarms to send SNMP events to SNMP trap
• Download the vSphere MIB and install in SNMP Trap
• Configure the Alarm actions on SNMP Trap
• Tickets get opened

Choose which vCenter Alarms need to be ticketed

You can get a list of current alarms with powercli as follows:

Get-AlarmDefinition

Configure Alarms to send SNMP

vCenter must be configured to use SNMP with the following lines:

$srv = vcenterservername

Get-AdvancedSetting –Entity $srv  –Name snmp.receiver.2.community | Set-AdvancedSetting –Value public
Get-AdvancedSetting –Entity $srv  –Name snmp.receiver.2.enabled | Set-AdvancedSetting –Value $true
Get-AdvancedSetting –Entity $srv  –Name snmp.receiver.2.name | Set-AdvancedSetting –Value 192.168.1.10

The following will add SNMP to the alarms:

Get-AlarmDefinition -Name "Alarm1" | New-AlarmAction -Snmp

Configure the Alarm Action in SNMP Trap

I ran into a number of issues that generated this community post .   The essential issue is that all SNMP events generated by vSphere come in as the same type of event vpxaAlarmInfo.  The details of the event contains information an internal name.   This is where the problem begins.   The name for any custom created event is the name of the event.  For example if I create a Alarm called JoeTest then it’s called JoeTest.   Sounds simple right?   Well… no because the VMware built in alarms don’t following this naming convention.   The Host connection and power state (easiest one for me to generate)  is named alarm.HostConnectionStateAlarm.   Making my mappings for any VMware generated events very hard.    So I went on a quest to locate these names.

 

The Quest for the names

My first stop was PowerCLI using the command:

$bob = Get-AlarmDefinition -Name "Host connection and power state"

$bob | fl

 

This fine powershell did not produce the alarm.HostConnectionStateAlarm name.  It did produce a Alarm-145 (unique to my vCenter).   I tried lots of ways to work on this object like get-view etc… without any luck.

 

My next stop was the MOB (Managed Object Browse)  also known as my least favorite place.   Using the following MOB I was able to learn everything about the alarm except the name for SNMP:

https://vcenter/mob/?moid=alarm-145

https://vcenter/mob/?moid=alarm-145&doPath=info

https://vcenter/mob/?moid=alarm-145&doPath=info.action.action

https://vcenter/mob/?moid=alarm-145&doPath=info.expression.expression

https://vcenter/mob/?moid=alarm-145&doPath=info.setting

 

This lead me to my last stop the vCenter database.  Some finely crafted searches produced a number of tables with the alarm.xxx information.   I was left with the VPX_EVENT_ARG table.  It seems to be a table of all events in the system.   Inside this I was able to locate names that seemed to fit.   A few more minutes did not produce any primary keys to link to the Alarm tables.   I was stuck so I punted.  The following is a SQL command I used to produce the Alarms names:

 

select distinct OBJ_NAME from [vCenter].[dbo].[VPX_EVENT_ARG] where obj_name like ‘%alarm%’

 

It produced the following built in Alarm names:

alarm.BatteryHealthAlarm
alarm.BMCHealthAlarm
alarm.ConsistencyGroupViolation
alarm.DatastoreDiskUsageAlarm
alarm.DatastoreInMultipleDatacenters
alarm.DatastoreStorageComplianceAlarm
alarm.ExitStandbyErrorAlarm
alarm.FanHealthAlarm
alarm.HAcannotFindMaster
alarm.HAfailoverFailed
alarm.HAfailoverInProgress
alarm.HAhostStatus
alarm.HAinsufficientFailoverResources
alarm.HAvmMonitoringAction
alarm.HAvmMonitoringError
alarm.HealthStatusChangedAlarm
alarm.HostConnectionStateAlarm
alarm.HostConnectivityAlarm
alarm.HostCPUUsageAlarm
alarm.HostErrorAlarm
alarm.HostEsxCosSwapAlarm
alarm.HostLicenseEditionNotAllowed
alarm.HostMemoryUsageAlarm
alarm.HostVendorProviderRegistrationAlarm
alarm.IormNonVIWorkloadAlarm
alarm.LicenseCapacityExceededAlarm
alarm.LicenseError
alarm.LicenseNonComplianceAlarm
alarm.LicenseUserThresholdExceededAlarm
alarm.LunCapabilityAlarm
alarm.MemoryHealthAlarm
alarm.MigrateBindToVMKAlarm
alarm.MigrationErrorAlarm
alarm.NetworkConnectivityLostAlarm
alarm.NetworkRedundancyDegradedAlarm
alarm.NetworkRedundancyLostAlarm
alarm.OtherHealthAlarm
alarm.PowerHealthAlarm
alarm.ProcessorHealthAlarm
alarm.SELHealthAlarm
alarm.SiocNotSupportedHostAlarm
alarm.StorageConnectivityAlarm
alarm.StorageHealthAlarm
alarm.StoragePodOutOfSpace
alarm.StoragePodSDRSNotSupportedHost
alarm.StoragePodSDSRecommendation
alarm.SystemBoardHealthAlarm
alarm.TemperatureHealthAlarm
alarm.ThinProvisionedLunAlarm
alarm.VCHealthStateChangedAlarm
alarm.VdsHCMTUMatchAlarm
alarm.VdsHCMTUSupportedAlarm
alarm.VdsHCTeamingMatchAlarm
alarm.VdsHCVlanTrunkedAlarm
alarm.VFlashResourceHealthAlarm
alarm.VFlashResourceUsageAlarm
alarm.VmCPUUsageAlarm
alarm.VmDiskConsolidationNeededAlarm
alarm.VmErrorAlarm
alarm.VmFaultToleranceLatencyStatusAlarm
alarm.VmFaultToleranceStateChangedAlarm
alarm.VmMemoryUsageAlarm
alarm.VmNoCompatibleHostForSecondaryAlarm
alarm.VmStorageComplianceAlarm
alarm.VmTimedoutStartingSecondaryAlarm
alarm.VoltageHealthAlarm
alarm.VsanClusterLicenseExpiryAlarm
alarm.VsanHostSsdOverUsageAlarm

 

Testing two additional events confirmed I was on the correct track.

 

End Result

Yep I don’t have a clue how they link but it produced a list that seems to work and I it.  I hope it helps you save some time.

 

vSphere 5 comes with a slew of really great alarms prebuilt by VMWare.   There is only one problem by default they all just alarm in the GUI.  I really want to avoid having someone login to find out status.  One organization I worked for had a ticketing system that accepted input as emails.  It was not the cleanest but it worked.  So here are some generic alarms that I recommend switching to email notifications.

 

“Datastore usage on disk”
“Health status changed Alarm”
“Health status monitoring”
“Insufficient vSphere HA failover resources”
“VMKernel NIC not configured correctly”
“vSphere HA failover in progress”
“vSphere HA host status”

 

Yes you can manually change them to send emails in the GUI… but it’s a lot quicker to use PowerCLI.  Here are the commands:

 

Get-AlarmDefinition "Datastore usage on disk" | New-AlarmAction -Email -To “your_email@domain.com” -Subject "Datastore usage on disk" 
Get-AlarmDefinition "Health status changed Alarm" | New-AlarmAction -Email -To “your_email@domain.com” -Subject "Health status changed Alarm" 
Get-AlarmDefinition "Health status monitoring" | New-AlarmAction -Email -To “your_email@domain.com” -Subject "Health status monitoring" 
Get-AlarmDefinition "Insufficient vSphere HA failover resources" | New-AlarmAction -Email -To “your_email@domain.com” -Subject "Insufficient vSphere HA failover resources"
Get-AlarmDefinition "VMKernel NIC not configured correctly" | New-AlarmAction -Email -To “your_email@domain.com” -Subject "VMKernel NIC not configured correctly"
Get-AlarmDefinition "vSphere HA failover in progress" | New-AlarmAction -Email -To “your_email@domain.com” -Subject "vSphere HA failover in progress"
Get-AlarmDefinition "vSphere HA host status" | New-AlarmAction -Email -To “your_email@domain.com” -Subject "vSphere HA host status"

 

 

I had a project where the customer wanted everything converted from thin to thick lazyZero.  Of course I wanted to do this automatically.  Here are a few things to know:

• Changing the disk type requires you move the disk between luns so it helps to have a go between
• If you convert EagerZero into thin it will still take up the whole disk

The name of the disk types in powerCLi are different

• Thin = thin provisioned
• Thick = lazy zero thick
• EagerZeroedThick = Eager Zeroed Thick

So here are the commands in powerCli

Move-VM $VM -Datastore $destDS -DiskStorageFormat $disktype

Just replace the following items

• $VM with name of virtual machine
• $destDS with destination datastore name
• $disktype with disk type

 

So the whole process for hundreds of VM’s would be this

1. Check destination disk format fits what you want to change from
2. Check for snapshots
3. Check for RDM
4. Move disk to temp storage lun and change format
5. Move disk back to original datastore and keep changed format
6. Rinse and repeat

Enjoy.

 

Ever wonder how much storage is used by your thin provisioned luns?  Well here is some power shell to find out:

 

Thin provisioned allocation

 

# Define Variables

$outputFile = 'C:\VMDiskCapacity.csv'

$myCol = @()    # Prepare output collection

$VMs = Get-VM | sort Name    # Get all VMs (sorted)

$counter = 0    # Initialize counter for progress bar

ForEach ($VM in $VMs)    # Loop through VMs

   {

   $counter++    # Increase counter for progress bar

   Write-Progress -Activity "Gathering disk information" -Status "Processing VM $VM" -PercentComplete (100*($counter/$VMs.count))    #

   $myObj = "" |

   select VM, TotalDiskSizeGB # Create output object

   $myObj.VM = $VM.Name    # Virtual Machine Name

   $TotalDiskSizeKB = 0

   ForEach ($DISK in $VM.HardDisks)

      {

      $TotalDiskSizeKB += $DISK.CapacityKB

      }

   $myObj.TotalDiskSizeGB = [math]::Round(($TotalDiskSizeKB * 1KB / 1GB),0) #Disk Size in GB

   $myCol += $myObj    # Add output to collection

   }

$myCol | Export-Csv $outputFile -NoTypeInformation  # Export output to csv