Networking – Joseph Griffiths

October 23, 2017

Basic NSX Network virtualization setup

This post will go over the basic setup for network virtualization in NSX. This is nothing new or exciting but I figured I would share as more users are deploying NSX in their home labs these days. I will assume that you already have the environment prepared by deploying the manager and controllers and all your ESXi hosts are prepared.

We are going to set up the subnet of 10.0.0.0/17 to be virtually routed as shown below:

This requires the following:

Static route on Linksys EA6200 router to point 10.0.0.0/17 to 192.168.10.223 (because my Linksys does not support any dynamic routing protocols)
A logical switch called Transport-10.0.0.0 between the border ESG and the Logical distributed router
OSPF configured between ESG-3 and LDR-3

Creation of the LDR-3 (pictures to follow steps)

First we need to create a logical switch by choosing Logical Switches, select green + button, Input Name (Transport-10.0.0.0) and description and click ok
Select NSX Edges in Navigator pane, select green + button
In Name and description pane: Install Type: Logical (distributed) route, Name: LDR-3, Hostname ldr3, leave deploy NSX Edge selected, Next
In settings, type your password, I like to enable ssh, click next
In configure deployment: Press the green + to deploy a NSX Edge Appliance, Select correct resource pool, datastore, host, and folder, click ok
Click Next
In Configure interfaces
Select connected to for HA interface: Port group DV-VM, press + below HA and add 192.168.10.224
press the green + button under interfaces
In Add NSX Edge Interface: Name Uplink, Connected to: Transport-10.0.0.0, Press green + to add IP: 10.0.0.2 subnet 24, Click ok
Click Next
In Default gateway settings: Set the gateway IP as 10.0.0.1 and click next
Ignore the Firewall and HA settings click next
Click finish to deploy LDR

Creation of the ESG-3 (pictures to follow steps)

Back at the NSX Edge section in Navigator
Press the Green + sign
In Name and description: Choose Edge Services gateway, Name: ESG-3, Hostname esg3 and select Next (in Production you might want high availability or ECMP)
In Settings: Type Admin password and enable ssh, Next
In Configure deployment: Press Green + sign, Select resource pool, datastore and host then ok and Next
In Configure Interfaces: press the green + sign
Name: Uplink, Connected To: DV-VM, Press green + to add interface: 192.168.10.223 subnet 24, click ok
Click Next
In Default gateway settings insert default gateway of 192.168.10.1 then next
Ignore firewall and HA settings and next
Click Finish to deploy appliance

Configure Physical router

This is unique per router in mine I added a static route for the subnet:

Configure LDR

We need to add at least one inside network and configure OSPF.

Logical Switch section we are going to add a switch for 10.0.1.0/24 called LS-10.0.1
In Logical Switch Section: Green + button, Name LS-10.0.1 then OK
Go to NSX Edges in Navigator
Double click on LDR-3
We need to add a interface for the new network Select Manage, Settings, Interfaces
Select Green +
Name: GW-10.0.1: Connected To LS-10.0.1, Green + button to add interface 10.0.1.1 subnet 24,
Select Routing tab, global Configuration
Go to Dynamic Routing configuration and click edit
Make sure the uplink interface is chosen then click ok
Press Publish Changes button
Click on OSPF button
Remove all current area definitions (51 ) with red X then publish changes
Click green + on area definitions and add area 2 (just type 2 in area button leave rest default)
Press green + in area to interface mapping button
Make sure Uplink is selected and area 2 and press OK
Press Edit button next to OSPF configuration and enable OSPF, For protocol address choose a free IP 10.0.0.3, forwarding is 10.0.0.2
Publish Changes
Go to firewall section
Disable firewall
Publish changes

Configure ESG-3

Return to Networking & Security main section
Select NSX Edges and double click on ESG-3
Select Manage, Settings, Interfaces
We need to add a interface for the transport between LDR and ESG
Select vnic1 and press Edit button
Connected to: Transport-10.0.0.0, IP: 10.0.0.1 subnet 24
Select Routing
In global configuration: Select edit next to dynamic routing configuration, ensure uplink is selected and press ok
Publish changes
Click on OSPF
Remove current area definitions with red X and publish changes
Add a new area for area 2 leaving everything else default
In the area to interface mapping make sure you chose vnic1 (internal link) and area 2
Select OSPF Configuration and Enable OSPF
Publish Changes
Select Firewall section and disable firewall and publish changes

Validate Configuration

Let’s validate configuration three ways: Confirming OSPF settings on ESG-3, Adding a new subnet, ping test

Confirming on ESG-3

Login to ESG-3 via SSH (username admin password set during deployment)
Type the following to see current routes (show ip route) ensure that the E2 learned route is showing:

Adding a new subnet

Stay logged into the ESG-3
Switch to the Networking and security console, navigate to Logical switches
Press green + to add a switch for LS-10.0.2
select NSX Edges, Double click on LDR-3
Go to Manage and settings
Select Interfaces and press green +
Name: GW-10.0.2, Internal, Connected to: LS-10.0.2, IP 10.0.2.1 subnet 24
Return to the ESG-3 ssh session and run the command show ip route to see 10.0.2.0/24

Test Via ping

Attempt to ping either gateway on the LDR (10.0.1.1 or 10.0.2.1)

Additional commands on ESG-3

Here are some commands that will help you in troubleshooting OSPF

show ip ospf neightbors – show other members of the areas

show ip ospf database – understand current ospf database

September 27, 2017

Redeploy NSX Edges to a different cluster / datacenter

First Issue my bad

I ran into an interesting issue in my home lab. I recently replaced all my older HP servers with Intel NUC’s. I could not be happier with the results. Once I replaced all the ESXi hosts I mounted the storage and started up my virtual machines including vCenter. Once vCenter and NSX Manager were available I moved all the ESXi hosts to the distributed switch. This normal process was complicated by NSX. I should have added the ESXi hosts to the transport zone allowing NSX to join the distributed switch. Failure to do this made the NSX VXLAN process fail. I could not prepare the hosts… ultimately I removed the VXLAN entries from the distributed switch and then re-prepared which re-created the VXLAN entries on the switch. (This is not a good idea if you use it in production so follow the correct path.

Second Issue nice to know

This process generated a second issue the original cluster and datacenter on which my NSX edges used to live was gone. I assumed that I could just re-deploy NSX edges from the manager. While this is true the configuration assumes that it will be deploying the Edges to the same datacenter, resource pool and potentially the same host as when it was created. So if I have a failure and expect to just bring up NSX manager and redeploy to a new cluster it will not work. You have to adjust the parameters for the edges you can do this via the API or GUI. I wanted to demonstrate the API method:

I needed to change the resource pool, datastore, and host for my Edge. I identified my Edge via the identifier name in the GUI. (edge-8 for me) Grabbed my favorite REST tool (postman) and formed a query on the current state:

Get https://{nsx-manager-ip}/api/4.0/edges/edge-8/appliances

This returned the configuration for this edge device. If you need to identify all edges just do

Get https://{nsx-manager-ip}/api/4.0/edges

Then I needed the VMware identifier for resource pool, datastore and host – this can all be gathered via the REST API but I went for Powershell because it was faster for me. I used the following commands in PowerCLI:

get-vmhost | fl - returned host-881

get-resourcepool | fl - returned domain-c861

get-datastore | fl - returned datastore-865

Once identified I was ready to form my adjusted query:

<appliances>
<applianceSize>compact</applianceSize>
<appliance>
<highAvailabilityIndex>0</highAvailabilityIndex>
<vcUuid>500cfc30-5b2a-6bae-32a3-360e0315ccd3</vcUuid>
<vmId>vm-924</vmId>
<resourcePoolId>domain-c861</resourcePoolId>
<resourcePoolName>domain-c861</resourcePoolName>
<datastoreId>datastore-865</datastoreId>
<datastoreName>datastore-865</datastoreName>
<hostId>host-881</hostId>
<vmFolderId>group-v122</vmFolderId>
<vmFolderName>NSX</vmFolderName>
<vmHostname>esg1-0</vmHostname>
<vmName>ESG-1-0</vmName>
<deployed>true</deployed>
<cpuReservation>
<limit>-1</limit>
<reservation>1000</reservation>
</cpuReservation>
<memoryReservation>
<limit>-1</limit>
<reservation>512</reservation>
</memoryReservation>
<edgeId>edge-9</edgeId>
<configuredResourcePool>
<id>domain-c26</id>
<name>domain-c26</name>
<isValid>false</isValid>
</configuredResourcePool>
<configuredDataStore>
<id>datastore-31</id>
<isValid>false</isValid>
</configuredDataStore>
<configuredHost>
<id>host-29</id>
<isValid>false</isValid>
</configuredHost>
<configuredVmFolder>
<id>group-v122</id>
<name>NSX</name>
<isValid>true</isValid>
</configuredVmFolder>
</appliance>
<deployAppliances>true</deployAppliances>
</appliances>

I used a PUT against https://{nsx-manager-ip}/api/4.0/edges/{edgeId}/appliances with the above body in xml/application. Then I was able to redeploy my edge devices without any challenge.

April 13, 2017

Migrating off a distributed virtual switch to standard switch Article 2

Normally people want to migrate from virtual standard switches to distributed switches. I am a huge fan of the distributed switch and feel it should be used everywhere. The distributed switch becomes a challenge when you want to migrate hosts to a new vCenter. I have seen a lot of migrations to new vCenters via detaching the ESXi hosts and connecting to the new vCenter. This process works great assuming you are not using the distributed switch. Removing or working with VM’s on a ghosted VDS is a real challenge. So remove it before you migrate to a new vCenter.

In this multi-article solution I’ll provide some steps to migrate off a VDS to a VSS.

Article 1: Creating a VSS and migrating the virtual machines
Article 2: Migrating the host off the VDS
Article 3: Migrating the host to a new vCenter and back into the distributed switch

Article 2: Migrating the host off the VDS. In the last article we moved all the virtual machines off the VDS to a VSS. We now need to migrate the vMotion and management off the VDS to a VSS. This step will cause interruption to the management of a ESXi host. Virtual machines will not be interrupted but the management / will be. You must have console access to the ESXi host for this to work. Steps at a glance:

Confirm that a switch port exists for management and vMotion
Remove vMotion, etc.. from VDS and add to VSS
Remove management from VDS and add to VSS
Confirm settings

Confirm that a switch port exists for management and vMotion

Before you begin examine the VSS to confirm that management and vMotion port groups were created correctly by Article 1's script. Once your sure the VLAN settings for the port group are correct then you can move to the next step. You may want to confirm your host isolation settings it’s possible these steps will cause a HA failure if you take too long to switch over and don’t have independent datastore networking. Best practice would be to disable HA or switch to leave powered on isolation response.

Remove vMotion, etc.. from VDS and add to VSS

#use the following command to identify virtual adapters on your dvs

esxcfg-vswitch -l

# sample output from my home lab

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks

dvSwitch 1792 7 512 1600 vmnic1

DVPort ID In Use Client

675 0

676 1 vmnic1

677 0

678 0

679 1 vmk0

268 1 vmk1

139 1 vmk2

# We can see we have three virtual adapters on our host use the following command to identify their use and IP addresses

esxcfg-vmknic -l

#Sample output from my home lab cut out some details to make it more readable

Interface Port Group/DVPort IP Family IP Address

vmk0 679 IPv4 192.168.10.16

vmk1 268 IPv4 192.168.10.26

vmk2 139 IPv4 192.168.10.22

Align you vmk# with vCenter to identify which adapter provides the function (vmk0 management, vmk1 vMotion, vmk2 FT)

# We can now move all adapter other than management which in my case is vmk0 #we will start with vmk1 on dvSwitch on port 268

esxcfg-vmknic -d -v 268 -s "dvSwitch"

# Then add to vSwitch0 vmk1

esxcfg-vmknic -a -i 192.168.10.26 -n 255.255.255.0 -p PG-vMotion

Remove FT

esxcfg-vmknic -d -v 139 -s "dvSwitch"

esxcfg-vmknic -a -i 192.168.10.22 -n 255.255.255.0 -p PG-FT

Remove management from VDS and add to VSS

Remove management (this stage will interrupt management access to ESXi host – make sure you have console access) You might want to pretype the add command in the console before you execute the remove. If you are having trouble getting the shell on a ESXi host do the following:

You will need to login to the console go to troubleshooting options -> Enable ESXi Shell
Press Alt-Cntr-F1 to enter shell and login

Remove management:

esxcfg-vmknic -d -v 679 -s "dvSwitch"

Add management to VSS:

esxcfg-vmknic -a -i 192.168.10.16 -n 255.255.255.0 -p PG-Mgmt

Confirm settings

Ping the host to ensure networking has returned to management. Ensure the host returns to vCenter by waiting 2 minutes. After you move the host to a new vCenter you can remove via:

Go to the host in vCenter and select dvs it should provide a remove button.

April 4, 2017

Migrating off a distributed virtual switch to standard switch Article 1

In this multi-article solution I’ll provide some steps to migrate off a VDS to a VSS.

Article 1: Creating a VSS and migrating the virtual machines
Article 2: Migrating the host off the VDS
Article 3: Migrating the host to a new vCenter and back into the distributed switch

It’s important to understand that assuming that networking is correct this process should not interrupt customer virtual machines. The movement from a distributed switch to a standard switch at most will lose a ping. When you assign a new network adapter a gratuitous arp is sent out the new adapter. If you only have two network adapters this process does remove network adapter redundancy while moving.

Step 1: Create a VSS with the same port groups

You need to create a standard switch with port groups on the correct VLAN ID’s. You can do this manually but one of the challenges of the standard switch is the name must be exactly the same including case sensitivity to avoid vMotion errors. (One great reason for the VDS) So we need to use a script to create the standard switch and port groups. Using PowerCLI (sorry orchestrator friends I didn’t do it in Orchestrator this time)

Code:

#Import modules for PowerCLI

Import-Module -Name VMware.VimAutomation.Core

Import-Module -Name VMware.VimAutomation.Vds

#Variables to change

$standardSwitchName = "StandardSwitch"

$dvSwitchName = "dvSwitch"

$cluster = "Basement"

$vCenter = "192.168.10.14"

#Connect to vCenter

connect-viserver -server $vCenter

$dvsPGs = Get-VirtualSwitch -Name $dvSwitchName | Get-VirtualPortGroup | Select Name, @{N="VLANId";E={$_.Extensiondata.Config.DefaultPortCOnfig.Vlan.VlanId}}, NumPorts

#Get all ESXi hosts in a cluster

$vmhosts = get-cluster -Name $cluster | get-vmhost

#Loop ESXi hosts

foreach ($vmhost in $vmhosts)

{

#Create new VSS

$vswitch = New-VirtualSwitch -VMHost $vmhost -Name $standardSwitchName -Confirm:$false

#Look port groups and create

foreach ($dvsPG in $dvsPGs)

{

#Validate the port group is a number the DVUplink returns an array

if ($dvsPg.VLANId -is [int] )

{

New-VirtualPortGroup -Name $dvsPG.Name -VirtualSwitch $vswitch -VlanId $dvsPG.VLANId -Confirm:$false

}

Explained:

Provide variables
Connect to vCenter
Get all port groups into $dvsPGs
Get all ESXi hosts
Loop though ESXi hosts one at a time
Create the new standard switch
Loop through port groups and create them with same name as DVS and VLAN ID

This will create a virtual standard switch with the same VLAN and port group configuration as your DVS.

I like to be able to validate that the source and destination are configured the same so this powercli script provides the checking:

Code:

#Validation check DVS vs VSS for differences

$dvsPGs = Get-VirtualSwitch -Name $dvSwitchName | Get-VirtualPortGroup | Select Name, @{N="VLANId";E={$_.Extensiondata.Config.DefaultPortCOnfig.Vlan.VlanId}}, NumPorts

#Get all ESXi hosts in a cluster

$vmhosts = get-cluster -Name $cluster | get-vmhost

#Loop ESXi hosts

foreach ($vmhost in $vmhosts)

{

#Write-Host "Host: "$vmhost.Name "VSS: "$standardSwitchName

#Get VSSPortgroups for this host

$VSSPortGroups = $vmhost | Get-VirtualSwitch -Name $standardSwitchName | Get-VirtualPortGroup

#Sort based upon name of VSS

foreach ($dvsPG in $dvsPGs)

{

if ($dvsPg.VLANId -is [int] )

{

#Write "VSSPortGroup: " $VSSPortGroup.Name

#Loop on DVS

$match = $FALSE

foreach ($VSSPortGroup in $VSSPortGroups)

{

if ($dvsPG.Name -eq $VSSPortGroup.Name)

{

#Write-Host "Found a Match vss: "$VSSPortGroup.Name" to DVS: "$dvsPG.Name" Host: " $vmhost.name

$match = $TRUE

$missing = $dvsPG.Name

}

if ($match -eq $FALSE)

{

Write-Host "Did not find a match for DVS: "$missing " on "$vmhost.name

}

Explained:

Get the VDS
Get all ESXi hosts
Loop through VM hosts
Get port groups on standard switch
Loop though the standard switch port groups and look for matches on DVS
If missing then output missing element

Now we need to give the standard switch an uplink (this is critical otherwise VM’s will fail when moved)

Once it has an uplink you can use the following script to move all virtual machines:

Code:

#Move Virtual machines to new Adapters

$vms = get-vm

foreach ($vm in $vms)

{

#grab the virtual switch for the hosts

$vss = Get-VirtualSwitch -Name $standardswitchname -VMHost $vm.VMHost

#check that the virtual switch has at least one physical adapter

if ($vss.ExtensionData.Pnic.Count -gt 0)

{

#VMHost

$adapters = $vm | Get-NetworkAdapter

#Loop through adapters

foreach ($adapter in $adapters)

{

#Get VSS port group of same name returns port group on all hosts

$VSSPortGroups = Get-VirtualPortGroup -Name $adapter.NetworkName -VirtualSwitch $standardSwitchName

#Loop the hosts

foreach ($VSSPortGroup in $VSSPortGroups)

{

#Search for the PortGroup on our host

if ([string]$VSSPortGroup.VMHostId -eq [string]$vm.VMHost.Id)

{

#Change network Adapter to standard switch

Set-NetworkAdapter -NetworkAdapter $adapter -Portgroup $VSSPortGroup -Confirm:$false

}

Explained:

Used same variables from previous script
Get all virtual machines (you could use get-vm “name-of-vm” to test a single vm
Loop through all virtual machines one at a time
Get the VSS for the VM (host specific)
Check for at least one physical uplink to switch (gut / sanity check)
Loop though the adapters on a virtual machine
For each adapter get VDS port group name and switch the adapter

December 2, 2016

VMkernel types updated with design guidance for multi-site

Holy crap what do all these VMware VMkernel type mean? I started this article and realized I had already written one here. Sad when google leads you to something you wrote… looks like I don’t remember too well… Perhaps I should just go yell for the kids to get off my lawn now. I wanted to take a minute to revise my post with some new things I have learned and some guidance.

From my previous post:

vMotion traffic – Required for vMotion – Moves the state of virtual machines (active datadisk svMotion, active memory and execution state) during a vMotion
Provisioning traffic – Not required will use management network if not setup – cold migration, cloning and snapshot creation (powered off virtual machines = cold)
Fault tolerance traffic (FT) – Required for FT – Enables fault tolerance traffic on the host – only a single adapter may be used for FT per host
Management traffic – Required – Management of host and vCenter server
vSphere replication traffic – Only needed if using vSphere replication– outgoing replication data from ESXi host to vSphere replication server
vSphere replication NFC traffic – Only needed if using vSphere replication – handles incoming replication data on the target replication site
Virtual SAN – Required for VSAN – virtual san traffic on the host
VXLAN – used for NSX not controlled from the add vmkernel interface.

I wanted to provide a little better explanation around design elements with some interfaces. Specifically I want to focus on vMotion and Provisioning traffic. Let’s create a few scenario’s and see what interface is used assuming I have all the VMkernel interfaces listed above:

VM1 is running and we want to migrate from host1 to host2 at datacenter1 – vMotion
VM1 is running with a snapshot and we want to migrate from host1 to host2 at datacenter1 – Provisioning traffic (if it does not exist management network is used)
VM1 is running with a snapshot and we want to storage migrate from host1 DC1 to host4 DC3 – storage vMotion – Provisioning traffic (if it does not exist management network is used)
VM1 is not running and we want to migrate from host1 to host2 at datacenter1 – Provisioning traffic (very low bandwidth used)
VM1 is not running has a snapshot and we want to migrate from host1 to host2 at datacenter1 – Provisioning traffic (very low bandwidth used)
VM2 is being created at datacenter1 – Provisioning traffic

So design guidance in a multi-site implementation you should have the following interfaces if you wish to separate the TCP-IP stack or use network IO control to avoid bad neighbor situations. (Or you could just assign it all to management vmk and go nuts on that interface = bad idea)

Management
vMotion
Provisioning

Use of other vmkernel interfaces depends on if you are using replication, vSAN or NSX.

Should you have multi-nic vMotion?

Multi-nic vMotion enables faster vMotion of multiple entries off a host (as long as they don’t have snapshots). It still is a good idea if you have large vm’s or lots of vm’s on a host.

Should you have multi-nic Provisioning?

No idea if it’s even supported or a good idea. Provisioning network is used for long distance vMotion so the idea might be good… I would not use it today.

November 11, 2016

Configuring a NSX load balancer from API

A customer asked me this week if there was any examples of customers configuring the NSX load balancer via vRealize Automation. I was surprised when google didn’t turn up any examples. The NSX API guide (which is one of the best guides around) provides the details for how to call each element. You can download it here. Once you have the PDF you can navigate to page 200 which is the start of the load balancer section.

Too many Edge devices

NSX load balancers are Edge service gateways. A normal NSX environment may have a few while others may have hundreds but not all are load balancers. A quick API lookup of all Edges provides this information: (my NSX manager is 192.168.10.28 hence the usage in all examples)

https://192.168.10.28/api/4.0/edges

        <edgeSummary>
            <objectId>edge-57</objectId>
            <objectTypeName>Edge</objectTypeName>
            <vsmUuid>420CD713-469F-7053-8281-A7BD66A1CD46</vsmUuid>
            <nodeId>92484cee-ab3c-4ed2-955e-e5bd135f5be5</nodeId>
            <revision>2</revision>
            <type>
                <typeName>Edge</typeName>
            </type>
            <name>LB-1</name>
            <clientHandle></clientHandle>
            <extendedAttributes/>
            <isUniversal>false</isUniversal>
            <universalRevision>0</universalRevision>
            <id>edge-57</id>
            <state>deployed</state>
            <edgeType>gatewayServices</edgeType>
            <datacenterMoid>datacenter-21</datacenterMoid>
            <datacenterName>Home</datacenterName>
            <tenantId>default</tenantId>
            <apiVersion>4.0</apiVersion>
            <recentJobInfo>
                <jobId>jobdata-34935</jobId>
                <status>SUCCESS</status>
            </recentJobInfo>
            <edgeStatus>GREEN</edgeStatus>
            <numberOfConnectedVnics>1</numberOfConnectedVnics>
            <appliancesSummary>
                <vmVersion>6.2.0</vmVersion>
                <vmBuildInfo>6.2.0-2982179</vmBuildInfo>
                <applianceSize>compact</applianceSize>
                <fqdn>NSX-edge-57</fqdn>
                <numberOfDeployedVms>1</numberOfDeployedVms>
                <activeVseHaIndex>0</activeVseHaIndex>
                <vmMoidOfActiveVse>vm-283</vmMoidOfActiveVse>
                <vmNameOfActiveVse>LB-1-0</vmNameOfActiveVse>
                <hostMoidOfActiveVse>host-29</hostMoidOfActiveVse>
                <hostNameOfActiveVse>vmh1.griffiths.local</hostNameOfActiveVse>
                <resourcePoolMoidOfActiveVse>resgroup-27</resourcePoolMoidOfActiveVse>
                <resourcePoolNameOfActiveVse>Resources</resourcePoolNameOfActiveVse>
                <dataStoreMoidOfActiveVse>datastore-31</dataStoreMoidOfActiveVse>
                <dataStoreNameOfActiveVse>SYN8-NFS-GEN-VOL1</dataStoreNameOfActiveVse>
                <statusFromVseUpdatedOn>1478911807005</statusFromVseUpdatedOn>
                <communicationChannel>msgbus</communicationChannel>
            </appliancesSummary>
            <hypervisorAssist>false</hypervisorAssist>
            <allowedActions>
                <string>Change Log Level</string>
                <string>Add Edge Appliance</string>
                <string>Delete Edge Appliance</string>
                <string>Edit Edge Appliance</string>
                <string>Edit CLI Credentials</string>
                <string>Change edge appliance size</string>
                <string>Force Sync</string>
                <string>Redeploy Edge</string>
                <string>Change Edge Appliance Core Dump Configuration</string>
                <string>Enable hypervisorAssist</string>
                <string>Edit Highavailability</string>
                <string>Edit Dns</string>
                <string>Edit Syslog</string>
                <string>Edit Automatic Rule Generation Settings</string>
                <string>Disable SSH</string>
                <string>Download Edge TechSupport Logs</string>
            </allowedActions>
        </edgeSummary>

This is for a single Edge gateway in my case I have 57 Edges deployed over the life of my NSX environment and 15 active right now. But only Edge-57 is a load balancer. This report does not provide anything that can be used to identify it as a load balancer from a Edge as a firewall. In order to identify if it’s a load balancer I have to query it’s load balancer configuration using:

https://192.168.10.28/api/4.0/edges/edge-57/loadbalancer/config

Notice the addition of the edge-57 name to the query. It returns:

<loadBalancer>
    <version>2</version>
    <enabled>true</enabled>
    <enableServiceInsertion>false</enableServiceInsertion>
    <accelerationEnabled>false</accelerationEnabled>
    <monitor>
        <monitorId>monitor-1</monitorId>
        <type>tcp</type>
        <interval>5</interval>
        <timeout>15</timeout>
        <maxRetries>3</maxRetries>
        <name>default_tcp_monitor</name>
    </monitor>
    <monitor>
        <monitorId>monitor-2</monitorId>
        <type>http</type>
        <interval>5</interval>
        <timeout>15</timeout>
        <maxRetries>3</maxRetries>
        <method>GET</method>
        <url>/</url>
        <name>default_http_monitor</name>
    </monitor>
    <monitor>
        <monitorId>monitor-3</monitorId>
        <type>https</type>
        <interval>5</interval>
        <timeout>15</timeout>
        <maxRetries>3</maxRetries>
        <method>GET</method>
        <url>/</url>
        <name>default_https_monitor</name>
    </monitor>
    <logging>
        <enable>false</enable>
        <logLevel>info</logLevel>
    </logging>
</loadBalancer>

Notice that this edge has load balancer enabled as true with some default monitors. To compare here is a edge without the feature enabled:

https://192.168.10.28/api/4.0/edges/edge-56/loadbalancer/config

<loadBalancer>
    <version>1</version>
    <enabled>false</enabled>
    <enableServiceInsertion>false</enableServiceInsertion>
    <accelerationEnabled>false</accelerationEnabled>
    <monitor>
        <monitorId>monitor-1</monitorId>
        <type>tcp</type>
        <interval>5</interval>
        <timeout>15</timeout>
        <maxRetries>3</maxRetries>
        <name>default_tcp_monitor</name>
    </monitor>
    <monitor>
        <monitorId>monitor-2</monitorId>
        <type>http</type>
        <interval>5</interval>
        <timeout>15</timeout>
        <maxRetries>3</maxRetries>
        <method>GET</method>
        <url>/</url>
        <name>default_http_monitor</name>
    </monitor>
    <monitor>
        <monitorId>monitor-3</monitorId>
        <type>https</type>
        <interval>5</interval>
        <timeout>15</timeout>
        <maxRetries>3</maxRetries>
        <method>GET</method>
        <url>/</url>
        <name>default_https_monitor</name>
    </monitor>
    <logging>
        <enable>false</enable>
        <logLevel>info</logLevel>
    </logging>
</loadBalancer>

Enabled is false with the same default monitors. So now we know how to identify which edges are load balancers:

Get list of all Edges via API and pull out id element
Query each id element for load balancer config and match on true

Adding virtual servers

You can add virtual servers assuming the application profile and pools are already in place with a POST command with a XML body payload like this (the virtual server IP must already be assigned to the Edge as an interface):

https://192.168.10.28/api/4.0/edges/edge-57/loadbalancer/config/virtualservers

<virtualServer>
<name>http_vip_2</name>
<description>http virtualServer 2</description>
<enabled>true</enabled>
<ipAddress>192.168.10.18</ipAddress>
<protocol>http</protocol>
<port>443,6000-7000</port> 
<connectionLimit>123</connectionLimit>
<connectionRateLimit>123</connectionRateLimit>
<applicationProfileId>applicationProfile-1</applicationProfileId>
<defaultPoolId>pool-1</defaultPoolId>
<enableServiceInsertion>false</enableServiceInsertion>
<accelerationEnabled>true</accelerationEnabled>
</virtualServer>

You can see it’s been created. A quick query:

https://192.168.10.28/api/4.0/edges/edge-57/loadbalancer/config/virtualservers

<loadBalancer>
    <virtualServer>
        <virtualServerId>virtualServer-5</virtualServerId>
        <name>http_vip_2</name>
        <description>http virtualServer 2</description>
        <enabled>true</enabled>
        <ipAddress>192.168.10.18</ipAddress>
        <protocol>http</protocol>
        <port>443,6000-7000</port>
        <connectionLimit>123</connectionLimit>
        <connectionRateLimit>123</connectionRateLimit>
        <defaultPoolId>pool-1</defaultPoolId>
        <applicationProfileId>applicationProfile-1</applicationProfileId>
        <enableServiceInsertion>false</enableServiceInsertion>
        <accelerationEnabled>true</accelerationEnabled>
    </virtualServer>
</loadBalancer>

Shows it’s been created. To delete just use the virtualServerId and pass to DELETE

https://192.168.10.28/api/4.0/edges/edge-57/loadbalancer/config/virtualservers/virtualserverID

Pool Members

For pools you have to update the full information to add a backend member or for that matter remove a member. So you first query it:

https://192.168.10.28/api/4.0/edges/edge-57/loadbalancer/config/pools

<?xml version="1.0" encoding="UTF-8"?>
<loadBalancer>
    <pool>
        <poolId>pool-1</poolId>
        <name>pool-1</name>
        <algorithm>round-robin</algorithm>
        <transparent>false</transparent>
    </pool>
</loadBalancer>

Then you form your PUT with the data elements you need (taken from API guide).

https://192.168.10.28/api/4.0/edges/edge-57/loadbalancer/config/pools/pool-1

<pool>
<name>pool-1</name>
<description>pool-tcp-snat</description>
<transparent>false</transparent>
<algorithm>round-robin</algorithm>
<monitorId>monitor-3</monitorId>
<member>
<ipAddress>192.168.10.14</ipAddress>
<weight>1</weight>
<port>80</port>
<minConn>10</minConn>
<maxConn>100</maxConn>
<name>m5</name>
<monitorPort>80</monitorPort>
</member>
</pool>

In the client we see a member added:

Tie it all together

Each of these actions have a update delete and query function that can be done. The real challenge is taking the API inputs and creating user friendly data into vRealize Input to make it user friendly. NSX continues to amaze me as a great product that has a very powerful and documented API. I have run into very little issues trying to figure out how to do anything in NSX with the API. In a future post I may provide some vRealize Orchestrator actions to speed up configuration of load balancers.

January 11, 2016

VMkernel interfaces in vSphere 6

Not every one has noticed the new types of vmkernel interfaces in vSphere 6. Here is a quick note to identify the types of interfaces available:

vMotion traffic – Required for vMotion – Moves the state of virtual machines (active datadisk svMotion, active memory and execution state) during a vMotion
Provisioning traffic – Not required will use management network if not setup – cold migration, cloning and snapshot creation (powered off virtual machines = cold)
Fault tolerance traffic (FT) – Required for FT – Enables fault tolerance traffic on the host – only a single adapter may be used for FT per host
Management traffic – Required – Management of host and vCenter server
vSphere replication traffic – Only needed if using vSphere replication – outgoing replication data from ESXi host to vSphere replication server
vSphere replication NFC traffic – Only needed if using vSphere replication – handles incoming replication data on the target replication site
Virtual SAN – Required for VSAN – virtual san traffic on the host

The purpose of the multiple interface types is you are now allowed to route all these types of traffic in 6. Allowing you to segment this traffic even more. (In ESXi 5.xx only management had a TCP/IP stack.) I recommend that you create unique subnets for each of these types of traffic you can use. In addition many of them support multiple concurrent nic’s (like multi-nic vMotion) which can improve performance. When possible setup multi-nic.

September 8, 2015

VCIX-NV VMware Certified Implementation Expert Network Virtualization Exam Experience

While attending VMworld I have made a habit of taking advantage of the discounted certifications. Each year I have pushed into a new certification and given a try. This year I have my sights set on VCIX-NV. I normally like to schedule these tests during the general session to avoid the impact of a 3 hour test during VMworld sessions. (I can always watch the general sessions on YouTube Later) This year they closed my secret loop-hole only allowing me to take it during Thursday’s general session. This has a profound impact on my plans for Wednesday night. Wednesday Night is the VMworld party which is always really awesome. I am sad to say I skipped it 100% this year and spent the time studying and going to bed early. I am happy to announce I passed the exam and earned a VCP-NV and VCIX-NV in the same day. Here is some details on what I studied:

The VMware HOL labs
The blueprint documents (yes I read all the documentation provided by VMware except the CLI guide.. it’s a really more of a reference guide)
I was lucky to spend some time with Elver Sosa and Ron Flax (both VCDX-NV’s) last year that helped me understand the technology
Plural Sight course on NSX by Jason Nash (these are really good)
Time in the HOL doing things not on the list (like more firewall rules)

This test requires that you do a series of live exercises that can build upon each other. Some time management tips are:

Skip ahead if allows and see how tasks fit in
Read carefully what is expected there are a lot of tips
Do what you can partial credit is points (at least I think it is)
Spend time before understanding how to troubleshoot nsx and verify your settings
Don’t be afraid to skip a question if you really don’t know time is not your friend

It was like the VCAP-DCA test something I really enjoyed doing… I really wish it didn’t have the time crunch but it was a fun exam. The best advice I can give you is read the blueprint and documents and use the HOL from VMware to gain experience.

March 4, 2015

VCIX-NV Study Guide Objective 1.2

To see other posts in this series go here.

This section deals with upgrading from older versions of vShield to NSX. The simple answer is there is a specific order that must be followed. Upgradeds from vShield require version 5.5. Most of it is in the GUI via vCenter except the vShield Manager which will be replaced by NSX Manager. Most of these processes roughly follow the documented process in this document.

Products name translation:

Roughly here are the old names to new names or new service providing function:

vShield Manager -> NSX Manager

Virtual Wires -> NSX Logical Switch

vShield App -> NSX Firewall

vShield -> NSX Edge

vShield Endpoint ->vShield Endpoint

Data security -> NSX Datasecurity

Practicing this process:

Unless you want to take a few hours configuring all vShield products it’s hard to practice. You can do the upgrade from vShield Manager to NSX manager really quickly. Just download the vShield Manager and setup with the following:

Deploy OVF
Power on
Console login as admin with password of default
type enable with password of default
type setup
Setup your IP settings
Wait 5 minutes
Login via IP with web browser and do upgrade

The rest of the upgrade requires you understand vShield products which is not required for NV so I vote you skip it and be familiar with process, order and requirements.

Objective 1.2 Denotes the following items:

Upgrade vShield Manager 5.5 to NSX Manager 6.x.

Upgrading vShield Manager to NSX Manager can only be done from version 5.5 of vShield. It also requires the following things:

vCenter 5.5
vShield Data Security uninstalled
vShield Edges be upgraded to 5.5

Process:

Download the NSX upgrade bundle called vCNS to NSX for vSphere xxx Upgrade Bundle
Login to vShield Manager and click Settings & Reports
Click Updates tab and click upload upgrade bundle
Click Choose file Browse to the vCNS to NSX for vSphere xxx Upgrade Bundle and click open
Click Upload file – this process will take some time
Click Install to begin the upgrade process
Click confirm install – this will reboot the vShield manager – none of the other components are rebooted
After upgrade visit the ip address of your vShield manager again via https
Login and look at summary page to confirm you are running NSX Manager
Log off all windows and close your browser to clear cache
Login to vSphere Web client
Shutdown your NSX Manager vm and increase memory to 12GB and vCPU to 4

Upgrade NSX Manager 6.0 to NSX Manager 6.0.x
Upgrade Virtual Wires to Logical Switches

Virtual wires must be upgraded to NSX logical switches to use NSX features. The process is required even if you don’t use virtual wires. In order for this to work you need to upgrade your vShield manager to NSX manager and make sure it’s connected to vSphere.

Process

Login to Web client
Networking and Security Tab click install
Click host prepare
Virtual wires will show as Legacy
Click update on each wire
Wait for them to show green and no longer legacy

Upgrade vShield App to NSX Firewall

You can only upgrade vShield App 5.5 to NSX. It requires that vShield manager be upgraded to NSX manager and virtual wires upgraded to NSX logical switches.

A pop up window should ask if you want to upgrade
Click update and wait
Done

Upgrade vShield 5.5 to NSX Edge 6.x

This upgrade requires the following:

vShield 5.5
NSX Manager
Virtual wires upgraded to NSX logical switches

Processes:

Login to web client
Networking & Security tab
NSX Edges button
Select upgrade version from actions menu on each edge
After compete check the version number tab

Upgrade vShield Endpoint 5.x to vShield Endpoint 6.x

This upgrade requires the following:

vShield Manager upgraded to NSX Manager
Virtual wires upgraded to NSX Logical switches

Process:

Login to web client
Networking & Security tab
Click Installation
Click Service deployments tab
Click on upgrade available
Select datastore (must be shared) and network and ok

Upgrade to NSX Data Security

There is no clean upgrade path you have to remove before install of NSX manager. You have to re-register the solution with NSX if available.

February 3, 2015

vSphere 6.0 What excites me

Yesterday VMware announced the general release of vSphere 6.0. It is not yet available for download or install but it’s exciting. There are lots of really awesome new features and improvements. I want to focus on what I consider to be the two most exciting features:

vMotion improvements
VVol’s

vMotion improvements:

Long Distance vMotion – You can now live vMotion across distances of up to 100ms RTT (up from 10ms RTT) – Removing the distance factors
Cross vSwitch and vCenter vMotion – Move virtual machines live between different virtual switches and different vCenters removing almost all boundaries on vMotion
vMotion of MSCS VMs using pRDMs – I could not find much on this I really want to know more here.
vMotion L2 adjacency restrictions removed – vMotion no longer requires layer 2 stretch you can now route vMotion – something I have been trying to get a RPQ on for a while. (When combined with NSX you now no longer require any spanning tree protocols, you could route everything)

What does all this mean. Better design removing past layer 2 requirements and the true ability to migrate active work loads into the cloud. Make no mistake all these changes are about flexible movement into the cloud. Combine with NSX and your workload can move anywhere. These changes have a big impact on vSphere metro clusters. I would love to see HA domains include more than one site using multiple vCenters and site replication for fail over of load. (I expect it to come next version just my personal thoughts)

VVol’s

This is the holy grail of software defined storage. Giving the storage system awareness to the individual object or virtual machine. This enabled granular level performance. It can enable flash clone/copy style backup removing the current kludgy process. I have been super excited about VVol’s for a while now. I saw a demo at VMworld 2013 done by HDS that was awesome. This object level storage solution enables awesome features like replication, deduplication, compression, storage tiering at the hard drive level, real performance metrics on the complete storage system and stack. This is really going to blow up the hyper-converged storage vendors solutions. Mark my words the first vendor to adopt and correctly market VVol’s will be huge.

Here is the problem: storage vendors have to enable and support the features. Expect some of the start up storage vendors to be supporting it right away. While larger vendors may take a while to solve the issue.

There are a ton of improvements to the new version including a solution to the lack of HA for vCenter (multiple processor FT).

Future

It is clear that VMware feels that the future is in hybrid cloud and mobility. I am sure they have lots of smart research to prove it. Compliance and configuration management continue to be my largest problem with public cloud. I think solutions like NSX start to resolve the public cloud problems. I look forward to the future with my friend VMware. If anyone from VMware ever reads this article please consider HA across vCenters with replication as an option, it would be just perfect (maybe combined with VMware VSAN to up sales).