Category: Networking

So you like to ssh/have console into your ESXi host don’t you?  You enjoy the raw power.  We all do but here are some reasons why it’s a bad idea:

• Security risk – one more way into all your virtual machines that is commonly hacked
• Memory waste – having it enabled eats memory, disabled uses none

If you are stuck on enabling it you might want to consider locking it down with the ESXi firewall enabling ssh connections from only a few known source ports.  Like this (from esxi host or vMA):

List current firewall rules:

esxcli network firewall ruleset list

This will output something like this:

Name Enabled
------------------ -------
sshServer true
sshClient false
nfsClient false
dhcp true
dns true
snmp true
ntpClient true
CIMHttpServer true
CIMHttpsServer true
CIMSLP true
iSCSI true
vpxHeartbeats true
updateManager false
faultTolerance true
webAccess true
vMotion true
vSphereClient true
activeDirectoryAll false
NFC true
HBR true
ftpClient false
httpClient false
gdbserver false
DVFilter false
DHCPv6 false
DVSSync false
syslog false
IKED false
WOL true
vSPC false
remoteSerialPort false
vprobeServer false

To show the firewall default actions run the following command:

esxcli network firewall get

Default Action: DROP
 Enabled: true
 Loaded: true

To see the current firewall rules on sshServer use the following:

esxcli network firewall ruleset list --ruleset-id sshServer

Name Enabled
--------- -------
sshServer true

See current allowed ip list:

esxcli network firewall ruleset allowedip list --ruleset-id sshServer

Ruleset Allowed IP Addresses
--------- --------------------
sshServer All

Turn the allow all list to false:

esxcli network firewall ruleset set --ruleset-id sshServer --allowed-all false

Add the 10.10.101.0/24 subnet to the ssh allow list:

esxcli network firewall ruleset allowedip add --ruleset-id sshServer --ip-address 10.10.101.0/24

Check current list

esxcli network firewall ruleset allowedip list --ruleset-id sshServer

Now you have locked down your host access to ESXi ssh but I keep getting a warning on the host that ssh/console is enabled…. I love this error it really shows you vmware’s feelings about this feature.   Here is how you make it stop, well you can do it via the gui but let’s do it in the console:

esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

I have run into this issue a few times now.  My vcenter is a virtual machine.  Running on a vDS.  My ESXi host fails in a odd way: for example the host does not fail but the nic’s stop passing traffic.  My isolation response keeps the vm powered on and HA see’s that the host is still accessing storage so no fail over happens.   How do I get it back?  Well I can power it off and move it to another host but I cannot power it on… Why?

Because my vDS is setup with static ports.  Which means when vcenter is not available each ESXi host cannot start new virtual machines because they cannot allocate ports on the vDS.   For a long time I ran my vcenter on a standard switch to avoid this issue.  Now Vmware has provided a new option.  In order to use this option we need to understand vDS port group types:

• Static Binding – The default setting, when you power on a virtual machine it is assigned a port and guarenteed that port going forward.  The only time it looses this connection is when a vm is removed from the port group.  The main issue with static binds port groups is they have a set number of ports available at creation time.  If you try to power on a vm without available ports it fails.  This has been fixed in 5.0 with a auto grow feature. Static assignments are made by virtual center not the host.  Virtual center then creates the port on the ESXi host.
• Dynamic Binding- This is now deprecated and should not be used.
• Ephemeral Binding – This is a type of port binding to vm that is created by the host (not virtual center as in Static) The assignment is released by the host when a virtual machine is powered off or deleted.  There are almost unlimited number of ports available (limited by host maximum number of ports).

So why don’t we just make all our port groups Ephemeral?

• They don’t scale as well you are only allowed to have 256 total port groups with ephemeral.
• They consume a lot more resources on each host.
• Vmware does not want you to do it… as evident by the auto-expand static binding.

So when do I use ephemeral?

On my vcenter port group so I can move it and power on without having a virtual center powered on.

Solved that Catch 22 …

So you just read my post on how you will want to change your vcenter to a ephemeral port binding… but how is it done in 5.1 web client?  Well here yo go:

1. Click vcenter
2. Click Distributed Switches
3. Select your dV switch
4. Select your port group
5. Click Manage and Settings
6. Select Edit
7. Select the drop down on Port binding
8. Select Ephemeral
9. Select OK

That’s it and your all set.

This is a common issue… you have run out of ports on your dVS or virtual switch port group.   You are running a Static binding port which means you have a set number of total ports and no more.  How do you fix it? There are two ways:

• Increase the number of ports
• Enabled auto-expand

Increase the number of ports:

-Remember that static binds assign a port per virtual machine… it does not matter if it is powered on or not as long as it’s on the port group it get’s a static port.  To increase the number of ports in the web client (5.1) do the following:

• Click vcenter
• Click Distributed Switches
• Select your dV switch
• Select your port group
• Click Manage and Settings
• Select Edit

Adjust the number of ports and click ok.

Enabled auto-expand:

To increase the number of ports in the web client (5.1) do the following:

• Click vcenter
• Click Distributed Switches
• Select your dV switch
• Select your port group
• Click Manage and Settings
• Select Edit

This time choose port allocation and select Elastic and click ok.

Just remember that new ports are allocated on dVS switches by vcenter and it has to be available to provide ports.

You can also automate this with vMA using the following command:

updatedvPortgroupAutoExpand.pl --operation enable --dvportgroup portgroupname

Classic question there are all these virtual networking adapter types which one do I choose?  99% of the people you talk to will tell you they let vmware choose when then select the operating system.  This will choose a compatible network adapter type but not always the best type.  Each generation of virtual adapter brings better performance and features.  As a rule of thumb you want the highest vmxnet adapter your system supports.  As of ESXi 5 the following adapters are available listed in order preference (worst to best):

• Flexible – Has two functions can function as a vlance or vmxnet (will be vlance unless vmware tools is installed)  vlance is an emulated 10Mbps nic available on almost all operating systems.  vmxnet is the first generation of virtualized only network cards and requires vmware tools to be installed.
•  e1000 – is an emulated Intel 82545EM Gigabit ethernet NIC with support in most operating systems.  It is the default adapter for all 64-bit operating systems and is required for guest VLAN tagging.
• vmxnet2 – Updated version of vmxnet that contains VLAN tagging, jumbo frames and hardware off-load with additional high-performance features
• vmxnet3 – Is not really related the vmxnet2 but does represent the next generation of nic drivers it includes all features of vmxnet2 plus multiqueue support, IPv6 offloads, MSI/MSI-X interrupt – this driver has limited OS support requires vmware tools like all vmxnet adapters and requires Esxi hardware version 7 (Esxi version 4 at least)

How do I choose?  The best answer is consult vmwares knowledge base for information:

http://kb.vmware.com/kb/1001805

This is a fun one.  I never used to have this problem with RHEL 5 but RHEL 6 and debian based distros have had this issue all along.   You clone from a Linux template and you get a new mac address.  But the new interface comes in as ETH1 or perhaps if you have multiple generations you might have eth6 or eth7.  How do you clear this up?  Well it’s all about the fact that the operating system keeps track of the mac address in it’s udev rules. So open up /etc/udev/rules.d/70-persistent-net.rules and delete all the mac address entries.  Once you reboot the machine you should have your interface come in as eth0 again.

In Debian it is normally named: /etc/udev/rules.d/z25_persistent-net.rules.

Enjoy!

Let’s create an example situation.  You have two machines a application server and a database.  These machines do a lot of network traffic to each other.  You want to ensure they have the best possible performance.  In vmware worlds there are a few tricks to help you out.  First make sure your using the latest vmknic available.  After you have upgraded the nic’s the only thing left is to take advantage of the vmware ring buffer.

What is the vmware ring buffer?

Since virtual switches are defined as software dumb switches they operate in ram.  Ram is really fast.  So if you can send a message without leaving your virtual switch it can be really fast.   The ring buffer exists on each esxi host and where possible allows guests to communicate with each other (layer 2 so they have to be in the same subnet)  without involving anything but RAM.  You can see a 2X performance gain by just using the ring buffer.

How do I use it?

Keep your talking vm’s on the same host using affinity rules and your all set.

I ran into this issue last week.  I was preparing a new cluster for VXLAN and failed to open the firewalls to the vib could be pushed out to the ESXi hosts.  This caused the prepare for network vitalization to fail.  When I tried to remove it everything looked ok until I prepared again… this threw the following error:

domain-cxx already has been configured with a mapping.

This is a big problem and threw me for a loop.  VMware support was able to help… and I suggest unless you don’t care about your cluster or vShield implementation that you call them to solve it… but here is the fix I have used many times to resolve this issue:

Login to a vCloud cell (or download curl to your pc)

Run the following command:

curl -i -k -H "Content-type: application/xml" -u admin:default -X DELETE https://<vsm-ip>/api/2.0/vdn/map/cluster/<domain-cXXX>/switches/dvs

Replace the admin:default with your username:password for vShield and the vsm-ip with your vShield IP address.  Replace domain-cXXX with the domain name from the error.  Remove all < and > they are just to show where I placed the variables.  Now your ready for a prepare again.