Memory Management in ESXi 6

A good friend recently reviewed the what’s new in vSphere 6 course and has some questions.  That generated a number of really great discussions and this blog article.   At about the same time I had a customer asking why their virtual machine was showing ballooning even thou there was no memory pressure.   This generated some research and though organization into this article:

Memory reclamation techniques

VMware uses a number of memory reclamation techniques when under pressure.   The implemented methods get worse in terms of overall cost as the pressure increases.  I don’t want to dedicate much time or duplicate others blog entries.  The following processes are in use:

• Transparent Page Sharing – This (pre 5.5 U4) used to share common pages between all guests on the same host.  Now it’s shared pages on each VM only due to security concerns.  See this article
• Guest Ballooning – This is a in guest driver that asks the guest operating system for system resources to force the guest to swap intelligently
• Memory Compression – Pages are compressed and stored on a cache setup on main memory (set to 10% of total memory by default)
• Hypervisor Swapping – Hypervisor swaps pages to disk… major effect on performance.

minFree the state to rule them all

minFree is an internal metric used by ESXi to denote when the reclamation techniques should be used.   Each technique is trigged once a limit of minFree is reached.  minFree is based upon the total RAM available on a ESXi host.  You determine minFree using these rules:

1. First 28GB of physical memory in host = 899MB
2. Add 1% of remaining physical memory to the 899MB value in step 1

 

For example here are some common minFree numbers:

5.5 Overview with Memory

Each version of ESXi has memory states that are tied to which technique gets used in 5.5 the states are as follows:

High 100 % of minFree – TPS

Soft 64 % of minFree – Ballooning

Hard 32 % of minFree – Memory compression

Low 16% of minFree – Swapping

So on a 144GB host minFree =2086.84MB it would look like this:

This model worked well but as you can see the difference between hypervisor swapping and TPS is a very little 1.5GB.   Two virtual machines could consume this at the same time making it impossible for TPS to break large pages down and save space.   This was a common problem that your host would go from a high state directly to low and swapping.   Normally when you reach swapping things go bad for your applications.

 

Changes in ESXi 6

vSphere 6 added an additional state to allow memory pages to be broken down (from perhaps 2MB to 4 kb) a lot sooner I believe this is due in part to two factors:

• The change to TPS to no longer be between VM’s but only on the same VM – meaning very few large pages will be TPS but breaking into 4KB’s might provide some savings
• The small overhead that ESXi gives between states – even on large memory hosts the difference between TPS and hypervisor swapping is 2GB’s or a single virtual machine

 

So the change introduces the clear state to replace the 5.5 high state as detailed below:

High 400 % of minFree – Large page break

Clear 100% of minFree – TPS begins

Soft 64 % of minFree – Ballooning

Hard 32 % of minFree – Memory compression – compressed and swapped out

Low 16% of minFree – Swapping

This new clear state allows the pages to be broken much sooner before TPS is enacted.   The odd thing is every ESXi host I have ever seen is in a high state even before it gets to 400% or 100% of minFree.   These are the documented levels but the state seems to be static unless a lower status is achieved.

 

How to identify which state you are in

Use esxtop on the ESXi host.   Choose M for memory and the state is listed at the top right side:

You can identify current virtual machine ballooning, swapping and compression via esxtop: