ESXi has some interesting techniques used in order to save memory. These techniques allow for memory over commitment and utilization.
Transparent Page Sharing
This is a process of identifying common items in memory. For example each operating system has a number of files it loads into memory to operate. These files are never changed but allow the system to run. For example windows has a lot of DLL files that are loaded and never changed unless Microsoft patches them. If you have two guest virtual machines with the same operating system these files can be shared in memory allowing for less memory allocation. For this reason it’s good to run as much of the same type of operating system / application together as possible. Creating a standard build and standard operating system allows for a huge gain with transparent page sharing. TPS is based on pages not specific files so the pages have to be exactly the same.
Memory Ballooning
Lots of people have discussed memory ballooning and many of them do a better job explaining it than I do. In order to explain ballooning it is important to have a few common terms:
- host = ESXi Server
- Guest = Virtualized server running an operating system (in this example we will use Linux)
- Reserved memory – memory that is guaranteed by ESXi to the guest it can never be swaped
In addition there are three times of memory:
- Active memory – Memory actively in use
- Idle memory – Allocated memory not currently in use
- Free memory – memory available
Guest Swapping
Each guest has memory and swap or paging capacity. When a process requests more memory than is available in free the operating system swaps out the oldest idle memory to disk. This is a very costly operation because of the speed of the disk. The guest operating system is in the best possible situation to choose which process should be swapped due to it’s knowledge of active processes. This is normal system swapping for all operating systems.
Host Swapping
Vmware hosts also has a .vswp file that is create when a guest is powered on. This file is stored with the virtual machine and is equal to allocated memory – reserved memory. This file is available for when memory becomes so over used that vmware has to swap. This is the worst type of swapping ESXi has no knowledge which pages on the guest are active, idle or free so it just guesses. This can really effect performance.
Remember that only TPS is in operation unless there is contention for memory.
Memory Ballooning
In order to avoid host swapping vmware implemented the balloon driver (known as memctl on esxtop). It is included with the vmware tools. What the driver does is work within the guest operating system and request memory pages. Since it is a driver it has high priority and does not have to return the memory. This then forces the guest operating system to swap to OS page files. Since the guest swap is better than a host swap this is a preferred operation. This can happen up to 65% of the guest allocated memory. In effect we are tricking the operating system into using less memory than it’s been allocated by having the balloon driver steal some RAM. This memory gain is then passed off to other hosts allowing for over commitment of RAM. The problem with ballooning is that it will eat into active processes if the need for RAM is too high, thus forcing the host swap. Ballooning can only be active if your running vmware tools and if your host has been up for a little while.
Ok so what steps does ESXi take in what order:
This only happens when there is memory contention:
- (State:High) If no there is no contention then it does nothing.
- (State:Soft) ESXi starts using the balloon driver to allocate memory from the guest OS up to 65% of allocated memory.
- (State:Hard) Vmware tries to do a compression on the memory page, if they can get 50% compression then it goes into a special memory location known as compression cache. (Can also be SSD)
- (State:Low) If compression cache is not possible then the memory is sent to the host swap .vswap file.
You can tell what state your host is in by using esxtop.
- Log into esxtop
- Press m for memory
- Find the state at top:
In my case it’s high state no memory management going on. The state is determined by a sliding scale configured by ESXi. Stored in the advanced variable Mem.MinFreePct. Once this limit has been reached the state will change until our host is under the limit again.
We can also use esxtop to see how much memory is ballooning on the same page as before
- Log into esxtop
- Press m for memory
- Press V for virtual machines only
- Find the MEMCTL/MB to tell us how much swapping we are doing.
You can see I am ballooning 633 MB’s.
How do I create an artificial situation to simulate ballooning and swapping?
Just put in a memory limit resource pool and choke your vm’s.
Memory Usage:
What is the difference between Consumed host memory and active guest memory?
- Consumed host memory – The amount of host memory allocated
- Active guest memory – Amount of memory actively in use by guest and applications.