After my memory management post there were a number of discussions about how memory ballooning works. It’s a rather neat way to free up memory by making the guest operating system choose which things to swap. There are three types of memory when discussing ballooning:
- Guest Operating system virtual memory – virtual memory mapping handled by the guest operating system requested by applications
- Guest Operating system memory – “Real memory” owned by the operating system – this would be the physical RAM on the operating system if it was not virtualized
- Hypervisor (ESXi) Memory – memory on the ESXi host mapped by ESXi and translated in parts into guest operating system memory
In order to understand ballooning it’s critical to understand how memory allocation works within a virtual machine:
- You start an application on your VM that requires 2GB of RAM
- Application requests 2GB of Guest virtual memory
- Guest operating system allocated system memory for the 2GB
- Hypervisor has Guest operating system memory mapped to physical RAM
- You stop application using 2GB of RAM
- The 2GB of Guest virtual memory is freed
- The 2GB of guest operating system memory is freed – known as free memory
- The 2GB of Hypervisor memory is not freed – in fact the hypervisor has not idea it’s not used – the operating system does not deallocate and erase 2GB of memory it just unlinks the memory
This creates an issue where the hypervisor is allocating RAM to a server that is not required.
For this discussion there are two types of memory on a guest:
- Active memory – in use memory
- Free memory – memory not used by operating system
What does the Ballooning driver do?
The drive which is part of VMware tools (no tools no Ballooning) is a process that can run with the highest guest privilege and request memory (runs as vmmemctl.sys). This process chews up free ram. The hypervisor as the initiator of this ram request understand which blocks are allocated in virtual and guest operating system memory and then deallocates them on hypervisor memory. Making that Hypervisor memory available for other guest operating systems.
Where does it go wrong?
Ballooning is normally a bad thing… because it does not just target free memory. It’s job is to remove memory pressure on the hypervisor to avoid hypervisor swapping (which is bad). It will eat into active memory on an operating system forcing the operating system to swap/page out active memory. This is better than hypervisor swapping but still a really bad thing for performance.
How much will the balloon driver eat?
Each operating system has a minimum recommended memory value (google minimum recommended memory for OS) for example RHEL 7 has 512MB’s. In addition to these values the advanced setting Mem.AppBalloonMaxSlack is added which is set to 16 MB by default. (it’s adjustable from 1mb – 256mb) So for my RHEL 7 system it will balloon all ram down to 528MB’s. The impact of this adjustment on high memory systems is huge while low memory may have no impact.
How can I avoid Ballooning?
- Avoid memory contention situations
- Avoid over provisioning servers
- Make a reservation on a server (reservation does not allow ballooning to claim any memory)
- Do not install VMware tools (Really bad idea)
Really the secret is just avoid memory ballooning. It’s better than hypervisor swapping but it’s still bad.