November 2015 – Joseph Griffiths

November 24, 2015

Phantom Ballooning on Virtual machine

Recently I ran into an issue where esxtop was showing a virtual machine that was ballooning even thou my ESXi host was in a high state (see this article for state information). A quick check of the guest operating system found the ballooning driver running but no additional memory than usual was used. (This virtual machine used a pretty static 3GB of RAM) I wanted to understand and duplicate this situation so I loaded up my home lab with two Linux virtual machines with 40GB of ram each (my ESXi hosts are 32GB each). I took the following steps:

Power on each virtual machine and bring up top
SSH to the ESXi host run esxtop and switch to Memory (M) and filter to virtual machines only (V) then add only the fields D, J, K, Q
Monitor the normal load of the virtual machines and ESXi host for 10 minutes

During this time I found that one of my virtual machines was using 3GB of ram and the other was .5GB of RAM. So I started to apply pressure. Knowing that I needed to create a soft state to force ballooning I added 27GB of synthetic ram usage to a single operating system (using linux command stress). I found that I quickly moved into a hard state and ballooning and compression began. After two minutes I turned off the stress application using 27GB and allowed the virtual machine to return to 3GB used (ESXi host 3.5GB used). The screen shot below was taken after 10 minutes of no memory pressure:

As you can see we are still showing ballooning even thou the operating system top showed it had returned to the requested original value of 3GB:

So we had a problem the guest was not ballooning but esxtop was showing ballooning. I can assume from this that ballooning is not reclaimed until requested by the guest again. I found that if I initiated a 25GB request the ballooning metrics in esxtop would be removed. So ballooning without active soft state can indicate over provisioned ram on a guest and that a soft state once existed.

What about vMotion

I wanted to test the effects of this phantom ballooning on vMotion as expected the metric is 100% cleared after a vMotion and not set again unless soft state is achieved on the destination esxi host.

November 24, 2015

Memory Management in ESXi 6

A good friend recently reviewed the what’s new in vSphere 6 course and has some questions. That generated a number of really great discussions and this blog article. At about the same time I had a customer asking why their virtual machine was showing ballooning even thou there was no memory pressure. This generated some research and though organization into this article:

Memory reclamation techniques

VMware uses a number of memory reclamation techniques when under pressure. The implemented methods get worse in terms of overall cost as the pressure increases. I don’t want to dedicate much time or duplicate others blog entries. The following processes are in use:

Transparent Page Sharing – This (pre 5.5 U4) used to share common pages between all guests on the same host. Now it’s shared pages on each VM only due to security concerns. See this article
Guest Ballooning – This is a in guest driver that asks the guest operating system for system resources to force the guest to swap intelligently
Memory Compression – Pages are compressed and stored on a cache setup on main memory (set to 10% of total memory by default)
Hypervisor Swapping – Hypervisor swaps pages to disk… major effect on performance.

minFree the state to rule them all

minFree is an internal metric used by ESXi to denote when the reclamation techniques should be used. Each technique is trigged once a limit of minFree is reached. minFree is based upon the total RAM available on a ESXi host. You determine minFree using these rules:

First 28GB of physical memory in host = 899MB
Add 1% of remaining physical memory to the 899MB value in step 1

For example here are some common minFree numbers:

Total RAM	MinFree
28GB	899MB
48GB	1099MB
72GB	1349.56MB
144GB	2086.84MB
244GB	3233.72MB

5.5 Overview with Memory

Each version of ESXi has memory states that are tied to which technique gets used in 5.5 the states are as follows:

High 100 % of minFree – TPS

Soft 64 % of minFree – Ballooning

Hard 32 % of minFree – Memory compression

Low 16% of minFree – Swapping

So on a 144GB host minFree =2086.84MB it would look like this:

Free	Used	State	Memory Reclamation method

2.03GB	141.97GB	High	TPS enabled
1.30GB	142.7GB	Soft	Ballooning
.65GB	143.3GB	Hard	Memory Compression
.32GB	143.68GB	Low	Swapping

This model worked well but as you can see the difference between hypervisor swapping and TPS is a very little 1.5GB. Two virtual machines could consume this at the same time making it impossible for TPS to break large pages down and save space. This was a common problem that your host would go from a high state directly to low and swapping. Normally when you reach swapping things go bad for your applications.

Changes in ESXi 6

vSphere 6 added an additional state to allow memory pages to be broken down (from perhaps 2MB to 4 kb) a lot sooner I believe this is due in part to two factors:

The change to TPS to no longer be between VM’s but only on the same VM – meaning very few large pages will be TPS but breaking into 4KB’s might provide some savings
The small overhead that ESXi gives between states – even on large memory hosts the difference between TPS and hypervisor swapping is 2GB’s or a single virtual machine

So the change introduces the clear state to replace the 5.5 high state as detailed below:

High 400 % of minFree – Large page break

Clear 100% of minFree – TPS begins

Soft 64 % of minFree – Ballooning

Hard 32 % of minFree – Memory compression – compressed and swapped out

Low 16% of minFree – Swapping

This new clear state allows the pages to be broken much sooner before TPS is enacted. The odd thing is every ESXi host I have ever seen is in a high state even before it gets to 400% or 100% of minFree. These are the documented levels but the state seems to be static unless a lower status is achieved.

How to identify which state you are in

Use esxtop on the ESXi host. Choose M for memory and the state is listed at the top right side:

You can identify current virtual machine ballooning, swapping and compression via esxtop:

November 18, 2015

Nutanix and Acropolis test drive for $1 an hour

Catchy title eh? Well I have been wanting to test drive Nutanix community edition for a while now. It allows you to set up a Nutanix cluster on almost any hardware. It also runs Nutanix’s new hypervisor Acropolis (KVM based). My desired to set this up has always been limited by my time and the need to clear out some hardware for the test. Nutanix was also kind enough to provide me with their training portal access for free. So I am able to learn about their products using their interactive learning system. It is by far one of the most advanced online teaching situations I have ever experienced. But no teaching tool is the same as playing with the real thing. We have Nutanix at work but I was not present for the original setup and don’t do much of the day-to-day configuration. So I wanted a low-cost play ground. When looking for online casino game like lucky irish slots, visit www.slotsbaby.com for more information.

Enter Ravello systems

Ravello made a huge smash this year at VMworld taking some of the best in show awards, in addition they offered all vExperts a free 1,000 hours per month access. Mix that with the 1,000 free hours I get as a RHCE and I have a lot of cloud space available. For those who have not used Ravello they provide a cloud front end to public cloud providers Amazon and google. Allowing you to set up nested hypervisors including Nutanix Acropolis and ESXi. You do have to bring your own licenses. The interface is simple and clean but very powerful. A perfect example of what cloud provisioning should be. In addition people can share templates via libraries for you to use.

Enter the Nutanix Library

It’s available here. So you login to your ravello account then visit that page and click add to library. Now you can deploy a complete community edition of Nutanix in the cloud with two clicks. It’s really impressive. Here are the steps to deploy:

Deployment

Select Library -> Blueprints
Select Nutanix Community Edition
Select Create application
Name your application
Click on the Nutanix CE icon in the center of the screen

On the right side information will be presented about the virtual machine (that will run everything included nested virtual machines
I wanted to make mine accessible via the internet (yes it’s very insecure)
Click on services
Under each service select Advanced and Enable SNAT
After they are all enabled click save at bottom
Now click the publish button
Select optimize for performance
You can select to auto power down after xx hours to avoid costs
You can see your billing rate per hour for your server (this is per hour of turned on server not deployed)

So you can see I am able to run this virtualized Hypervisor for $1.0131 per hour.

It will take a few minutes to boot up and generate all required SSL keys for the first time (mine took about 20 minutes before it was totally ready to go)
Once it’s booted up select application and your application name
The summary tab will show the status and ports
Click open on the 9440 and you should be presented with Nutanix login

First time login is admin:admin and will require you change the password. If all goes well you are now able to deploy nest virtual machines on your hypervisor. It does require that you have a free Nutanix account… it even offers to let you sign up live. For bonus points deploy two of them and get them to replicate the virtual machine. The sky is the limit.

Wrap up

I am personally really amazed at how easy it all worked. It took literally hours of configuration and fiddling (which would be a great learning experience) into 20 minutes. Both Nutanix and Ravello should be commended for these awesome services. I wish that other vendors could provide a complex POC in 20 minutes. I think Ravello has a major future in the market. Give it a try it’s worth $20 to play with it. Let me know what crazy things you try. In know in the future weeks replication is in my mind. A quick guide to Nutanix community edition can be found here.

November 16, 2015

NSX Controllers all show as disconnected

I was recently upgrading my home lab to the newest version of NSX. Since it’s my home lab I didn’t backup or snapshot before I did the upgrade. Don’t try this at work. The upgrade of the NSX manager went fine but the controllers were all disconnected. I logged into all three of the NSX controllers (running 6.0) and found them all to be in this state:

As you can see they are all showing waiting to join majority with no cluster id. I attempted to force the first machine to join it’s self using

join control-cluster 192.168.10.29 force

This command rips out previous cluster configuration and reconfigures. That node came back as normal and became the master. I then tried to force the other nodes. Once they finished everyone was disconnected again. I then removed two controllers and tried to force the single into being the master. This seemed to work but when I tried to add a controller it failed again. This left me with a few choices:

Wipe out NSX and start from scratch
Try something else

I went for something else with a wipe out fall back. I figured since the logical switch know their own config without the controllers they would be ok as long as nothing changed. They were set to communicate updates via unicast mode. I switched them to multicast (yes it works in my environment) and then ripped out my last controller (you can switch it on the transport zone instead of each switch). I then deployed a new set of controllers one at a time. I configured the transport zone back into unicast and everything seemed ok. I also redeployed the edge gateways to complete the upgrade (I don’t think this was essential to the process). I hope it helps you if you failed to back up before an upgrade gone bad.

November 4, 2015

ESXi Thousands of failed logins from 127.0.0.1

A co-worker brought this to my attention. We run a daily vCheck on all clusters and it identified thousands of failed logins like this:

info 'Vimsvc.ha-eventmgr' opID=hostd-2bce] Event 11090 : Cannot login user @127.0.0.1: no permission

They all appeared in the /var/log/hostd.log. It was identified as caused by the script:

/sbin/hostd-probe ++group=host/vim/vmvisor/hostd-probe

which runs out of crontab on ESXi hosts every 5 minutes. (/var/spool/cron/crontab/root). This would cause the problem every time. The problem is cause by the dcui user not being in the /etc/vmware/hostd/authorization.xml file. You can see a healthly example of a ESXi host without any local users except the default here:

<ConfigRoot>

  <ACEData id="11">

    <ACEDataEntity>ha-folder-root</ACEDataEntity>

    <ACEDataId>11</ACEDataId>

    <ACEDataIsGroup>false</ACEDataIsGroup>

    <ACEDataPropagate>true</ACEDataPropagate>

    <ACEDataRoleId>-1</ACEDataRoleId>

    <ACEDataUser>dcui</ACEDataUser>

  </ACEData>

  <ACEData id="12">

    <ACEDataEntity>ha-folder-root</ACEDataEntity>

    <ACEDataId>12</ACEDataId>

    <ACEDataIsGroup>false</ACEDataIsGroup>

    <ACEDataPropagate>true</ACEDataPropagate>

    <ACEDataRoleId>-1</ACEDataRoleId>

    <ACEDataUser>vpxuser</ACEDataUser>

  </ACEData>

  <ACEData id="17">

    <ACEDataEntity>ha-folder-root</ACEDataEntity>

    <ACEDataId>17</ACEDataId>

    <ACEDataIsGroup>false</ACEDataIsGroup>

    <ACEDataPropagate>true</ACEDataPropagate>

    <ACEDataRoleId>-1</ACEDataRoleId>

    <ACEDataUser>root</ACEDataUser>

  </ACEData>

  <NextAceId>18</NextAceId>

While my ESXi host was missing the dcui entry. So that account could not login to execute it’s function. I have some idea’s how it got removed but the fix is simple. Login to the host with the C# client and add the dcui user as a administrator. Then restart hostd:

/etc/init.d/hostd restart

Careful though restarting hostd does not always come back clean so I would move off any workload before the restart. I doubt many people will run into this issue but figured I would post in case to save people time when googling.

November 3, 2015

Can we judge a company by the quality of their documentation?

I had been thinking about this for a while. Before I look into joining a software company I ask to see their documentation for products. I have learned a lot about the future of the company and my interest from their documentation alone. Here are some thoughts:

But my product is so simple it does not require documentation

Yep I have heard that one before… allow me to translate that into my language… my product is so simple that it really should not be something you buy… nothing in IT is simple… you can write software to automate it and make it appear simple but it’s not. Write the documentation… explain your technology. Open your doors so we can geek out with you.

We try to reduce the nerd knobs to keep it simple

Translation = we don’t want you messing with our product because it will break and we will not support it. Fair point… if you know it breaks your product tell customers not to use it that way and why… Does it expose a weakness? yes it does… is that a problem? Depends on the weakness. Do you have something to hide? Sounds like it. Do we want simple IT… sure but everything integrated. I am a spider making a web of products connected. No one buys your off the self total solution.

We cannot allow our competitors to steal our IP so we cannot explain our tech

Get a lawyer everyone else has one… You don’t know they can reverse engineer your secret sauce in 10 minutes with the right people. Heck save reverse engineering just buy your lead developer.. Get a lawyer and protect your tech like everyone else.

We need you to sign a NDA to see our documentation

Yes that happens when the company lawyers up… see previous post and sign the document.

We don’t want our customers discussing this technology

Educate me and then don’t allow me to become your advocate… not smart. I am getting sick of tech gag orders. They don’t help anyone… you turn your potential supporter into a enemy for life… neat idea.

Can we judge a company by the quality of their documentation?

Yes you can. Too many companies take the view of you don’t need to know the secret sauce that makes our xxxx work. This model is created from an attempt to protect intellectual property I hope. The reality is it makes me mistrust your product. Every product has limits why try to hide them. Publish your products limits and strengths, explain you technology difference to help people make choices based upon what fits their needs. Stop hiding how it works. Stop making me go though a pay wall to get anything but white papers. If your product is great you have nothing to hide.

Advise to Companies

Stop giving me feature webinars start educating me on your tech. Product solid up to date technical documentation on your products. Create a living documentation source like a knowledge bank. Don’t put it all behind a customer only pay wall. If your support organization cannot provide a customer a solution by pointing to your documentation or knowledge bank add it to the KB. Create a community of customers via forums and social media and support them with rewards and assistance. I have really been impressed over the last two years with two companies documentation even thou they are at odds with each other at times:

Nutanix – insane level of documentation on everything and awesome training program
VMware – Huge amount of products, very well-developed community, lots of documentation on everything including teaching how to troubleshoot, great community forums

In both cases the marketing runs most of the webinars and presentations I see. They are all focused on the value proposition instead of the awesome tech. It’s possible as I rant that I am the only person who really want to understand the tech. Let me know if you agree or disagree.

November 2, 2015

Second VCDX VCDX5-CMA and lessons learned

I was very pleased to receive news on Friday morning that I had been awarded a second VMware certified design expert (VCDX) certification this time for Cloud Management and Automation.

Recently a coworker asked me my secret to career achievement. I told him outright it was a lot of hard work. It takes work every day. It’s not a one time achievement and you reach the end. It’s work every day. I am lucky my hobby and my job align. A lot of days I get to do what I really enjoy as a job. I can be found in my free time doing lots of the same things I do at work… without deadlines. It makes the work easier when you are having fun. Without purpose I find it hard to learn… so I started setting goals each year and promising myself I will achieve the goals the easiest goals were around certifications.

Backstory

I had not planned on working toward a cloud certification this year but when they removed the advanced certification requirements in March I tried for the one month challenge. A VCDX cloud application in one month. I produced a lot of documentation and lost all my weekends and vacation days to complete the application. Ultimately I was not happy with the results but fearing it would be my last chance without the advanced certifications I went for it. The result was a failed submission. I figured it was worth the gamble and it had not paid off. I expected the new VCIX certification to be announced by May removing all chance to submit again without taking the design and administration exams. They were not ready leaving the door open for another run at submission. My wife and I determined that we would take another run at it given the opportunity. I determined that this attempt had to be different from the last… it needed to not effect my family.

VCDX Game Plan

I had a solid submission for cloud it just lacked details. I needed to apply my thoughtful approach and magnifying glass to the submission. I needed to find a time that would not effect my family to work on the VCDX. My children are getting older and they notice me hanging out in the basement all day with the computers. I determined that the best time would be 7:30 – 8:30 Monday – Thursday and I would submit the application again when I was ready. This time was after my kids retired to their bedrooms and was traditionally my email and internet browsing / blogging time (mostly browsing as blog history will prove). I had to give it up… at 8:30 – bedtime it was my wife’s time. I had to shove the VCDX work into one hour a night. It was awesome. Every one of those hours produced meaningful results.

An Advocate for Balance

I have long been an advocate for balance in our lives. More hours does not mean more quality work. We need to have balance. For me it’s a challenge between work, career improvement, marriage improvement, being a father and personal time and hobbies. I have come to understand that the most important titles I will ever hold is husband and father, not VMware Certified design expert or something else. I have gained my knowledge one day at a time via hard work and will continue to learn but in thirty years my career will be over. The daily investment in my wife and children will continue to pay off. In the desire to provide and achieve we each need to grasp on to the most important things and make sure they are part of our life. Incremental investments in family pay off just like the hour at a time does. We all can be found doing many good things but perhaps we only have time for the best things. Choose carefully the best things. This is one of the main reasons I have not taken a new job since getting my first VCDX, I refuse to travel away from my family for long periods of time. I became a VCDX to force myself to learn for personal development, not for the job. I got my current job before I was a VCDX, it’s an awesome job working with people who teach me new things every day. I got my second VCDX without any plans to change jobs, it was about forcing me to learn and an opportunity. In both cases the journey to VCDX has taught me about hard work and infrastructure, but mostly about myself and the things that matter the most.

Advice for VCDX Seekers

There is a lot of advice on the internet around this topic. Everyone has tips, tricks and plans. As more people get this certification and the market for people with the certification grows information will just increase. Here is my advice:

Set goals but if life gets in the way adjust
Set time to work on the certification and time not to work on it… work only during your time frame assigned
Be sure you want the VCDX for the right reasons…. if it’s for a new job there are a lot easier ways… trust me. If it’s about fame… there is very little of that… do you really want to be hired because of your certification instead of merits?
Make sure you family understands and goal and agrees to it (my wife is happier about the second VCDX results than me because it was a family goal)
VCDX is not about perfect infrastructure … it does not exist it’s about aligning business requirements with design
Read the blueprint and use it… ignore most everyone else just follow the blueprint
If you are defending get some practice speaking in public… I recommend VMUG presentations
Spend some time helping others along their path
Get a VCDX mentor by visiting the vcdx.vmware.com page and locating mentors they cannot design it for you but they can help move you along the path (you might even see me there as a mentor)

End

Thanks for reading my rant. Please take it as advice to not kill yourself getting a certification or anything in life. Balance is the best approach. You may have to sacrifice your fantasy football, twitter, facebook or drinking buddies to achieve your goals but don’t sacrifice the most important things. I remember when I made the choice to become a missionary for two years when I was twenty, the life style required I not touch a computer for two years. I worried that technology would change too much in that time and I would be far behind everyone else, nothing was further from the truth. I learned a lot of life lessons while being a missionary including better work ethic and study habits. My ability to learn had increased along with a lot of other things. Sacrifice is an important life habit to happiness but don’t sacrifice the wrong thing though neglect.