Why I took a pay cut to work at VMware

Warning:  This is a love rant for VMware.  You might want to skip if you are looking for the normal technical details of my blog.

Great title eh?  Really catchy and intended to get you to read and it’s true.    Two months ago I left a great job with IBM to work with VMware and took a pay cut to do it.   When you switch jobs there are lots of reasons money is cut part of the deal, you might leave a job for the following reasons:

  • Too much travel
  • Bad situation with management
  • No career growth potential
  • Money
  • A new challenge
  • etc…

The reasons are often a combination of these and other factors.   I left my job as a Senior VMware Architect for IBM to work as a Solutions Architect for VMware.   I wanted to take this blog post to explain why I took the job.   Some years ago I was a happy yet bored systems administrator.   My boss suggested that I attend an industry conference as a perk.   I went to VMworld.   I came back from the conference really excited about the future of VMware and the cloud.    I was invigorated by the energy and the vision of VMware’s executives.   I continued to learn about their technology and found it very refreshing in the market.   This lead me to focus my career away from Linux and into VMware technology.


I have taken two runs at VMware jobs in the past and did not make it.   Each time I was interviewed by multiple people.   Each of those people took the interview time to teach me new skills and help shape my thinking for success.   I love that attitude.   It’s a simple attitude that we are stronger as team instead of individuals.    Most of the company has this attitude and I love it.


VMware’s technology continued to push the limits of traditional datacenter while proving real business value.    They have proven to be innovative through aquisitions (NSX) and internal research and development (vSAN).   I love this approach too many companies stop internal research and loose the innovative spirit, this is simply not true at VMware.

Career Growth

VMware takes career growth serious.  Two weeks after starting with the company my boss asked me for specific goals that can effect my bonus structure.   These goals are recorded and tracked and VMware is serious about enabling me to meet these goals.   Managers seem to be interested in retaining talent by supporting growth and interests.

New Challenges

Yes it’s true I am a challenge junkie.  It’s what has caused me to get two VCDX’s in two years.  Once a goal is on paper I am a nut case to achieve it.   Working for VMware as a Solutions Architect represents some new challenges and lots of learning which does keep me going for a little while.

It’s all great when you are new

I completely agree I am very young in the company so my view is narrow.   I believe the future to be very good for VMware and I am excited to join them on the journey.

Storage in Virtualization is it a real problem?

As a VMUG leader and a double VCDX I have seen one technology trend only increase over the years.  It’s the number of storage vendors!   Last year at our VMUG UserCon every sponsor looking for a presentation slot was a storage vendor.   We had to choose between storage vendors and other storage vendors I would have killed for another type of vendor.  In past years we had presentations from Backup vendors, management tools, monitoring tools and IT service companies.   Now it’s all storage companies.   As a double VCDX I get contacted by start-up companies looking to sell their products to VMware customers.  Some are well known company’s others are still in stealth but they all have the same request… how do we get VMware guys to buy our awesome technology.   Almost all of these companies are using Super Micro white box solution with some secret sauce.  The sauce is what makes them different, some are web-scale while others are all flash or awesome dedupe ratios.   All attempting to address some segment of storage problems.  It really begs to question is there a storage problem?


What does storage provide?

Storage essentially provides two things that virtualization professionals care about:

  • Capacity (Space to store information)
  • Performance (divided into IOPS and latency)
    • IOPS – input/output per second number of commands you can shovel into the system
    • Latency – how long it takes to shovel each IOP end to end

There are subsections of software that each vendor provides in order to improve these metrics for example dedupe for capacity or Hot blocking for performance.   Essentially this is the role of storage systems to provide these functions.

How has virtualization made it worse?

Virtualization has made management of these metrics a challenge.   In traditional storage a single entity controls a LUN or mount.  It runs an application that has certain predictable patterns for usage of the lun.   For example, a web server does a lot of reads and a few writes.   We can identify and classify this usage pattern and thus “right size” the lun to meet these needs.  This right sizing can take the form of both capacity and performance metrics.   Virtualization created a new pattern lots of guest servers with different applications sharing the same lun.   This makes the usage metrics pretty wild.   The storage system has not idea what the virtual machines are doing beyond a bulk understanding of reads and writes.   This seems like a problem but in reality the storage system just see’s reads and write and does not care, unless capacity or performance for that lun are exhausted.    This issue might drive the acquisition of more performance storage in order to meet the needs of our new “super luns” but in most cases it just takes advantage of unused capacity on a storage array.

What does desktop virtualization have to do with storage?

Desktop virtualization taught us a very important lesson about storage.  During boot operating systems do a lot of IOPS.   Operating systems are 90% idle except during a boot.  During boot lots of reads and some writes happen putting pressure on disk.   Desktop virtualization introduced a new pattern of pressure on disk.  At eight and nine AM everyone would boot up their virtualized desktop (spawning new desktops and booting the OS’s) putting massive pressure on storage.   The caused storage systems to fail and if shared with traditional server virtualization everything failed.   Traditional storage vendor’s solution to this problem was buy a bigger array with more cache and capacity.  This created stranded capacity and was a huge CapEx expenditure when desktop virtualization was “supposed” to save us money.

Role of Cache

The rise of SSD has provided a dramatic improvement to the size of cache available in arrays.   Cache provide ultra-fast disk for initial write and common reads thus reducing latency and improving IOPS.   I remember the days when 1GB of cache was awesome these days’ arrays can have 800GB cache solutions or more.   Cache allows you to buy larger and slower capacity disks while getting better performance to the virtualized application.    Cache is a critical component in today’s storage solutions.

How to solve desktop virtualization

Vendors saw a gap in technology with desktop virtualization not being filled with traditional array vendors.    This gap can be defined as:

  • The array was not meeting my performance needs without buying more arrays
  • I need to separate my IOPS for desktop virtualization away from servers


This gave rise to two solutions:

  • Hyper-converged infrastructure
  • All Flash arrays




Hyper-converged infrastructure has many different definitions depending on who you ask.  For the purpose of this article it’s a combination of x86 hardware with local hard drives.  This combination provides the compute and software based clustered storage solution for virtualization.    The local hard drives on each compute node contribute to the required cluster file system.  This model has long been used by large service providers like Google and Amazon.  These are normally implemented for ESXi over NFS.  The market leader at this time is Nutanix who really cut their teeth solving desktop virtualization problems.  They have since moved successfully into traditional server virtualization.   Their success has encouraged other vendors to enter the market to compete including Simplivity (OmniCube) and VMware (Virtual SAN).   Each vendor has some mix of the secret sauce to address a perceived problem.   It’s beyond the scope of this article to compare these solutions but they all take advantage of at least one SSD drive as a per compute cache.   This local cache can be very large compared to traditional arrays with some solution using 1TB or more local cache.   Each compute node serves as a storage controller allowing for a scale up approach to capacity and performance.  Hyper converged solution have seen huge growth in the market and does effectively resolve the desktop problem depending on scale.  Hyper converged solutions do introduce a new problem; balanced scalability.   Simply put I may need additional storage without needing more controllers or compute capacity, but in order to get more storage I have to buy more nodes.   This balanced scale issue is addressed by vendors providing different mixes of storage / compute nodes.


All Flash Arrays

With the rise of SSD the cost keeps getting lower.   So traditional array vendors starting producing all flash arrays.   Flash provided insane amounts of IOPS per disk, but lower capacity.  Each month the capacity increases and the cost reduces on SSD making the All flash array (AFA) a very real cost effective solution.   Years ago I was asked to demo a newly emerging Flash solution called RamSAN.  The initial implementation was 150,000 IOPS in a single 2 U unit.   I was tasked with testing its limits.  I wanted to avoid artificial testing so I threw a lot of VMware database workloads at the array (all test of course).   I quickly found out that the solution may be able to do 150,000 IOPS but that my HBA’s (2 per host) did not have enough queue depth to fulfill the 150,000 IOPS.   All flash arrays introduced some new problems:

  • Performance bottleneck moved from the disk to the controller on the array
  • Capacity was costly
  • New bottlenecks like queue depth could be an issue

I remember buying 40TB’s of ssd in more recent array.  The SSD drives combined was capable of 300K IOPS while the controllers could not push more than 120K IOPS.    A single controller was able to do 60K IOPS.   Quickly the controller became my problem, one that I could not overcome beyond buying a new array with additional controllers.    Traditional array vendors struggled with this setup bound by their controller architecture.  A number of startup vendors entered the market with scale up controllers.  All flash based solution can potentially solve the desktop problem but at a steep cost.


Problem with both solutions

All solutions suffer from the same problems:

  • Stranded capacity in IOPS or storage capacity (more of either than you need)
  • Storage controllers cannot meet performance needs

All of these issues happen because of a lack of understanding of the true application metrics.   vCenter understands the application metrics the array understands reads and writes at a lun level.   This lack of understanding of each virtual machine as an independent element does not allow the administrator to increase priority or preference of individual machines.  Hyper converged have two additional challenges:

  • Increased network bandwidth for data replication (assuming Fiber arrays NAS have this issue)
  • Blades rarely have enough space for multiple hard drives

The value proposition for hyper converged is that you can replace your costly array with just compute with hard drives.  This is a real cost savings but only if you are due for a technology refresh on both compute and storage and your budgets are aligned and agreed to spend on hyper converged.  Getting storage to give up funds for a compute hard drive can be a hard proposition.


How to understand the smallest atomic unit

Lots of vendors understand this problem and have different ways of approaching this problem including:

  • VVols
  • Local compute cache
  • NFS

Essentially to understand the small you have to understand the individual files and how they are connected.   VMFS file system handles all this information, block based arrays only understand block based reads and write.   Individual files are invisible to the block based file system.



Developed by VMware VVol’s provide a translation method between block based storage systems using protocol endpoints.  These protocol endpoints run on the storage controllers or in-line with controllers to allow the array to understand the file system and individual files.   This translation allows the array to act upon a single virtual machine on a lun instead of running on the whole lun.   We can apply performance, snapshots and all array operations on the individual virtual machines.   This is a great solution but has two problems:

  • The protocol endpoints much like controllers have scalability issues if not implemented correctly
  • Vendor adoption has been very slow


Local compute cache

This process adds SSD or RAM and creates a cache for virtual machine reads and writes.  This cache can be assigned to individual machines or shared between the whole compute node.  This method has an understanding of individual machines and accelerate reads and writes.   In order to cache writes it’s critical that the writes be redundant so normally the writes have to be committed to at least two different compute nodes cache before acknowledged to the operating system.  This ensures that the data is protected during a single compute node failure.   The current leader providing read and write cache solutions like this is PernixData.  This process ensures local performance enhancement at the lowest atomic level but does endure some common challenges with hyper converged including:

  • Every compute node must have local SSD to accelerate solution
  • Network bandwidth for replication is used (meaning your need more 10GB or you have to share it)


NFS has been around for years.  It’s a method for sharing a file system to Linux and Unix hosts.   VMware supports it natively and it’s the only supported file system (other than VMware VSan) that is not running VMFS.  VM’s on NFS are files on the NFS file system.  This allows the storage array / server full understanding of the individual files.   This exposure can be a huge advantage when looking at backup products and site to site replication.    Until NFS version 4 support (vSphere 6) there were a number of draw backs to NFS including multipathing.  They have been removed and NFS provides the full object based storage solution that VVols promise.   Scalability can be a problem with a maximum number of virtual machines and objects on a single lun, or with capacity around controllers.   NFS based solution are network based and thus create network workload.  In addition natively NFS does not provide any performance by file enhancement method it just deals with IO in and out.   Lots of vendors have implemented solutions to enhance NFS.

What is best and does it solve the issue?

I started this post with the question is there a problem with storage… well lots of vendors seem to think so and want to sell us stuff to solve the issue.   I suggest that from my experience we have a few issues:

  • Backup is a major mess, in vSphere it’s hard to manage and keep working without constant care and feeding
  • Storage arrays don’t have any understanding of the lowest atomic unit and thus cannot protect us from bad neighbors on the same lun, this becomes more of an issue in large hosting environments.
  • Performance (IOPS) is rarely the issue except in specific use cases or small business thanks to oversized arrays
  • Queue Depth is rarely the problem except in specific use cases
  • Capacity seems to be the buzz problem and the price per year just keeps getting lower


I believe we need to get to object based storage so we can solve the backup problem.  Doing VDP backups or lun snapshots does not allow management at the lowest atomic unit.  The current model causes crashes and outages and struggles to work well.  It’s not a product issue it’s an implementation and technology issue that needs a dramatic change to resolve.

Local knowledge at the lowest level

The object I manage is a virtual machine.  My storage array friend manages a lun with multiple virtual machines (sometimes 100’s – yes I am looking at your NFS).  Until we manage at the same atomic level we will have problems aligning policies and performance.   I think policy based enforcement with shares is a great way to go… something like SIOC that is enforced by the array.    Hot blocking, all flash etc… are all fixes to attempt to get around the essential communication issue between arrays.   Future storage cannot be bound by two storage controllers it needs to scale to meet needs.   The hyper converged folks have a big advantage on this problem.    Future of storage is not block, except in mixed enterprise environments (I am looking at you mainframe).   You need to get comfortable with network based storage and architect for it.   Buy switches and interfaces on your compute just for storage traffic don’t mix it.  Architect a super highway to your storage that is separate from your normal network traffic.


If performance is your issue, then solve it locally don’t buy another array.  Local cache will save you a lot.   Scale up solutions in arrays or hyper converged are both options but local SSD will be a lot cheaper than a rip and replace.  It’s easier on management cost.

What should I choose?

It depends on your needs.   If I was presented with a green field that is going to be running all virtualized workloads today I would seriously consider hyper converged.  Storage arrays are more mature but move a lot slower on updates.  I would move toward a more software defined solution instead of hardware installed.   I think that central understanding of the lowest atomic unit is critical going forward.   If you have a mixed storage environment or an investment in fiber channel large arrays with cache makes sense.   If you are looking for solve VDI issues I would consider hyper converged or lots of cache.   The future is going to hold some interesting times.  I need storage to provide the following:

  • No controller lock in I need it to scale to meet my needs
  • It needs to understand the virtual machine individual identity
  • It should include backup and restore capabilities to the VM level
  • It has to include data at rest encryption (yes I didn’t mention this but it’s huge)
  • Policy based performance (allocate shares, limits and reservations)
  • Include methods to move the data between multiple providers (move in and out of cloud)

Does it sound like a unicorn… yep it is… Someone go invent it and sell it to me.


vRO Scriptable tasks

An old friend contacted me today with some vRO questions.  He is struggling with learning vRO like so many administrators before him.  It’s not easy to learn vRO it’s a very different type of programming.  Once you learn the basics of the editor and especially JavaScript in scriptable tasks it becomes really powerful.  So a few tips that I provided to my friend that may help you out when learning vRO:

  • JavaScript is case sensitive – this is hard for PowerShell or Windows users to remember
  • Variables are local to your scriptable task unless you make them an input or output (or both)
  • JavaScript variables don’t have defined types during creation (they are just pointers to memory locations)
  • Errors from JavaScript are not always very helpful (they point you to the incorrect or offending line)
  • You really should validate your input before taking an action


Case Sensitivity

I have rewritten a whole script to find out I missed a case for example a common one would be that is valid is:




Will all fail with a odd error.  You have to watch case on reserved commands.  This is true of API explorer objects VcVirtualMachine is VcVirtualMachine not VCvirtualmachine etc…


Javascript has lots of different variable types including user defined objects.   You can discover the object type using System.log.  Assume that my object / variable is called new.


Will output the type of object into the log window when run.  This is very useful for discovering the object type.  In the vCenter API you might have something like


If you want to understand your object type just use:


And it will tell you the object.  (this is a example and will not work)

Almost everything in vRO is a complex object which can be loosely defined as a combination of key value pairs in an array.   A traditional array have multiple objects referenced like this:

array[0] = "magic"
array[1] = "cheese"
array[2] = "need"

This is a array of strings with a length of 3.   If I called this:


The log would display “magic”

An object is an array with multiple key value pairs for each array element for example

array[0].firstname = Joseph
array[0].lastname = Griffiths
array[1].firstname = Tom
array[1].lastname = Bobo
array[2].firstname = Mac
array[2].lastname = Cheese

This allows me to store multiple elements that are connected and entries in the object don’t have to be the same type (e.g. firstname and lastname are both strings but I could add a age entry to hold a number).  This provides a huge flexibility… those familiar with powercli this is the same as:

get-vm | fl

This will display all vm’s with a full list of the elements on the object of vm.

Error Handling

Errors will display the line number.  Use that to determine the cause or source.   I know almost every programmer needs to have the ability to test their assumptions around variables when writing programs use the System.log for this:

System.log("variable 1: " + var1 + " variable 2: " + var2);

System log is readable only by someone in vRO and not exposed to customers (unless they are running it from vRO)

Input validation

First check for null

if (variable1 == null)
    //Null action

Then validate your object type using the System.log method above.  There is a way for checking for complex system defined objects like this:

if (vm instanceof VcVirtualMachine)


//Do some action


This can be very useful when working with objects validate before you act.


Hope it helps

vRo scriptable task to return the top level folder for a VM

Have you been nesting your virtual machines deep into folders.  If you want to return only the top level folder you can do it with this simple javascript function.  It can also be turned into a action if you would rather:

function return_top_folder(vm)
                //get the parent object for vm
                var parent = vm.parent;
                //create array to hold data
                var parentNames = new Array();
                var vmPathName = "";
                //create an array of VcFolder
                while (parent instanceof VcFolder){
                                //push folder names on top of array
                                parent = parent.parent;
                //get length of array
                j = parentNames.length;
                //Identify top level folder
                vmPathName = parentNames[j-2];
                //Identify vm's at root of vCenter
                if (vmPathName == null)
                                vmPathName = "root no folder";
                //Return folder name or root no folder
                return vmPathName;



Execute it by calling :

folder = return_top_folder(vm);

The vm must be an object of VCVirtualMachine.  Enjoy the scriptable task. You can download it here: topfolder

vRO Scriptable task issue with get all hosts

I was working with some scriptable tasks in vRO around hosts and ran into an issue calling the script.  It would fail when hosts were disconnected.  This brings up a good point that you should use good programming practices including checking for valid input (or null) in everything you do.  In this case I can filter my select into only hosts that are connected like this:


for each (host in hosts)


if (host.runtime.connectionState.value == “connected”)



Hope it saves you some time troubleshooting.


vRO scriptable task to expand a datastore

A quick vRO time saver.  Here is the scriptable task to expand a datastore.  This is not extents it’s an expand.  In other words the lun has expanded and you want the datastore to use newly available space.   In the next article I will show how to create a vRA service to expand the datastore.

datastorename = "your-test-datastore-name";

//Get all datastores
var ds = System.getModule("com.vmware.library.vc.datastore").getAllDatastoresMatchingRegexp(datastorename);

//Locate a ESXi host that connects to the datastore (from ds into host)
esx = ds[0].host[0].key;

//Get VimHostDatastore system to be used to gather path and extent
dsSys = esx.configManager.datastoreSystem;

//Get the local path to vmfsSystem
vmfsPath = ds[0].host[0].mountInfo.path;

//Identify free space on the lun and put into VcVmfsDatastoreExpandSpec
extend = dsSys.queryVmfsDatastoreExpandOptions(ds[0]);

//As long as extend is not null
if ( extend != null)
 extend_spec = extend[0].spec;
 //expand the volume to max size on disk
 System.log("Expanding Datastore : " + datastorename);
 dsSys.expandVmfsDatastore(ds[0], extend_spec);

//Rescan is initiated by the expand for all nodes mounting so it does not need to be done manually

You can download it here: datastore_Expand

Let me know if you have any questions.

vRO scriptable task to identify VMtools

One of the blog articles that gets the most hits on my blog is a article about how to get VMtools status from Powershell.  I figured I would share similar methods using vRO.  If you are wondering the Powershell method takes 10 minutes while the vRO method takes 10 seconds.  It’s pretty impressive.

First you create a scriptable task and get all virtual machines (this will be across all vCenters connected to your vRO instance)

//get list of all VM's
 vms = System.getModule("com.vmware.library.vc.vm").getAllVMs();

now the variable vms has array of VC:VirtualMachine containing all vm’s in your environment.

We now want to run through all vm’s one at a time using for each and check for two things:

  • Tools not running
  • Machine is powered on
for each (vm in vms)
if (vm.guest.toolsStatus == VcVirtualMachineToolsStatus.toolsNotRunning && vm.runtime.powerState.value === "poweredOn") {
         //do your reporting or action here
         System.log(vm.name + " tools is not running");


The VcVirtualMachineToolsStatus scripting class exposes a number of methods to check status including the following:

  • VcVirtualMachineToolsStatus.toolsNotInstalled
  • VcVirtualMachineToolsStatus.toolsNotRunning
  • VcVirtualMachineToolsStatus.toolsOk
  • VcVirtualMachineToolsStatus.toolsOld

As seen here:


These can be used to identify almost any status of tools.   You can then push your results into an array to use in vRO or vRA.  The full script can be found here: tools