ESXi Thousands of failed logins from 127.0.0.1

A co-worker brought this to my attention.   We run a daily vCheck on all clusters and it identified thousands of failed logins like this:

 

info 'Vimsvc.ha-eventmgr' opID=hostd-2bce] Event 11090 : Cannot login user @127.0.0.1: no permission

They all appeared in the /var/log/hostd.log.   It was identified as caused by the script:

 

/sbin/hostd-probe ++group=host/vim/vmvisor/hostd-probe

 

which runs out of crontab on ESXi hosts every 5 minutes.  (/var/spool/cron/crontab/root).  This would cause the problem every time.  The problem is cause by the dcui user not being in the /etc/vmware/hostd/authorization.xml file.   You can see a healthly example of a ESXi host without any local users except the default here:

 

<ConfigRoot>

  <ACEData id="11">

    <ACEDataEntity>ha-folder-root</ACEDataEntity>

    <ACEDataId>11</ACEDataId>

    <ACEDataIsGroup>false</ACEDataIsGroup>

    <ACEDataPropagate>true</ACEDataPropagate>

    <ACEDataRoleId>-1</ACEDataRoleId>

    <ACEDataUser>dcui</ACEDataUser>

  </ACEData>

  <ACEData id="12">

    <ACEDataEntity>ha-folder-root</ACEDataEntity>

    <ACEDataId>12</ACEDataId>

    <ACEDataIsGroup>false</ACEDataIsGroup>

    <ACEDataPropagate>true</ACEDataPropagate>

    <ACEDataRoleId>-1</ACEDataRoleId>

    <ACEDataUser>vpxuser</ACEDataUser>

  </ACEData>

  <ACEData id="17">

    <ACEDataEntity>ha-folder-root</ACEDataEntity>

    <ACEDataId>17</ACEDataId>

    <ACEDataIsGroup>false</ACEDataIsGroup>

    <ACEDataPropagate>true</ACEDataPropagate>

    <ACEDataRoleId>-1</ACEDataRoleId>

    <ACEDataUser>root</ACEDataUser>

  </ACEData>

  <NextAceId>18</NextAceId>

 

While my ESXi host was missing the dcui entry.    So that account could not login to execute it’s function.  I have some idea’s how it got removed but the fix is simple.   Login to the host with the C# client and add the dcui user as a administrator.  Then restart hostd:

 

/etc/init.d/hostd restart

 

Careful though restarting hostd does not always come back clean so I would move off any workload before the restart.  I doubt many people will run into this issue but figured I would post in case to save people time when googling.