So it seems that we have found yet another limitation of VIM as well as vCenter.
During an ongoing migration of our vCenter servers from 5x to 6x, we have observed instances where SolarWinds Virtual Infrastructure Monitor (a.k.a: IVIM 2.1.0) cannot get stats back for managed ESX hosts via the vCenter API.
Some stats for a subset of ESX that were migrated from a vCenter 5.0 environment into a vCenter 6.0 are showing hardware status and memory/CPu stats, etc.
Other ESX hosts may show up in the VIM tree under their new vCenter, but do not have any stats; and some are "orphaned" after being deleted and rediscovered using ICMP only... They never show up under the new vCenter server tree or as standalone ESX in the VIM tree (even several days after being rediscovered).
The tell is a specific error in the VIM.CollectorJobs_[xxxx].log file on the polling engine, that indicates VIM can't get his stats from the vCenter API because the operation is restricted:
SolarWinds.VIM.Pollers.VMware.VMwareService.Vim25.VMwareVim25Poller - VMware Job Polling node 10.221.12.53 - can't get performance counters. All values set to zero.
System.Web.Services.Protocols.SoapException: This operation is restricted by the administrator - 'vpxd.stats.maxQueryMetrics'. Contact your system administrator.
The issue appears to be a new imposed limit configured as a default in vCenter 5.5 and 6.0 that sets a cap of 64 "entities" (metrics) per query:
"In vSphere 5.5 Update 2d and later and vSphere 6.0, a limit is set to the number of entities that are included in a database query. The limit protects the vCenter Server database from receiving large queries. "
The rest of the detail on this issue, as well as a mthod for changing this limit (which does not discuss the potential risks of doing so) are listed in this KB:
After seeing the VIM error on a polling engines hitting a vCenter 6.0 server with a large number >135) ESX hosts under it, I cross-checked against another Orion instance running the same NPM and VIM release and patch level to see if it was also reporting the error.
On a VIM polling engine hitting another vCenter 6.0 server to poll stats on only about 35 ESX, I saw the same error.
***Now that I've attempted to frame up the issue; here are my questions:
1. Can VIM be configured to limit the stats queries to the vCenter API in such a way as to expect a result set =<64 metrics, with each subsequent query request a subset for the next set of ESX in the vCenter inventory?
2. Has anyone else observed this same problem when using "Poll Through vCenter" collection method in VIM?
3. Has anyone managed to successfully tune their vCenter 6.0 servers to accept an API query with a larger result set, and not suffered ill afffects in vCenter DB performance?
This is a deal-breaker issue with vCenter API polling if this limit cannot be adjusted without affecting the vCenter environment.