Smarts ESM & VirtualMachines not monitored via the VMware API

Please consider moving this question as-is (no need to recreate) to the proper forum for maximum visibility.  Questions written to the users' own "Discussions" space don't get the same amount of attention and questions can go unanswered for a long time. 

You can do so by selecting "Move" under ACTIONS along the upper-right.  Then search for and select: "Ionix Support Forum" depending on what model you own.

Hello JWO,

I see that no one has provided you with any feedback on whether they have encountered this issue or not.  I do not have the SR number for the ticket you had open with support so I do not know the full details.

Let me make sure I am clear on what you are seeing:

1:  When discovering virtual machine hosts in ESM via SNMP that the VMVirtualizes relationship or SNMPAddress set are not populated.

2:  You set these values yourself via a hook script but have found that these hosts that were not discovered via the VMwareAPI are being deleted by the ESM post discovery.

When virtual machine hosts are discovered using SNMP and DO NOT have the VMVirtualizes relatiohshop and SNMPAddress set set by your hook script are they still being deleted by the post processing?

As it states in the documentation.  SNMP should not be used to do discovery unless there is no vCenter or the credentials are not available.  This can be seen on Page 28 of the ESM 9.0 documentation portfolio in the EMC Ionix Server Manager Version 9.0 User Guide.pdf.

I am not completely sure why it is explained in the documentation.  When you intermix the discovery between VMwareAPI and SNMP there are problems.  There is no way for the snmp probe in the Smarts software to distinguish between which devices were discovered via SNMP and which where discovered via VMwareAPI.  The VMwareAPI is more robust than the SNMP and allows for better Topology analysis, notification monitoring, and more accurate RCA (root cause analysis).

When the environments are intermixed there is a strong possibility of the SNMP probe over-writing a VMwareAPI discovery with SNMP based discovery information or creating duplicate topology objects with different names.  The solution to having a mixed discovery is very complex.  All ESX servers (with exception to ESXi 4.0) are supposed to have SNMP turned on.  So the risk is that anything discovered via VMware API may end up getting rediscovered via SNMP and having an over all negative impact on monitoring.  The view was very few, if any, customers would be using SNMP for ESX discovery once we implemented vCenter discovery.

As you can imagine this could cause needless data duplication in terms of topology objects, notifications, and monitoring, as well as problems with topology inconsistency between discoveries.

Your best option is to proceed with having one ESM domain that is discovering all your SNMP ESX hosts and another ESM domain that is discovering all your VMwareAPI ESX hosts (vCenter discovery).

I hope this information is helpful.

Cheers,
Sean

Hi Sean,

On point 1:

These Virtual Machine hosts and their respective VMware ESX servers are discovered via SNMP in the IP Suite.  We did not enable the VMware SNMP DiscoveryDriver Probe in DISCOVERY_VMWARE.import for ESM.  We do not have access to the vCentre that manages these VMwareESX server or their related Virtual Machines, but still ESM classifies them as VirtualMachines and VMwareESX servers.  For these ESX servers the ESXNotManagedViaVC property is TRUE and DiscoveredViaSDK is FALSE.  These VirtualMachines do not get their VMVirtualizes & SNMPAddress properties populated even though the Hosts Virtualized by these VirtualMachines are in the Host class of ESM.  We have however, provided the VMware API credentials for another vCentre server that sits in a different network but forms part of the same topology that is managed by IP & ESM.

On point 2:

That seems to be the case.  The esm-post.asl script seems to check the DiscoveredViaSDK property and if it is FALSE, does what it calls BookKeeping and deletes the ESX servers and VirtualMachines, in effect nullifying our function call in a topology post-hook script.  We even noticed that when running this function call as part of post discovery in custom-end-post.asl, that it is not guaranteed that the custom-end-post.asl would run after esm-post.asl.  Turning debugging on showed debug output for custom-end-post.asl prior to the completion of esm-post.asl for example.

We then started up another ESM domain and did not provide it with any vCentre credentials.  All Hosts from the IP Suite were classified as Hosts in ESM and ESM did not create any VirtualMachine or VMwareESX classes which seems to limit ESM's use without the availability of a vCentre server's credentials.  I did not test whether ESM's insight into the VMware environment would change if I enabled ESM's VMware SNMP probe.

We did notice though that when we then provided this new ESM Domain with VMware API credentials to a particular vCentre server, systems not managed by the particular vCentre server were placed into the VMwareESX and Virtual Machine classes, but without the VMVirtualizes & SNMPAddress properties populated for these Virtual Machines.

We have not seen any situation with regard to duplicate topology objects.

Regards

JWO

Hello JWO,

I am currently trying to run a few simulations to get some answers for you.  But in regards to point 1.  What you are seeing makes sense and should be the expected behavior.  This is based off of how the ESM domain pulls information from the IP domain to build its initial topology.  The SNMP address list for a host in the IP domain is not stored in the host object but instead stored in the SNMP agent object.

It will take me some time to get you an answer because without spinning up a simulation and running a few tests to make sure everything is behaving as I expect it to behave the answer could end up very convoluted and confusing.

I will try to get you an more thought out answer by mid-week next week.

Thank you very much for your time.

Cheers,
Sean

Thanks Sean,

Thank you for all your feedback to date.

I've heard back from EMC Support whom have indicated that there used to be a feature available in ESM which allowed one to add the credentials for a standalone ESX server in a similar manner to ESM as one does with the vCentre credentials today.  I've asked for confirmation as to whether this functionality still works in ESM 9.2 as it may provide the ability to obtain the required information via the VMware API.

Regards

JWO

Hello JWO,

Based on my tests I would suggest you check the DISCOVERY_VMWARE.import in your /ESM/smarts/local/conf/esm and make 100% sure the SNMP discovery is disabled.  The reason I make this suggestion is in my testing I ran two IP environments, one with a vCenter and one without a vCenter.  Then I ran 2 ESM environments, one where I discovered the IP domain containing the vCenter and one where I discovered the IP domain that did not contain a vCenter.

Here are the resulting screen shots of the discoveries for each domain:

Here is the ESM without a vCenter:

As you can see in this image there is no virtual machine class and the vcenter credential screen is blocked out.

Here is the ESM with a vCenter:

As you can see these both look exactly the same.  The properties that are imported from the IP domain are the exact same as far as I could locate.  Even after running a full discovery.  Notice neither domain as a VirtualMachine class in it.

Here is the ESM domain after the vCenter credits were entered and a full discovery was run:

As you can see there is a large difference in the number of classes between the SNMP only ESM and the API ESM.  I double checked to make sure the SNMP discovery was disabled.  As you can see this domain now contains a class that says VirtualMachine, while the domain that does not contain a vCenter still does not (even after a second full discovery) have any VirtualMachine class.

So if you have no SNMP agent discovery enabled and no vCenter for API discovery ESM topology should mimic your IP topology almost exactly in regards to the properties and classes related to hosts and host topology items.  If you have no VMware API discovery going on and you have VMware classes in your domain then I would like to know how you accomplished this task, unless you used a customer script to build the classes.  If you used a custom script to do so then chances are it isn't grabbing all the necessary topology information, making the correct classes and sub-classes, and not filling in the properties/attributes correctly.

If you have any questions or anything else you can tell me then please feel free to do so.

Thank you very much for your time.

Cheers,

Sean

Hi Sean,

I've double checked and we do not have the SNMP discovery enabled in DISCOVERY_VMWARE.import.

I have the same findings as you, i.e. when importing topology from IP without providing any vCentre credentials to ESM, there is no VirtualMachine class.

When providing vCenter credentials, the VirtualMachine class is created and populated.  It is even populated by VirtualMachines not part of the vCenter for which we had provided credentials (but this is a minor issue for us).

Based on feedback from EMC support, we can add credentials for the stand-alone ESX servers in a similar fashion to how one would add the credentials for vCentre and this seems like a way to resolve the issue between which VirtualMachines runs on which Hosts and on which ESX servers.  We are looking at getting this tested on one of the stand-alone ESX servers under monitoring.

Thanks for all your responses to date.

Regards

JWO