Trap events sometimes resolve the hostname and some times not

Hi Habadir,

Please see KB article here which describes the behavior you are seeing:

https://support.emc.com/kb/87278

Basically the INCHARGE-OI domain relies on the Smarts IP domain for topology information (as it does not have a discovery manager itself).

Run the following command to check if the device is available in the topology:

./dmctl -s INCHARGE-OI invoke ICIM_ObjectFactory::ICIM-ObjectFactory findComputerSystem 172.28.163.73

Is the device with IP address 172.28.163.73 discovered in your Smarts IP domain (INCHARGE-AM-PM)?

If so, do you have this IP domain added to INCHARGE-OI as a topology source?

To see this, open the Smarts Console and attach to the INCHARGE-OI domain.

Click Configure -> Global Manager Administration Console.

Drill into ICS Configuration -> IC Domain Configuration -> Domains.

Do you see your IP domain listed here? If so, is it enabled?

If not, add the domain by right clicking on Domains and selecting New Domain.

Kind Regards,

Paul O'Rourke

Hi Habadir,

In addition to my information above, please check the value of the: RESOLVE_IP parameter in the trap adapter configuration file (trap_mgr.conf).

This flag determines whether unrecognized IP Addresses will be resolved before being assigned to the $SYS$ variable.  For objects already known to the system, this option will have no effect.

This should resolve any IP addresses from incoming traps .However, as this name resolution is done by the trap adapter it will have an impact on performance if there are a lot of DNS lookups occurring (CACHE_HOSTS can be used to mitigate some of these performance drawbacks).

Thanks,

Paul O'Rourke

Thanks Paul for replying. Actually this IP device isn't configured in the topology as we don't worry about it other than it's part of an application infrastructure sending traps...

the INCHARGE AM-PM domain is listed and enabled in the INCHARGE-OI domain.

I see the RESOLVE_IP value is set to FALSE,

Again, the issue here is that nslookup is working fine from the commandline, and why some servers are resolved and some aren't?

Hi habadir,

Thanks for the update.

If you set both the RESOLVE_IP and CACHE_HOSTS parameters to TRUE in the trap adapter configuration file (trap_mgr.conf) and restart the trap adapter, then this should force DNS lookup for all IP addresses.

Note: When editing the trap_mgr.conf file, please use the sm_edit utility to update the file as this will ensure the changes are made to the file in the local directory, rather than the original file.

If you wish to troubleshoot why some IPs are resolving and others are not, then the next step is to set the follow parameters:

DEBUG = TRUE

LOGGING = ALL

Once you have received traps from devices resolving and not resolving, review the trap adapter log file and follow the flow of the trap processing. Do you see any differences?

Also, can you confirm that neither the unresolved and resolved IP addresses were listed in OI or IP domains?

For example:

./dmctl -s INCHARGE-OI get IP:172.28.163.73

./dmctl -s INCHARGE-AM-PM get IP:172.28.163.73

If these are listed, check the DisplayName attribute.

Can you please run the same commands with an IP address that is resolving?

Thanks,

Kind Regards,

Paul O'Rourke

Hello Paul it's still not working after enabling all of the entries in the trap_mgr.conf

Hello Paul,

Hi habadir,

As mentioned above, if you wish to troubleshoot why some IPs are resolving and others are not, then enable debug logging by setting the follow parameters in the trap_mgr.conf file and restart the trap adapter:

DEBUG = TRUE

LOGGING = ALL

Once you have received traps from devices resolving and not resolving, review the trap adapter log file and follow the flow of the trap processing to see how the trap adapter attempts to resolve the IP address.

Please upload the log file here if you it be to reviewed by EMC.

Thanks,

Kind Regards,

Paul O'Rourke

Paul,

I've enabled the debugging as suggested. I've attached this application log file

Hi Habadir,

We need the trap adapter log file, not the log file which the traps are written to after processing.

The attached file does not contain any debug information regarding the attempted name resolution...etc.

Can you please attach the trap adapter log file, which is located at:

/smarts/local/logs

Thanks,

Kind Regards,

Paul O'Rourke

Here go Paul

Hi habadir,

Thanks for the debug logs.

I can see that the trap adapter calls findComputerSystem when a trap is received:

[February 24, 2015 11:54:17 AM EST +410ms] t@2550139200 Trap-Driver-00
ASL_MSG-*-ASLP-icoi-trapd/trap_mgr_parse.asl: February 24, 2015 11:54:17 AM EST trap_mgr_parse.asl[00]: FIND_SYSTEM : Executing findComputerSystem(192.168.211.27)

The response comes from a cached value as there is already an existing entry in the topology, so the DNS lookup is not attempted:

[February 24, 2015 11:54:17 AM EST +413ms] t@2550139200 Trap-Driver-00

ASL_MSG-*-ASLP-icoi-trapd/trap_mgr_parse.asl: February 24, 2015 11:54:17 AM EST trap_mgr_parse.asl[00]: FIND_SYSTEM : Returning the cached value = 192.168.211.27

[February 24, 2015 11:54:17 AM EST +415ms] t@2550139200 Trap-Driver-00
ASL_MSG-*-ASLP-icoi-trapd/trap_mgr_parse.asl: February 24, 2015 11:54:17 AM EST trap_mgr_parse.asl[00]: NOTIFY_EVENT : Found it as an IP Address (agent): '192.168.211.27'.

This entry exists in the INCHARGE-OI topology as, from your trap rules, each time a trap is received from an an Unknown Agent, an instance is created in the topology:

[February 24, 2015 11:54:17 AM EST +438ms] t@2550139200 Trap-Driver-00
ASL_MSG-*-ASLP-icoi-trapd/trap_mgr_parse.asl: February 24, 2015 11:54:17 AM EST trap_mgr_parse.asl[00]: SysNameOrAddr:       '192.168.211.27'

[February 24, 2015 11:54:17 AM EST +438ms] t@2550139200 Trap-Driver-00
ASL_MSG-*-ASLP-icoi-trapd/trap_mgr_parse.asl: February 24, 2015 11:54:17 AM EST trap_mgr_parse.asl[00]: UnknownAgent:        'CREATE'

To resolve this issue:

1.) Stop both the INCHARGE-OI domain and the trap adapter:

./sm_service stop INCHARGE-OI

./dmquit -s TRAP-INCHARGE-OI

2.) Clean the INCHARGE-OI topology:

     a.) Use the following command:

./sm_server -n INCHARGE-OI --config=icoi --output --clean-topology --daemon

     b.) Once the OI domain starts and is listed in the ./brcontrol output, stop the domain again:

./dmquit -s INCHARGE-OI

3.) Edit the trap adapter configuration file(trap_mgr.conf) and change the caching value to FALSE (ensures fresh lookup each time):

CACHE_HOSTS = FALSE

Ensure that RESOLVE_IP is set to TRUE

RESOLVE_IP = TRUE

4.) Check if these IP addresses are listed in Smarts IP. Use the following dmctl command:

./dmctl -s INCHARGE-AM-PM geti IP | grep 192.168.211.27

If the instance is found, delete as follows from Smarts IP:

./dmctl -s INCHARGE-AM-PM delete IP::IP-192.168.211.27

5.) Start INCHARGE-OI as normal:

./sm_service start INCHARGE-OI

6.) Start trap adapter:

./sm_trapd --name=TRAP-INCHARGE-OI --server=INCHARGE-OI --config=icoi --port=162 --model=sm_actions --rules=icoi-trapd/trap_mgr_parse.asl --seed=seedfile --output=TRAP-INCHARGE-OI.log

Please let me know how this goes.

Kind Regards,

Paul O'Rourke

Hello Paul,

I tried that in a dev env but still.

questions:

1- Does the clean topology command will delete all topology entires?

./sm_server -n INCHARGE-OI --config=icoi --output --clean-topology --daemon

2-  I didn't find any of the IPs that are not resolving the dns name in INCHARGE-AM-PM but found them in the INCHARGE-OI

and when trying to delete it from either domain, didn't find it.

./dmctl -s INCHARGE-OI geti IP | grep -i 192.168.237.9

IP-192.168.237.9

./dmctl -s INCHARGE-OI delete IP::192.168.237.9

Server INCHARGE-OI User: admin

admin's Password: XXXXXXXXXX

MR-E-OBJECT_NOT_FOUND-Object class::name 'IP::192.168.237.9' not found; in file "/work/tancurrent/DMT-9.3.0.0/116/smarts/repos/mr/dyn_acc.c" at line 1773

Hi habadir,

1.) Yes. The clean topology option will remove all topology objects from INCHARGE-OI. INCHARGE-OI gets its topology from the topology sync with Smarts IP and the incoming traps which have the a rule with: UnknownAgent:        'CREATE'

Therefore, the INCHARGE-OI topology will be recreated.

2.) The command you used to delete the IP instances is not correct. The instance name is IP-192.168.237.9 , therefore the correct dmctl command is:

./dmctl -s INCHARGE-OI delete IP::IP-192.168.237.9

Kind Regards,

Paul O'Rourke