Receiving SYSLOG (512) & SNMP (162)

Hi Mark,

The SAM suite has an adapter platform which can import topology and event information from sources other than

Smarts products. These sources include SNMP traps and syslog messages, but there can be many more.

Once these events (traps or syslog messages) are parsed by the adapter, the information is transferred to SAM, where it will be used in RCA and the generation of notifications.

ic-syslog-adapter and ic-trapd-receiver are adapters for incoming syslog messages and SNMP traps respectively.

Adapters are highly customizable and can be configured to discard messages, process messages, create notifications....etc. As you are new to SAM and the SAM Adapter platform, I suggest you review the documentation for the SAM Adapter platform. Please find the user guide attached. Documentation regarding other adapters is available here :

https://support.emc.com/products/6159_Smarts-Service-Assurance-Manager-Adapters/Documentation/

NOTIF on the other hand represents an improvement over the standard trap/syslog processing design using the adapter platform (as described above).

In order to configure this standard trap adapter to process traps, you must decide which traps should become notifications and how the notifications should appear at the Global Console. You do this by editing the trap_mgr.conf file and then restarting the server.

However, NOTIF makes this process much more user-friendly and robust. With NOTIF, you can make instantaneous trap processing configuration changes through the NOTIF GUI Editor without restarting any servers.

The user guide for NOTIF is also attached and is available to download on the SAM Documentation page:

https://support.emc.com/products/6175_Smarts-Service-Assurance-Manager/Documentation/

Please let me know if you have any additional questions.

Kind Regards,

Paul O'Rourke

Many thanks for that information, it is starting to make sense now...slowly!

Can I ask a few general questions that are bothering me thus far from reading the documents - I am worried about performance (adaptor/notif) and also configuration activities.

Performance

In my existing Netcool environment (which I am looking to migrate to SMARTS) my TrapD (SNMP) listeners process a lot of SNMP traps, around 50-100 per second (or more under flood conditions) - on a significant number of these traps a number of Boolean|Regexp test are made and based upon results the traps are enriched with data from Inventory - this type of load and enrichment is handled ok & operations teams see no significant lag in visualizing these events.

Configuration

In my environment I integrate an SNMP MIB into a configuration file called a "Netcool Rulesfile", this is a one-off process and basically we are ready to monitor that SNMP device, some MIBS can have 100s of traps. What worries me with SMARTS method is that whether we use NOTIF or trap_mgr.conf it seems that we have to configure for each trap whether we want that to be a Notification (matches ECI object) & then whether it is publshed (matches NCI object).

A Few Questions

1. How long would it take to integrate into SMARTS & for Notifications to be seen in a client - an SNMP MIB file that contained 800 traps?

2. How easy will it be to lift & shift a Netcool rulesfile that contains approx. 400 SNMP MIBS and is approx. 10k lines long & is full of Boolean|Regexp tests/enrichment?

3. Using Notif and/or trap_mgr.conf where would I interrogate the contents of the inbound raw SNMP varbind values, i.e. $1, $2, $3.....these may contain which disk number has failed.

thanks again, Mark.

Hi Mark,

You may need to involve professional services in your deployment as it could get quiet complex, especially as you wish to import your netcool rules into Smarts. This is outside the scope of our support here, but Professional Services may have implemented something similar previously. If you wish to involve Professional Services, please contact you EMC Account Manager.

Performance:

50-100 traps per second would be considered a very high trap rate, however, I have seen customers who are handling this traprate successfully in Smarts. Obviously your hardware is important, but also how the trap adapters are configured. For instance, you can create multiple trap adapters and then use a trap exploder to implement load balancing. In this configuration, the trap will first hit the trap exploder and, without processing the trap, the trap exploder will forward the trap to one of your trap adapters, usually based on source IP address.

Configuration:

As mentioned above, you may need the help of Professional Services for the intergration of your Netcool rules and Smarts trap adapter. We do not have a method of importing a SNMP MIB into the trap adapter or NOTIF. To import your rules may take some manual configuration, however, with the use of the wildcard character (*) , you should not need to create a rule for every single trap. Instead, create one rule for a group of 'interesting' traps which the trap adapter (or NOTIF) will process into a notification.

Other questions:

1.) As we do not import SNMP MIB files, this is a difficult question to answer. It mainly depends on the number of rules you need to create to filter for the 'interesting traps'. Using the wildcard character, you can simply create one rule and declare all 800 as 'interesting traps'.

2.) This is outside the scope of our support and would probably be better suited to a Professional Services engagement.

3.) In both NOTIF and the trap adapter, varbinds can be referenced using the following notation :

$V1$,  $V2$.....etc. $V*$ equates to all varbinds.

Kind Regards,

Paul O'Rourke

Thanks for those responses Paul, I have contacted our Account Manager and asked how we engage PS so hopefully I will be able to flesh out those issues further.

Can you help me with this little build issue, I am ignoring scalability,topology & performance issues for the moment as I just want to get something up & running so I can see how this all hangs together in my own mind & represents what I understand in Netcool.

I am trying to get an SNMP trap from a server into the adaptor and presented as a notification in the Global console notification log, here is what I have done thus far but been able to see the notification at the top.

I have the following componenst started:-

# ./sm_service show

RUNNING ic-sam-server

NOT RUNNING ic-syslog-adapter

RUNNING ic-icoi-server

RUNNING ic-broker

RUNNING ic-trapd-receiver

I have configured "/apps/InCharge/SAM/smarts/conf/icoi/trapd_mgr.conf" with the following line, which I am hoping will turn everything received on port 162 with an OID matching 594 and is a specific trap into a notification.

BEGIN_TRAP .1.3.6.1.4.1.594.* 6 *

       ClassName:             SNMPTrap

       InstanceName:          $V1$

       ElementClassName:      $V2$

       ElementName:           $V3$

       EventName:             SystemRestarted

       Severity:              3

       EventText:             System Restarted

       Expiration:            600

       State:                 NOTIFY

       UnknownAgent:          CREATE

END_TRAP

I have created a notification list that should capture the above notification, i.e. it is looking for a Severity between <1-5>, all I can see in the notification log are some ic-sam-server Critical DISCONNECT notifications - nothing from the server I can see hitting the "sm_trapd_en_US_UTF-8.log" below.

[November 13, 2012 9:38:43 PM GMT +244ms] t@11 Trap-Driver-00
ASL_MSG-*-ASLP-/apps/InCharge/SAM/smarts/rules/icoi-trapd/trap_mgr_parse.asl: November 13, 2012 9:38:43 PM GMT trap_mgr_parse.asl[00]: START : -> '0|xx.xxx.xxx.xx|.1.3.6.1.4.1.594.4.1.9.1|6|0|.1.3.6.1.2.1.1.1.0|Heartbeat message|.1.3.6.1.4.1.594.4.1.3.7.1||.1.3.6.1.4.1.594.4.1.3.7.4|FF 26 E3 2C |.1.3.6.1.4.1.594.4.1.3.7.5|FF 3F 40 B8 FF 36 07 28 |.1.3.6.1.4.1.733.6.3.18.1.5.0|1|.1.3.6.1.4.1.733.6.3.18.1.6.0|3E EF BB 0A |'

[November 13, 2012 9:38:43 PM GMT +261ms] t@11 Trap-Driver-00
ASL_MSG-*-ASLP-/apps/InCharge/SAM/smarts/rules/icoi-trapd/trap_mgr_parse.asl: November 13, 2012 9:38:43 PM GMT trap_mgr_parse.asl[00]: FIND_SYSTEM : Executing findComputerSystem(xx.xxx.xxx.xx)

[November 13, 2012 9:38:43 PM GMT +263ms] t@11 Trap-Driver-00
ASL_MSG-*-ASLP-/apps/InCharge/SAM/smarts/rules/icoi-trapd/trap_mgr_parse.asl: November 13, 2012 9:38:43 PM GMT trap_mgr_parse.asl[00]: FIND_SYSTEM : As findComputerSystem(xx.xxx.xxx.xx) is NULL, Returning = xx.xxx.xxx.xx

[November 13, 2012 9:38:43 PM GMT +266ms] t@11 Trap-Driver-00
ASL_MSG-*-ASLP-/apps/InCharge/SAM/smarts/rules/icoi-trapd/trap_mgr_parse.asl: November 13, 2012 9:38:43 PM GMT trap_mgr_parse.asl[00]: NOTIFY_EVENT : Cannot find it as an IP Address (agent): 'xx.xxx.xxx.xx'.

Can you see anything that I have missed in trying to get from A->B?

I have also configured in "/apps/InCharge/SAM/smarts/conf/icoi/trapd.conf" the following line but I am unsure if this is doing anything as surely the adaptor I have running is listening on port 162 anyway - I am guessing this is for when I am also running an adaptor as an "exploder".

FORWARD: * .* * * hws000a004:162

thanks, Mark.

Hi Mark,

From the log file snippet it looks like the trap receiver is processing the trap.

Is the trap appearing in the OI domain?

You can attach to the OI domain from the Notification Log Console by clicking Manager -> Attach. Ensure the broker details are correct and select the OI domain from the Manager drop down. Enter the authentication details and click OK.

Back on the Notification Log Console, select the OI domain from the Manager drop down. Do you see the trap notifications listed there? They may already be auto-acknowledged and/or archived if the last trap was received a few hours ago.

If the trap is appearing in OI, but not in SAM, this is probably due to SAM not subscribing to the OI domain. From the Notification Log Console select Configure -> Global Manager Administration Console. Select the SAM domain from the Manager drop down and then drill into ICS Configuration -> IC Domain Configuration -> Domains. The OI domain should be listed here and should be enabled. If it is not listed, right click on Domains and select New Domain. Enter the OI domain name and select Next. Select "Select Existing Type" and then from the drop down, select INCHARGE-OI-SUITE. Click Finish. Hit the Reconfigure button to reconfigure the SAM domain.

The notifications should appear in SAM once the reconfigure is complete.

If you are not seeing notifications in the OI domain either, then it must be a configuration issue. Can you please provide the following information:

1.) Output of ./brcontrol command.

2.) Output of ./sm_service show ic-trapd-receiver command

3.) Full sm_trapd_en_US_UTF-8.log file (zipped if it is large).

4.) trapd_mgr.conf and trapd.conf files.

Are you using sm_edit to edit the trapd_mgr.conf and trapd.conf files? If so, the relevant version of the file will be in the local directory ("/apps/InCharge/SAM/smarts/local/conf/icoi/"). Files in the local directory take precedence.

As regards the forwarding statement, you are correct that this should only be relevant if you are using the trapd service as a trap exploder.

Kind Regards,

Paul O'Rourke

Unfortunately Paul from my console installation I only have access to the ic-sam-server, I am still awaiting firewall configurations to get that connectivity down to OI & Trap Adapter – so I cannot check if the notifications are at that lower level unless there is another way to do this locally.

I installed/started the receibver with the following startup parameters.

/apps/InCharge/SAM/smarts/bin/sm_service install --force --unmanaged --startmode=runonce --name=TRAP-RECEIVER --description='SMARTS Trap Rec

eiver' /apps/InCharge/SAM/smarts/bin/sm_trapd --name=TRAP-RECEIVER --server=INCHARGE-ADAPTOR-PLATFORM --config=icoi --port=162 --sport=86 --

model=sm_actions --rules=icoi-trapd/trap_mgr_parse.asl --output

I am configuring the trapd.conf files using sm_edit, the 2 files are in my /local/conf/icoi directory. At the moment I have trapd.conf configured with the following:-

PORT: 162

ENABLE_FWD: TRUE

FORWARD: * .* * * hws000a004:162

I am configuring the trap_mgr.conf files using sm_edit, the 2 files are in my /local/conf/icoi directory. At the moment I have trap_mgr.conf configured with the following – IS MY BEGIN TRAP regexp statement correct????:-

BEGIN_TRAP .1.3.6.1.4.1.594.* 6 *

ClassName: SNMPTrap

InstanceName: $V1$

ElementClassName: $V2$

ElementName: $V3$

EventName: SystemRestarted

Severity: 3

EventText: System Restarted

Expiration: 600

State: NOTIFY

UnknownAgent: CREATE

END_TRAP

Here is the output you asked for

1. ./sm_service show TRAP-RECEIVER

RUNNING TRAP-RECEIVER

Here is the output of brcontrol – I have changed the name of the TRAP services

Hi Mark,

The following dmctl command will give you the list of notifications currently in the OI domain:

./dmctl -s geti ICS_Notification

What is the name of your OI domain? From the service definition for the trap receiver I can see :

--server=INCHARGE-ADAPTOR-PLATFORM

The value for the --server argument should be the OI domain name. Is the name of your OI domain INCHARGE-ADAPTOR-PLATFORM? If not, can you please uninstall the service and then re-install using the correct arguments?

Yes, the trap definition in trap_mgr.conf looks correct.

Kind Regards,

Paul O'Rourke

I forgot to show you the same command output from the SAM DM

netnsg@hws000a004 # ./dmctl -s INCHARGE-SAM geti ICS_Notification

Server INCHARGE-SAM User: admin

admin's Password: XXXXXXXXXX

NOTIFICATION-Session_SESSION-APP-InChargeService__INCHARGE-SAM->APP-InChargeService__INCHARGE-AM-PM_Disconnected

NOTIFICATION-Session_SESSION-APP-InChargeService__INCHARGE-SAM->APP-InChargeService__INCHARGE-AM_Disconnected

My thoughts are something is stopping the notifications getting from ADAPTOR to SAM, you have already seem my trap_mgr.conf looks ok so I am a bit stumped, is trapd.conf ok – I have provided the key config below:-

/apps/InCharge/SAM/smarts/local/conf/icoi/trapd.conf

PORT: 162

ENABLE_FWD: TRUE

FORWARD: * .* * * hws000a004:162

**Should the above line forward to 162 or to the adaptor platform port 81

Domain Host Name Port PID State Last Change Time

Yes I can confirm that is the Domain name I gave to the OI platform – I was just see what I could configure – here is the output which seems to indicate there are Notifications in the OI

netnsg@hws000a004 # ./dmctl -s INCHARGE-ADAPTOR-PLATFORM geti ICS_Notification

Server INCHARGE-ADAPTOR-PLATFORM User: admin

admin's Password: XXXXXXXXXX

NOTIFICATION-SNMPTrap_HPMPCHECK_:AMBER,mpcheck_20in_20progress,14/11/12_2013_:42_:03_SystemRestarted

NOTIFICATION-SNMPTrap_HPMPCHECK_:AMBER,mpcheck_20in_20progress,14/11/12_2015_:38_:46_SystemRestarted

NOTIFICATION-SNMPTrap_Heartbeat_20message_SystemRestarted

NOTIFICATION-SNMPTrap_SMM_20Heartbeat_20RECHARGE[1440]_SystemRestarted

NOTIFICATION-SNMPTrap_Secure_20Shell_20Authorized_20Keys_20File_20has_20Changed_20dys000a013_20-_20sum_20has_20changed_20on_20ssh_20authorized_20keys_20file_20(is_2013959_20/_20should_20be_2042476)_SystemRestarted

NOTIFICATION-SNMPTrap_Secure_20Shell_20Authorized_20Keys_20File_20has_20Changed_20dys000a014_20-_20sum_20has_20changed_20on_20ssh_20authorized_20keys_20file_20(is_2061202_20/_20should_20be_2042476)_SystemRestarted

NOTIFICATION-SNMPTrap_Secure_20Shell_20Authorized_20Keys_20File_20has_20Changed_20hws000a004_20-_20sum_20has_20changed_20on_20ssh_20authorized_20keys_20file_20(is_2061202_20/_20should_20be_2042476)_SystemRestarted

NOTIFICATION-Session_SESSION-APP-InChargeService__INCHARGE-ADAPTOR-PLATFORM->APP-InChargeService__INCHARGE-AM-PM_Disconnected

NOTIFICATION-Session_SESSION-APP-InChargeService__INCHARGE-ADAPTOR-PLATFORM->APP-InChargeService__INCHARGE-AM_Disconnected

Am I looking in the right place for these notifications? in my Notification log console it says “INCHARGE-SAM” (which is what it is called when I run ./brcontrol) – I still do not see any Notifications ?

Mark.

Thanks Paul that has cracked it, all alerts now appearing in the SAM – a few things to consolidate now like use of ASL scripts, Lookups & Tables to get this working in a similar fashion to my Netcool environment (I know they are chalk & cheese but I need to compare first before looking at some of the more complex IP discovery/RCA capabilities) – then moving onto Syslog.

Many thanks, Mark.

No problem Mark.

Can you please mark this post as Answered? Thanks.

Kind Regards,

Paul O'Rourke

Hi Marktho45 ,

Need some help , we are trying to configure snmp traps for EMC  VMAX, we have configured from unsiphere but we don't know how to define traps in Mib file , do you have any idea how to edit or do you have any sample mib for VMAX ? I highly appreciate your reply