Clear Syslog notification

Hi Arbas,

You will need to create a new custom rule in the my_hook_syslog.asl file to handle these syslog messages.

As you mentioned, the notification is getting generated correctly when the Down syslog message is received.

The custom rule will need to match the string associated with the adjacency coming back up : (Tunnel306) is up: new adjancency, and clear the existing notification.

The SAM Adapter Platform User Guide (available here: https://support.emc.com/products/6175_Smarts-Service-Assurance-Manager/Documentation/), has information on how to edit this file and setup the custom rule.

Please let me know if you have any additional questions.

Kind Regards,

Paul O'Rourke

Hi Arbas,

Further to the above I have tested this with the example syslog messages you provided.

Adding the following statement to the end of the MODIFY_ATTRIBUTES rule will clear the syslog messages as you requested:

    if ( glob("*is up: new adjancency*", EVENTTEXT)) {

        CLEAR_SYSLOG = "TRUE";

      }

I have attached the my_hook_syslog.asl I was using to successfully test this customization.

Please note that the Event field in the parsed syslog notifications will need to be identical for the second notification to clear the first notification.

For instance the syslog message:

Nov 7 09:16:41 xxx.xxx.xxx.xx Local7.Notice xx.xxx.xxx.xx 248249: devicename: 249779: Nov 7 01:21:26.297 MAL: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor devicename(unresolved) (Tunnel306) is down: holding time expired

Will be cleared correctly by the following syslog message:

Nov 7 09:17:41 xxx.xxx.xxx.xx Local7.Notice xx.xxx.xxx.xx 248249: devicename: 249779: Nov 7 01:21:26.297 MAL: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor devicename(unresolved) (Tunnel306) is up: new adjancency

Using the attached script, the following text is parsed as the event field:

devicename: 249779: Nov 7 01:21:26.297 MAL: %DUAL-5-NBRCHANGE: IP

I was unable to use the my_hook_syslog.asl script provide in your initial post due to errors in the script.

Please let me know if you have any additional questions. Otherwise please mark this post as Answered.

Thanks.

Kind Regards,

Paul O'Rourke

Hello Paul,

Many Thanks for the response.

By adding below condition in the script it is only clearing the up messages. Our requirement is to clear down message and there should not be any up message.

  if ( glob("*is up: new adjancency*", EVENTTEXT)) {

        CLEAR_SYSLOG = "TRUE";

      }

Example

UP message which was cleared already exists

November 9, 2012 10:45:25 AM GMT+08:00 syslog_mgr.asl: Clear = TRUE

November 9, 2012 10:45:25 AM GMT+08:00 syslog_mgr.asl: Discard = FALSE

November 9, 2012 10:45:25 AM GMT+08:00 syslog_mgr.asl: Cleared DXA::Syslog::Device: Neighbor 20.23.10.1(unresolved) (Tunnel305) is up: new adjancency::SyslogEvent

November 9, 2012 10:45:25 AM GMT+08:00 syslog_mgr.asl: Status: Cleared DXA::Syslog::Device: Neighbor 20.23.10.1(unresolved) (Tunnel305) is up: new adjancency::SyslogEvent

But if there is no up message it will not clear existing down alarm from same tunnel .....

November 9, 2012 10:46:38 AM GMT+08:00 syslog_mgr.asl: Clear = TRUE

November 9, 2012 10:46:38 AM GMT+08:00 syslog_mgr.asl: Discard = FALSE

November 9, 2012 10:46:38 AM GMT+08:00 syslog_mgr.asl: ==============  Syslog attributes [syslog] ===============

Hope I am able to explain properly ......although by the required changes the Clear is set to TRUE for up alarms but it is only clearing up alarms not the down one,Also up alarm are still showing.

I have reattached the hook script. Please help me out .

Regards,

Arbas

Hi Arbas,

The hook script you attached does not have the lines mentioned above :

  if ( glob("*is up: new adjancency*", EVENTTEXT)) {

        CLEAR_SYSLOG = "TRUE";

      }

Is this the hook script you tested with?

Have you tested with the hook script I provided above? When I tested with this hook script, and used the syslog messages you provided in your initial post, the Down notifications were cleared successfully.

If the Down events are not being cleared, it is most likely due to the Event field in the parsed syslog notifications being slightly different between the Down and Up notification. 

Can you please attach the audit log for the domain in question so I can review the notifications generated by these syslog messages?

Thanks,

Paul O'Rourke

Hello Paul,

Thanks for the response.

Yes I have tested by adding below lines as shown in your script to our script.

  if ( glob("*is up: new adjancency*", EVENTTEXT)) {

        CLEAR_SYSLOG = "TRUE";

      }

The reason I have not deployed your script because below is the difference in our scripts,seems it parsing all "%DUAL-5-NBRCHANGE" message.

  if (glob("%DUAL-5-NBRCHANGE",SYSLOG_EVENTNAME)) {/

/* For those syslog message that needs to create event, set DISCARD to FALSE */

DISCARD = "FALSE";

/* Exception cases for discarding */

if (glob("*rsync*",MESSAGE)) {DISCARD = "TRUE";return;}

/* Determine Neighbour IP and Tunnel name from MESSAGE */

GET_NBR();

/* Get slice message */

GET_SLICE_MSG();

/* Set properties for Syslog Neighbour change event */

EVENTNAME = "SyslogEvent" ? LOG;

INSTANCENAME = HOSTNAME.": ".SLICE_MSG ? LOG;

}

I have update audit logs and /var/log message file. Thanks you so much for your inputs...The problem is I am not able to identify which part is checking up and down in message fields as its not menationed anywhere (Not even in the Event field). If I mentioned clear logic below this statement  %DUAL-5-NBRCHANGE",SYSLOG_EVENTNAME still it clears up alarms only. Also when I discard up messages its not cleairng down event just filtering up messages.

if (glob("*is up:*",MESSAGE)) {DISCARD = "TRUE";return;}

Thanks & Regards,

Arbas

Hello Paul,

Please check attached a comparision between your(Left one) and my scripts looks bit different.

As in my script, severity and level are also defined.

Hope this will help

Thanks & Regards,

Arbas

Hi Arbas,

Thanks.

I am reviewing this now.

What version of SAM/OI are you running?

Thanks,

Kind Regards,

Paul O'Rourke

Hi,

SAM is on version 8.1.3

Thanks Paul

Hello Paul,

May be our syslog_mgr.asl file could be different ? Although we are using standard one no customization...

Regards,

Arbas

Hi Arbas,

Sorry about the delay.

It took me some time to work through your script as it has some errors (incorrect placement of '/' characters) and problems with customizations (if statement was matching on a variable called syslog_eventname, even though this variable was never assigned a value).

I resolved these issues by removing the '/' characters and using EventName rather than syslog_eventname.

After making these changes the syslog messages were still not getting cleared.

This was due to the Discard variable being set to TRUE by default. This should be set false by default.

After making the above changes, the script is now creating notifications for the syslog messages which report the Tunnel is down: holding time expired and clear the notifications for the syslog messages which report the Tunnel is up: new adjancency.

Please find script attached.

Please let me know if you have any additional questions.

Kind Regards,

Paul O'Rourke

Hello Paul,

Thanks for your response.

I have tried with the attached script but it's not showing incorrect output.

As per script the Instance Name should be : HOSTNAME : SLICE_MSG but it's showing incorrectly like Deviceip_deviceip_Process ID    Example : 20.233.254.101_20.233.254.101_248249. I have checked and I think this is beacuse there is condition where discard is set to FALSE for  if (glob("%DUAL-5-NBRCHANGE",EVENTNAME)).

This condition does not satisfy becuase Eventname doesnot contain %DUAL-5-NBRCHANGE value so it is showing eventname and instance message shown in attached file. I have tried matching %DUAL-5-NBRCHANGE in SYSLOG_EVENTNAME and it generate notification correctly but did not clear the alarm.

Intially there were 2 errors for level and syslogevent which got rectified when i uncomment those in your script.

'SYSLOG_EVENTNAME' has not been assigned a value

'LEVEL' has not been assigned a value

Thanks

Arbas

Hello Paul,

Some more inputs from my side..

Tried putting Clear syslog condition under Modify attributes and it worked as expected but only once.

Again when I sent the clear syslog message,although the value was set to true but no Notification was cleared.

MODIFY_ATTRIBUTES {

} do {

DISCARD = "FALSE";

CLEAR_SYSLOG = "FALSE";

if ( glob("*is up: new adjancency*",MESSAGE))

{

print("MESSAGE clear = " .MESSAGE);

  CLEAR_SYSLOG = "TRUE"; }

______________________________

Event and Instance name issue has been resolved by adding below lines under Modify attribute.

EVENTNAME = "SyslogEvent" ? LOG;

EVENTTYPE = "DURABLE" ? LOG;

GET_NBR();

GET_SLICE_MSG();

INSTANCENAME = HOSTNAME.": ".SLICE_MSG ? LOG;

Till now I have tried so many permutation and combination but no success.

Thanks & Regards,

Arbas

Hi Arbas,

I've done some more testing with this and the reason why the notifications are not getting cleared is that the Notification generated during the Notify and Clear do not match.

The three fields which uniquely identify a notification are:

1.) Classname

2.) EventName

3.) InstanceName

In your script, Classname is always "Syslog", so this field matches. EventName gets set to "SyslogEvent" during the MODIFY_ATTRIBUTES, therefore this always matches also. However, it is the InstanceName field which is failing the match.

In the MODIFY_ATTRIBUTES section of your script, GET_SLICE_MSG() sets the SLICE_MSG variable to include the full syslog message, including the time/date of the syslog message and the text which indicates whether the tunnel is Up or Down. The SLICE_MSG is then used as part of the InstanceName variable.

Therefore the clear notification will not match the notify as:

1.) Time/Date of the of the syslog messages will not match

2.) The text will not match either as the first message will indicate that the tunnel is down, and the next will indicate the tunnel is Up.

I edited the revelant section of the code as follows(assigning SLICE_MSG to EVENTTEXT not InstanceName):

if (glob("*%DUAL-5-NBRCHANGE*",EVENTTEXT)) {

DISCARD = "FALSE";

if (glob("*rsync*",MESSAGE)) {DISCARD = "TRUE";return;}

GET_NBR();

GET_SLICE_MSG();

EVENTNAME = "SyslogEvent" ? LOG;

EVENTTEXT = HOST.": ".SLICE_MSG ? LOG;

if ( glob("*is up: new adjancency*", EVENTTEXT)) {

CLEAR_SYSLOG = "TRUE";

}

}

Can you pleese try this and let me know how it goes.

Kind Regards,

Paul O'Rourke

Hello Paul,

Thanks for your inputs.

I have tried printing EVENTTEXT which does not show any value due to this its not matching the below condition.

if (glob("*%DUAL-5-NBRCHANGE*",EVENTTEXT))

__________________________________________________________________

Logs output :

November 15, 2012 12:56:50 PM GMT+08:00 my_hook_syslog.asl: LEVEL =Notice

November 15, 2012 12:56:50 PM GMT+08:00 my_hook_syslog.asl: SYSLOG EVENTNAME =%DUAL-5-NBRCHANGE

November 15, 2012 12:56:50 PM GMT+08:00 my_hook_syslog.asl: HOSTNAME =Test_Device

November 15, 2012 12:56:50 PM GMT+08:00 my_hook_syslog.asl: EVENTTEXT =

November 15, 2012 12:56:50 PM GMT+08:00 my_hook_syslog.asl: Executing CUSTOM_RULE

November 15, 2012 12:56:50 PM GMT+08:00 my_hook_syslog.asl: Done with my_hook_syslog.asl

November 15, 2012 12:56:50 PM GMT+08:00 syslog_mgr.asl: Clear = FALSE

November 15, 2012 12:56:50 PM GMT+08:00 syslog_mgr.asl: Discard = FALSE

========== ICS_Aggregate attributes [syslog-mgr] ============

November 15, 2012 12:56:50 PM GMT+08:00 syslog_mgr.asl: Notified DXA::Syslog::20.202.135.10_20.202.135.10_1135551::EIGRP-IPv4 10: Neighbor 11.160

November 15, 2012 12:56:50 PM GMT+08:00 syslog_mgr.asl: Status: Notified DXA::Syslog::20.202.135.10_20.202.135.10_1135551::EIGRP-IPv4 10: Neighbor 11.160

Regards,

Arbas