Dell S4128T-ON -- OS10 10.5.5.5.105 -- SNMP Traps not sending

Hi,

Can you check if your VRF configurations are similar to the article: https://www.dell.com/support/kbdoc/en-us/000184302/dell-emc-networking-os10-how-to-configure-out-of-band-management?lwp=rt

Thanks for the help. Like the first half of the guide, I do have an IP assigned to the management port, and a management route set:

interface mgmt1/1/1
 no ip address dhcp
 ip address 10.2.100.253/24
management route 0.0.0.0/0 10.2.100.1

Also, like in the first half of the guide, I can verify that configuration:

OS10# show ip interface brief | grep man
Management 1/1/1           10.2.100.253/24      YES      manual       up          up

OS10# show ip route

OS10# show ip management-route
 Destination                   Gateway                      State                Source
----------------------------------------------------------------------------------------------------
0.0.0.0/0                     10.2.100.1                     Active              Static
10.2.100.0/24                 managementethernet             Connected           Connected

OS10# show running-configuration | grep route
management route 0.0.0.0/0 10.2.100.1

And, as expected, various management connectivity functions work fine, on the local management segment (e.g. NTP), and routed to other networks (SFLOW out, SNMPwalk in).

However, unlike the second half of the guide, I am not using a separate management and data VRF on the switch, as I don't use the switch for routing, so I don't have a L3 data plane to separate from my L3 control plan. Instead, all interfaces (only used for the control plane) are just in the default VRF:

OS10# show ip vrf
VRF-Name                          Interfaces
default                           Mgmt1/1/1
                                  Vlan1,10-12,21-22,30,41,50,60,68-70
management

Again, this doesn't seem to be a problem with the other services running on management interface on the default VRF. Of course, these services, have a source-interface clarifying command, i.e.

sflow collector 192.168.99.25 agent-addr 10.2.100.253 9995
snmp-server source-interface mgmt1/1/1
ntp source mgmt1/1/1

Assuming the `snmp-server source-interface` command doesn't apply to the `snmp-server host` function (and thus that function doesn't know what interface to use) is there somewhere else to clarify that network path? I see that there is a `snmp-server vrf` command that I can specify to default (below), but that doesn't seem to change anything (either enable traps flowing out, or stop sflow out, or snmp in).

snmp-server host 192.168.99.25 traps version 2c Comm2c snmp lldp config
snmp-server source-interface mgmt1/1/1
snmp-server vrf default

Does the snmp-server just not send out traps in this configuration (management port in default VRF)? In the linked documentation, it only seems to recommend separate data and control VRF in cases where the routing would conflict, i.e.:

WARNING:  Do not configure the same prefix in both the static route and management route. If the same prefix must be used, use management VRF for proper communication.

I do not have a static route set, nor for that matter an IP addressed assigned to any port/VLAN/etc except the management port.

In case there was any unseen conflict, I even experimented with making the management route more particular to hopefully avoid conflict with an unseen default 0.0.0.0/0 route:

management route 10.0.0.0/8 10.2.100.1
management route 172.16.0.0/12 10.2.100.1
management route 192.168.0.0/16 10.2.100.1

But, that neither adds nor detracts from the existing connectivity (i.e. working SFLOW, SMTP in, and non-working SNMP Traps out.)

Is there more clarification of what settings/connectivity is required for traps to work, and any particular test/verification I can do to troubleshoot their operation?

Thanks for any guidance!

Hello,

Looks like you have mgmt1/1/1 configured with ip 10.2.100.253.

From there you have a management route sending packets to gateway 10.2.100.1 (next hop router).

At 10.2.100.1, the packets are routed to another subnet where the snmp-server host is located snmp-server 192.168.99.25.

You are saying that you ran tcpdump on the trapbox [root@trapbox ~]# and you can see that the packets reached the destination snmp-server 192.168.99.25 (which listens on port 162). After that you say that you “call poll SNMP (UDP 161) from the trap collector box”, this means that the SNMP response was sent from the trap collect and may be got lost on the way back.

If the above understanding is correct, this means that on the way back the SNMP response has to go first to router where 10.2.100.1 is located and be routed back to our switch management subnet 10.2.100.x.

For this return routing to occur through the correct return path, you can check what is the Default Gateway Ip for the snmp collector box, because the return packets will go there first. Let say the default gateway is 192.168.99.1. From there the packets have to be routed back to subnet 10.2.100.1.

You can check if 192.168.99.1 and 10.2.100.1 are located on the same router and if the packets are reaching our subnet 10.2.100.x via intervlan routing.

Thanks for the help, @DELL-Charles R​. You got it right in your first paragraph, but you went off a bit in the 2nd paragraph.

I said that I can

... drop down to a shell on the switch and execute a manual `snmptrap` command:

OS10# system bash
admin@OS10:/home/admin$ sudo su -
[sudo] password for admin:
root@OS10:~# snmptrap -v 2c -c Comm2c 192.168.99.25 '' 1.3.6.1.4.1.8072.2.3.0.1 1.3.6.1.4.1.8072.2.3.2.1 i 123456

[root@trapbox ~]# tcpdump -i ens192 -n "host 10.2.100.253 and udp port 162"
12:38:48.260659 IP 10.2.100.253.55465 > 192.168.99.25.snmptrap: C="Comm2c" V2Trap(79)  .1.3.6.1.2.1.1.3.0=49371489 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.4.1.8072.2.3.0.1 .1.3.6.1.4.1.8072.2.3.2.1=123456

This proves that the SNMP trap traffic, which should be travelling over UDP port 162 is not blocked and has connectivity from switch to trapbox, across the router. Moreover, since UDP is a connectionless protocol, it doesn't need to talk back for a SNMP trap from the switch to get sent to the trapbox. However, I do have routing working back from the trapbox to the switch (e.g., the switch can receive SNMPwalk polls (UDP 161) from the trapbox, and I can SSH from trapbox to switch). 

Likewise, I can drop to shell on the switch in another console and run TCPDUMP there as well, and watch the manually sent trap packet go out the switches interface, while simultaneously watching in TCPDUMP on the trapbox and see the packet arrive at the trapbox:

## Manual trap test

# sent from switch
root@OS10:~# snmptrap -v 2c -c Comm2c 192.168.99.25 '' 1.3.6.1.4.1.8072.2.3.0.1 1.3.6.1.4.1.8072.2.3.2.1 i 123456

# monitoring send (outbound UDP port 162) from switch (in different console)
root@OS10:~# tcpdump -i eth0 -n "host 192.168.99.25 and udp port 162"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:55:56.047954 IP 10.2.100.253.60397 > 192.168.99.25.162:  C="Comm2c" V2Trap(79)  .1.3.6.1.2.1.1.3.0=82594268 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.4.1.8072.2.3.0.1 .1.3.6.1.4.1.8072.2.3.2.1=123456

# recieved at trapbox
[root@trapbox ~]# tcpdump -i ens192 -n "host 10.2.100.253 and udp port 162"
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
08:55:56.052803 IP 10.2.100.253.60397 > 192.168.99.25.snmptrap:  C="Comm2c" V2Trap(79)  .1.3.6.1.2.1.1.3.0=82594268 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.4.1.8072.2.3.0.1 .1.3.6.1.4.1.8072.2.3.2.1=123456

However, if I leave that TCPDUMP running on the switch, looking for any SNMP traps leaving the switch, on eth0, (other than one manually sent from the shell):

root@OS10:~# tcpdump -i eth0 -n "host 192.168.99.25 and udp port 162"

I never see any traps going out, despite having this running config:

OS10# show running-configuration | grep "ip |vrf|manage|mgmt|ntp|sflow|snmp|trap"
ip vrf default
ip vrf management
ip http vrf default
interface mgmt1/1/1
 no ip address dhcp
 ip address 10.2.100.253/24
management route 10.0.0.0/8 10.1.50.1
management route 172.16.0.0/12 10.1.50.1
management route 192.168.0.0/16 10.1.50.1
sflow enable all-interfaces
sflow polling-interval 20
sflow sample-rate 4096
sflow collector 192.168.99.25 agent-addr 10.2.100.253 9995
snmp-server community Comm2c rw
snmp-server contact "Contact Support"
snmp-server enable traps snmp linkDown
snmp-server enable traps snmp linkUp
snmp-server enable traps snmp authentication
snmp-server enable traps config
snmp-server enable traps lldp rem-tables-change
snmp-server host 192.168.99.25 traps version 2c Comm2c snmp lldp config
snmp-server source-interface mgmt1/1/1
snmp-server vrf default
ntp server 10.2.100.12
ntp source mgmt1/1/1

... which leaves the question, why isn't the switch sending out traps itself? Or, at least, if it's not sending out traps (that I can see on eth0), is it sending them out a different interface?

Does anything look wrong on the above config? With it running, shouldn't I at least see some outgoing trap packets on the switch's eth0 (like I can when sending a manual trap from the switch's shell?)

Thanks for any suggestions!

Hello,

About the switch configuration, set the “snmp-server vrf management” and the “interface management” to be under “ip vrf management” as in the following example:

ip vrf default

!

ip vrf management

interface management

!

snmp-server community dell ro

snmp-server contact "Contact Support"

snmp-server enable traps entity

snmp-server enable traps snmp authentication

snmp-server enable traps snmp warmStart

snmp-server enable traps snmp coldStart

snmp-server enable traps snmp linkDown

snmp-server enable traps snmp linkUp

snmp-server host 192.168.99.25 traps version 2c dell snmp envmon dom

snmp-server source-interface mgmt1/1/1

snmp-server user dellos10user dellos10group 2c

snmp-server vrf management

management route 0.0.0.0/0 10.2.100.1

@DELL-Charles R​, Thanks for the reply!

Is it really necessary for the management interface to be in a separate VRF for the snmp-server to send out traps? If so, is that a unique requirement to the trap function -- since I have working sflow out the management port on the default VRF, and inbound SNMP to the management port on the default VRF?

I ask especially, since in the docs linked earlier -- Dell EMC Networking OS10 How to Configure Out Of Band Management | Dell US -- it says:

Configuring Management VRF

...

WARNING:  Remove all the configured settings, including the IP address, on the management interface before adding to management VRF.

So, wouldn't this test require me to drop IP connectivity, get physical access, and connect via console cable, which is non-trivial.

Again, in the documentation, it doesn't prescribe using a separate VRF for management; it just says:

NOTE:  When you configure Management route ensure the following is followed

- Nexthop must be within same subnet
- Prefix cannot be exact same subnet as assigned IP

And

WARNING:  Do not configure the same prefix in both the static route and management route. If the same prefix must be used, use management VRF for proper communication.

For example: Consider we have to configure a default route in default VRF and management VRF.



Here the management default route will be ignored.
For such situations use management VRF

... which is consistent with my configuration, since 1) I don't have assigned data plane IPs on the switch, and 2) I'm using more specific management interface routing than a default route.

Hello,

We provided snmp configuration that will send the packet out the management interface for sure.

You have an “ip route 0.0.0.0 ..” configured and currently the packet may be trying to go out via the data interface not management.

Making configuration changes, can impact other applications. We do not fully understand your network connectivity and design, which applications have to go via management interface and which applications via another interface. If making management interface changes, there should be a console connection, and downtime scheduled, configuration should be backed up in order to restore if needed. For such wide scope questions, it may be best to open a ticket with Dell Support or contact our Deployment team.

@DELL-Charles R​, actually, all of a sudden, I started seeing some traps getting received on the trapbox -- looks like a trap triggered for a port going up and down:

[root@trapbox ~]# tcpdump -i ens192 -n "host 10.2.100.253 and udp port 162"
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
11:20:15.522964 IP 10.2.100.253.45968 > 192.168.99.25.snmptrap:  C="Comm2c" V2Trap(135)  .1.3.6.1.2.1.1.3.0=83460215 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.6.3.1.1.5.3 .1.3.6.1.2.1.2.2.1.1=38 .1.3.6.1.2.1.2.2.1.7=1 .1.3.6.1.2.1.31.1.1.1.1="ethernet1/1/22" .1.3.6.1.2.1.2.2.1.8=2
11:20:19.098196 IP 10.2.100.253.53037 > 192.168.99.25.snmptrap:  C="Comm2c" V2Trap(135)  .1.3.6.1.2.1.1.3.0=83460572 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.6.3.1.1.5.4 .1.3.6.1.2.1.2.2.1.1=38 .1.3.6.1.2.1.2.2.1.7=1 .1.3.6.1.2.1.31.1.1.1.1="ethernet1/1/22" .1.3.6.1.2.1.2.2.1.8=1
11:20:31.329962 IP 10.2.100.253.57355 > 192.168.99.25.snmptrap:  C="Comm2c" V2Trap(135)  .1.3.6.1.2.1.1.3.0=83461796 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.6.3.1.1.5.3 .1.3.6.1.2.1.2.2.1.1=38 .1.3.6.1.2.1.2.2.1.7=1 .1.3.6.1.2.1.31.1.1.1.1="ethernet1/1/22" .1.3.6.1.2.1.2.2.1.8=2
11:20:31.500154 IP 10.2.100.253.41123 > 192.168.99.25.snmptrap:  C="Comm2c" V2Trap(135)  .1.3.6.1.2.1.1.3.0=83461813 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.6.3.1.1.5.4 .1.3.6.1.2.1.2.2.1.1=38 .1.3.6.1.2.1.2.2.1.7=1 .1.3.6.1.2.1.31.1.1.1.1="ethernet1/1/22" .1.3.6.1.2.1.2.2.1.8=1

Then, I saved the config on the switch and saw a trap for that show up as well:

11:26:02.922316 IP (tos 0x0, ttl 63, id 50119, offset 0, flags [DF], proto UDP (17), length 241)
    10.2.100.253.47413 > 192.168.99.25.snmptrap: [udp sum ok]  { SNMPv2c C="Comm2c" { V2Trap(196) R=1031442316  .1.3.6.1.2.1.1.3.0=83494955 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.4.1.674.11000.5000.100.4.1.3.1.35 .1.3.6.1.4.1.674.11000.5000.100.4.1.3.2.1=1 .1.3.6.1.4.1.674.11000.5000.100.4.1.3.2.2=-1 .1.3.6.1.4.1.674.11000.5000.100.4.1.3.2.3="Copy-config from running-config to startup-config succeeded" } }
 

I can't however seem to trigger (what I'd think would be) an SNMP Authorization by logging into the switch.

Looking again at the docs, I see there are commands like

snmp-server view view-name oid-tree [included | excluded]

and

snmp-server group group-name {v1 | v2c | v3 security-level} [access acl-name] [read view-name] [write view-name] [notify view-name] 

Do I need to configure these to fully enable traps, or do traps get sent with default settings (like I'm seeing port up/down and config changes without special configuration)?

I see some examples of these command in the documentation, like:

Configure read-only view
OS10(config)# snmp-server view readonly 1.3.6.1.2.1.31.1.1.1.6 included

Configure notify view
OS10(config)# snmp-server view notifyView 1.3.6.1.2.1.31.1.1.1.6 included
OS10(config)# snmp-server view notifyView 1.3.6.1.2.1.31.0.0.0.0 excluded

Configure SNMPv1 or v2c group
OS10(config)# snmp-server group v2group 2c read readview notify GetsSets

Configure SNMPv3 group
OS10(config)# snmp-server group v3group 3 priv read readview write writeview notify alltraps

... but there isn't much guidance on what would be the settings for various desired applications?

The main traps I'm hoping to collect is:
1) port going up/down

2) the switch being administratively accessed

3) running and saved configuration changes

and

4) MAC address getting added to the bridge table

... which I was thing was addressed by these traps enable commands:

snmp-server enable traps snmp linkDown
snmp-server enable traps snmp linkUp
snmp-server enable traps snmp authentication
snmp-server enable traps config
snmp-server enable traps lldp rem-tables-change

Do I have that right? Is there more guidance on what needs to be set/allowed for what traps to get forwarded, and/or what passes by default?

Thanks for any guidance!

Hello,

That’s great to know that you are able to see the traps being generated now.

You seem to be doing good on selecting traps to enable, you can also refer to OS10 documentation:

Dell SmartFabric OS10 User Guide Release 10.5.4 | Dell United Arab Emirates

https://www.dell.com/support/manuals/en-ae/smartfabric-os10-emp-partner/smartfabric-os-user-guide-10-5-4/snmp-server-enable-traps?guid=guid-1102fb4e-091c-4f37-861f-58997fe1088d&lang=en-us

@DELL-Charles R​

you can also refer to OS10 documentation...

The trouble is documentation that simple says:
"snmp-server enable traps  snmp" means "Enable SNMP traps"
and
"snmp-server enable traps authentication" means "Enable authentication traps"

doesn't really provide any technical information, nor much useful guidance.

While, the documentation's entry that "snmp-server enable traps config" means "Enable startup configuration change traps" at least provides the details that it's only doing traps for startup-configuration changes, there's no discussion of the obvious follow-up question of "how about traps for changes to the running-configuration?"

Likewise, the section for snmp-server enable traps doesn't include explanation of the 'dom' notification type, which is referenced in the snmp-server host command section -- whereas elsewhere in the documentation (under "Digital optical monitoring"), it explains that to enable dom traps you must enter command `dom enable` and then `snmp-server enable traps dom`.

Of course, likewise, traps for functions like bfd, bgp, and vrrp probably depend up those functions being enabled by other configuration mechanisms elsewhere as well.

So far, with my config, I'm only seeing traps show up for:

snmp-server enable traps snmp linkDown
snmp-server enable traps snmp linkUp
snmp-server enable traps config

But, I don't see any guidance in the docs for what it takes to get traps working for 

snmp-server enable traps snmp authentication

... despite the fact that I've done all sort of logging into the switch, running commands, dropping to shell, etc. Thinking maybe something additional might need to be configured, I tried enabling 

aaa accounting commands all console start-stop logging
aaa accounting commands all default start-stop logging

in hopes that would generate some authentication activity, but no luck, switch logins still result in no traps.

I likewise don't see any trap activity for 

snmp-server enable traps lldp rem-tables-change

... although, I can see lldp neighbors and activity via the CLI:

OS10# show lldp traffic
LLDP Traffic Statistics:
Total Frames Out                   : 339040
Total Entries Aged                 : 0
Total Frames In                    : 66826
Total Frames Received In Error     : 0
Total Frames Discarded             : 0
Total TLVS Unrecognized            : 66826
Total TLVs Discarded               : 0

Likewise, I see no trap activity for:

snmp-server enable traps envmon fan
snmp-server enable traps envmon power-supply
snmp-server enable traps envmon temperature

... although, I do see environment information available via the CLI:

OS10# show environment
Unit    State             Temperature
-------------------------------------
1       up                41
Thermal sensors
Unit   Sensor-Id        Sensor-name                               Temperature
------------------------------------------------------------------------------
1       1           CPU Ambient temp sensor                           27
1       2           QSFP Ambient temp sensor                          34
1       3           Right PHY Ambient temp sensor                     29
1       4           Near NPU Ambient temp sensor                      34
1       5           Left PHY Ambient temp sensor                      39
1       6           Left of front vents temp sensor                   36
1       7           Ambient temp sensor                               25
1       8           NPU temp sensor                                   41

Is there somewhere else where environment thresholds need to be configured for envmon traps? Is there default values that will only trigger traps?

And, I'm not even sure what 

snmp-server enable traps entity

is supposed to provide.

Is there a technical reference that explains these things in better somewhere?

How do I know if I have the correct configuration, what events should trigger what traps, and if things are operating correctly?

Thanks for any help!

Hello,

Since you are using NetFlow collector, you may want to check if there are any settings that need to be done on the NetFlow Optimizer side.

SNMP Trap Inputs | NetFlow Logic

https://docs.netflowlogic.com/nfo_administration_guide/inputs/snmp_trap_inputs/

You may want to check with the Netflow tool support.

You can also open a ticket with Dell Support if you would like to make a request for documentation improvements or problem reports. 

Ordinarily we don't do new or initial configuration as that is beyond the scope of this community chat.

We have Deployment Service that can be contracted for that. If you need that you could contact Sales for a quote.