Linux Magazine June 2002

Copyright Linux Magazine ©2002

GURU GUIDANCE
Monitoring Linux Hosts with SNMP
by Æleen Frisch

Last month's column introduced the Simple Network Management Protocol (SNMP) and described how SNMP can be used to monitor devices and hosts connected to your network. This month, we'll configure a Linux machine as an SNMP agent and learn how to keep SNMP secure.

Configuring Net-SNMP

Net-SNMP (http://www.net-snmp.org/), formerly known as UCD-SNMP, is the preferred SNMP package for Linux. Net-SNMP includes the snmpd daemon, an agent for Linux hosts, and a suite of client utilities. Net-SNMP does not include a manager (remember that an SNMP manager is the central monitor and the SNMP agent is the client); you'll need to separately acquire SNMP management software if you want to talk to the Net-SNMP agent software.

The snmpd daemon is started via the /etc/init.d/snmp boot script and uses /usr/share/snmp/snmpd.conf as its default configuration file. (Be aware that the RPMs provided with recent SuSE distributions use the /etc/ucdsnmpd.conf configuration file instead, although you can change this by editing the boot script.) Figure One shows a sample Net-SNMP snmpd.conf file.

Figure One: Sample snmpd.conf file

 1  rocommunity    somethingsecure
 2  rwcommunity    somethingelse
 3  trapcommunity  anothername
 4  trapsink       dalton
 5  trap2sink      dalton
 6
 7  syslocation    "Building 2 Main Machine Room"
 8  syscontact     "chavez@ahania.com"
 9
10  # Net-SNMP-specific items
11  #keyw [args] limit(s)
12  load   5.0 6.0 7.0  # 1,5,15 load average max
13  disk   / 3%        # root filesystem < 3% free
14  proc   portmap 1 1 # Only 1 portmap process 
15  proc   cron  1 1   # Only 1 cron process 
16  procsendmail    # 1 or more sendmails 

Lines 1 and 2 specify the community names used to access the agent in read-only and read-write modes. Line 3 defines the community name to use when sending traps. It's best to use different names for each community, but it's not required.

Lines 4 and 5 specify the trap destination for SNMP Version 1 and Version 2, respectively. In this case, they're both sent to the host dalton. Lines 7 and 8 specify the values of two mib-2 variables that describe the location of the device and its primary contact. Both values are located under mib-2.system. Note that it is not necessary to specify the complete "path" to these two data nodes.

The final section (lines 10-16) defines monitoring items specific to Net-SNMP. Line 12 checks whether the 1-, 5-, or 15-minute load average is above 5.0, 6.0, or 7.0, respectively. Line 13 checks whether the free space in the root filesystem (/) has dropped below three percent. Finally, the three proc directives (found on lines 14-16) ensure that exactly one portmap and cron daemon are running (the two "1"s mean at least one and no more than one), and that at least one sendmail daemon is running.

If a monitored value falls outside the allowed range, the SNMP daemon will set a value in one of the data nodes defined in the MIB underneath enterprises.ucdavis. The nodes correspond to the specific item being monitored: laTable.laEntry.laErrorFlag for load average, dskTable. dskEntry.dskErrorFlag for disk, and prTable.prEntry. prErrorFlag for processes. The exact value stored depends on which error flag is being set. Note that traps are not generated.

You can also use the snmpconf -g command to configure a snmpd.conf file. Add the -i option to have the command automatically install the new file into the proper directory (rather than placing it in the current directory).


The Net-SNMP Utilities

Net-SNMP includes several useful utilities for querying SNMP devices. snmptranslate provides information about the MIB structure and its entities (but not any actual data). Table One shows some useful options.

Table One: Useful options for snmptranslate

OPTIONDESCRIPTION
-Tp OID

Display subtree underneath this data node

-Td OID

Display the complete definition of the data node

-IR -On name

Display the full numeric OID for this name

-IR name

Translate a data node name to its numeric equivalent

-On number

Translate a data node number to its string equivalent

If the values provided for the OID, name, or number arguments are "absolute" (i.e., starting at the top of the MIB tree, before the iso or the first 1 ), they must start with a period (".").

To show how snmptranslate works, let's define an alias (in tcsh) that takes the name of a leaf node (in the mib-2 tree) as its argument and prints out the definition of that item, including its full OID string and numeric equivalent. The shell alias and sample usage are shown in Figure Two. The alias uses two calls to snmptranslate. The one in backquotes (snmptranslate -IR -On \!:1) finds the full OID for the specified name (substituted in with !:1). Its output becomes the argument of the second command which displays the description of this data item.

Figure Two: Using the snmptranslate command

% alias snmpwhat 'snmptranslate -Td `snmptranslate -IR -On \!:1`'
% snmpwhat sysLocation
.1.3.6.1.2.1.1.6
sysLocation OBJECT-TYPE
  -- FROM       SNMPv2-MIB, RFC1213-MIB
  -- TEXTUAL CONVENTION DisplayString
  SYNTAX        OCTET STRING (0..255)
  DISPLAY-HINT  "255a"
  MAX-ACCESS    read-write
  STATUS        current
  DESCRIPTION   "The physical location of this node (e.g., `telephone
            closet, 3rd floor').  If the location is unknown, the value
            is the zero-length string."
::= { iso(1) org(3) dod(6) internet(1) mgmt(2) mib-2(1) system(1) 6 }

Figure Three shows how the snmpget command retrieves data from an SNMP agent. The first command displays the value of the sysLocation data node from the agent on beulah, specifying the community string as somethingsecure. The specified data node (sysLocation) is followed by a period and an instance number. When a data node holds only a single value, the number 0 is used.

Figure Three: Using the snmpget command

# snmpget beulah somethingsecure sysLocation.0
system.sysLocation.0 = "Receptionist Printer"

# snmpget beulah somethingsecure laLoad.2
enterprises.ucdavis.laTable.laEntry.laLoad.2 = 1.22

By convention, if a leaf node has a parent whose name contains "Table", then that leaf node holds multiple values. The presence of the laTable data node shows that its leaf node, laLoad, holds multiple values (in this case, the 1-, 5-, and 15-minute load averages). Multiple values are stored as tables, with each value in a separate row. To access a specific row, you use the row number as the instance number. Row numbers start from 1.

The second command in Figure Three retrieves the 5-minute load average, which is stored in the second row of the enterprises. ucdavis.laTable.

The snmpwalk command displays the entire subtree below a specified data node. Figure Four shows how snmpwalk displays all the data under iso.org.dod.internet. mgmt.mib-2.host.hrSystem. Each line of output is of the form OID.instance = [datatype:] value, where datatype is optional. Use the snmptranslate command to get a full description of each data node in the MIB.

Figure Four: Using the snmpwalk command

# snmpwalk beulah somethingsecure host.hrSystem
host.hrSystem.hrSystemUptime.0 = Timeticks: (31861126) 3 days, 16:30:11.26
host.hrSystem.hrSystemDate.0 = 2002-2-8,11:5:4.0,-5:0
host.hrSystem.hrSystemInitialLoadDevice.0 = 1536
host.hrSystem.hrSystemInitialLoadParameters.0 = "auto BOOT_IMAGE=linux ro
  root=2107 BOOT_FILE=/boot/vmlinuz enableapic vga=0x0314."
host.hrSystem.hrSystemNumUsers.0 = Gauge32: 1
host.hrSystem.hrSystemProcesses.0 = Gauge32: 205
host.hrSystem.hrSystemMaxProcesses.0 = 0

Finally, we can use the snmpset command to modify writable data values. Figure Five shows how to use snmpset to set the device's primary contact. The s parameter indicates a string datatype; other useful datatypes are i for integer, d for decimal, and a for IP address (consult the snmpset man page for the entire list).

Figure Five: Using the snmpset command

# snmpset beulah somethingelse sysContact.0 s "r_chavez@ahania.com"
system.sysContact.0 = r_chavez@ahania.com

Generating and Handling Traps

SNMP-managed devices are usually shipped with pre-defined traps that can be enabled or disabled during configuration. Some agents are extensible and allow you to define additional traps. The Net-SNMP package includes the snmptrap command for manually generating traps. Figure Six shows an example of using snmptrap.

Figure Six: Using snmptrap to generate a trap

# snmptrap -v2c dalton anothername '' .1.3.6.1.6.3.1.1.5.3 \
  ifIndex i 2   ifAdminStatus i 1   ifOperStatus i 1

The -v2c option indicates that an SNMP Version 2c trap is to be sent. The next two arguments (dalton and anothername) are the destination for the trap message (the SNMP manager) and the trap community name. The next argument is the device uptime, which is required for all traps. In this example, we specify a null string, which defaults to the current uptime. The final argument in the first line is the OID for the trap itself. Traps are defined in a MIB just like regular data nodes and therefore have OIDs. This OID corresponds to the trap (or node) linkDown, and means a network interface has changed state (from up to down or from down to up).

The remaining arguments (starting with ifIndex) are determined by the specific trap being sent. This particular trap (linkDown) requires three additional arguments, where each is, in turn, composed of three values. The three arguments are the interface number, the administrative status and the operational status. The three values that specify each of these arguments are a keyword (ifIndex, ifAdminStatus, and ifOperStatus), the type of data (i for integer), and the actual (integer) value (2, 1, and 1).

In this case, the trap specifies interface number 2. The two status values of 1 indicate that the interface is up, so this trap means that the interface is back online after being down.

Net-SNMP also includes the snmptrapd daemon that runs on the manager host to handle incoming traps (although you might not need this daemon if your management software handles traps). Start the daemon manually by running the snmptrapd -s command, which by default sends trap messages to the syslogd daemon, using the Local0 facility (for more on syslogd, see the September 2000 Guru Guidance column, online at http://www.linuxmagazine.com/2000-09/guru_01.html). Figure Seven shows the syslog message generated by the trap command in Figure Six.

Figure Seven: The syslog log message for a trap

Feb 25 11:44:21 beulah snmptrapd[8675]: beulah.local [192.0.9.84]: 
Trap system.sysUpTime.0 = Timeticks: (144235905) 16 days, 16:39:19.05, 
.iso.org.dod.internet.snmpV2.snmpModules.snmpMIB.
snmpMIBObjects.snmpTrap.snmpTrapOID.0 = OID: 1.1.5.3, 
interfaces.ifTable.ifEntry.ifIndex = 2, 
interfaces.ifTable.ifEntry.ifAdminStatus = up(1), 
interfaces.ifTable.ifEntry.ifOperStatus = up(1)

To start snmptrapd at boot time, you'll need to add the startup command to the /etc/ init.d/snmp boot script. Its configuration file is /usr/share/snmp/snmptrapd.conf.

If you want the daemon to perform certain tasks upon receiving various traps, add entries to the configuration file that look like this:

traphandle OID|default program 
[arguments] 

The second field is either the OID of the trap that should be handled or the word default, which handles traps when there is no matching traphandle definition. The rest of the line contains the name of a program and its arguments. This program (or action) is run whenever the specified trap is received.


SNMP and Security

As with any network service, SNMP presents some security concerns and demands tradeoffs be made. At the time this article was written, a major SNMP vulnerability was uncovered and widely publicized (see http://www.cert.org/advisories/CA-2002-03.html). Interestingly, Net-SNMP was one of the few implementations that did not include the problem; all the commercial network management packages were affected.

Perhaps the biggest problem with the current SNMP versions is that community names are sent unencrypted over the network (and can thus be "sniffed" by listening to the network). Additionally, poor coding in SNMP agents means that some devices are vulnerable to buffer overflow attacks until their vendors provide patches. Thus, a decision to use SNMP involves balancing security needs with the advantages and convenience that it provides.

SNMP Version 3 will greatly improve the security of SNMP, but many devices don't support version 3 yet, because it's still in development and is only a draft standard. Version 3's best new feature is that it sends encrypted community strings only. It also provides user-based access control to SNMP data. Due to its increased security, migrating to SNMP version 3 as soon as you can will be very worthwhile.

If you aren't able to run SNMP Version 3, you'll definitely want to take steps to increase the security of your SNMP setup. Even if you are running SNMP Version 3, implementing the following best practices is advisable.

  • Disable SNMP on devices that are not being managed. On Linux systems, remove any links to /etc/init.d/ snmp in the /etc/rc*.d directories.
  • Choose community names that are not likely to be easily guessed. Change default community names before devices are added to the network.
  • Block external access to the SNMP ports: TCP and UDP ports 161 and 162, as well as any additional vendor-specific ports (e.g., TCP and UDP ports 1993 for Cisco).
  • If you want to perform SNMP operations over the Internet (e.g., from a Web browser), use VPN or an SSL-encrypted connection to protect your data.
  • Configure agents to only accept requests from a small group of hosts (not all agents have this feature). In the configuration file snmpd.conf, community string definitions can have additional parameters to restrict access. For example:
rwcommunity localonly 192.0.10.0/24
rocommunity onlylocal callisto .1.3.6.1.2.1.25

The first entry defines the read-write community name as "localonly", and will only accept requests from hosts within the 192.0.10.0 subnet. The second entry specifies a read-only community name ("onlylocal") usable only by the host callisto. It also limits access from callisto to the nodes underneath the mib-2.hosts subtree.

So Much to Monitor, So Little Time

SNMP can dramatically increase the amount of information you can gather about the devices on your network and can help you more quickly identify problems when they arise. If you have good management software, SNMP can also help identify recurring problems (such as constantly full file systems or high load averages) that should be addressed.

Once you've set up SNMP, your next problem will be finding the time to analyze and interpret all of the information it gives you.


Æleen Frisch is the author of Essential System Administration. She can be reached at aefrisch@lorentzian.com.

Linux Magazine June 2002

Copyright Linux Magazine ©2002