system profiles with syssumm

by Bruce W. Mohler
<[email protected]>

Bruce is a Senior UNIX System Administrator for SAIC, where he is on the Systems Engineering team and provides support for Corporate Solaris, HP-UX, and Linux systems.


I joined the corporate server support group for SAIC as a senior UNIX system administrator in June of 1998. Immediately I began to get lots of questions about the configuration of various UNIX boxes that the group had on hand:

"How much memory?"

"How much disk storage?"

"How are the file systems laid out?"

"Is it running NIS?"

I began to profile a number of these systems by creating a template in an MS Word document that described the general identity, hardware, network, and software configuration of each of these systems. Up to seven or eight systems, this arrangement worked just fine. However, we kept getting involved with more systems (or people saw our profiles and asked us to do ones for them). I'd also think of another piece of information to add to the profile and would have to go through and edit each of the profiles to keep them in sync. All in all, the unautomated task of maintaining these profiles was becoming unwieldy.

I began looking around for existing tools for automatically profiling UNIX systems (especially useful when you have 40 or 50 servers from several vendors, with varying configurations, rather than hundreds of identical workstations), but I found very little existing software. Russ Allbery pointed me to one C program, SysInfo, but it seemed to deal mostly with tunable parameters.

Ultimately, I wrote a series of Perl scripts to try to automate the profiling process, mostly as a "proof of concept." The requirements were:

• Collect information using Perl 5 or later.
• Don't require any additional CPAN modules, if possible.
• Avoid exotic "deep magic" system calls, if possible.
• Allow certain fields to be filled in by the sysadmin.
• Allow these sysadmin fields to "include" multiple external files (in the same directory), including text and graphic files.
• Allow the fields collected to be extended.
• Be aware of security concerns, since some of the information can only be collected if you are running as root.
• Allow transmission of the collected information by email (so the process can be automated with cron).
• The system that receives the messages needs to able to have an "agent" process these incoming messages without human intervention.
• Allow the profiles to be displayed by Web browser.
• The Web server displaying the profiles needs to able to support Perl CGI scripts.
• Allow the final format of the profile to be determined by a format file rather than be hard-coded.

As of this writing, these scripts work for HP-UX 9.X and 10.X and for Linux systems. By the time you read this, I hope to have extended them to Solaris 2.5 and 2.6 systems, and perhaps to Windows NT. One set of scripts automatically generates profile information. The profile information is emailed to another system where a Perl script handles the incoming message. If it's the first profile for this system, then a subdirectory is created and the profile is stored there; otherwise it is merged in with existing profile information for that system. A third Perl script serves as a CGI script to generate a form to query for system profiles, summarize all of the profiles that exist, and create the HTML for the profile of a requested system. The code that generates the profile output is driven by a format file so that a local site can customize the format of its Web-based system profiles without having to hack a single Perl script. This entire process, while perhaps not elegant, works end-to-end today. The rest of this article provides more detail about the software, which I call syssumm, as it exists. It is taken from the existing README file and includes some examples of the files and Web pages generated.

Components of syssumm

The syssumm software has two parts: the software that runs on the remote systems and collects the configuration information, and the software that runs on the Web server, processing incoming messages and merging the information into the appropriate subdirectories.

The "remote" software is composed of a Perl "driver" script called syssumm.pl that makes calls into OS-specific Perl modules, plus a common Perl module of utility subroutines. In general, there is a Perl module for each vendor's OS. For example, for Solaris, HP-UX, and Linux you would have SunOS.pm, HPUX.pm, and LINUX.pm.

Only the module for the local OS running on the system is "used" by the syssumm.pl script. For example, if you are running the script on a Sun box, the HPUX.pm and LINUX.pm modules are never loaded. When it came to extracting information about the remote system, keeping things simple was valued over being extremely clever.

The "Web server" software is composed of a Perl script called incoming.pl that extracts and processes incoming email messages, and a CGI Perl script called sysquery.pl that generates an HTML form, then processes the returned results of that form and displays a system summary. The CGI script also generates a Web page summarizing all existing system profiles.

Software Requirements

On the remote systems the only requirement is Perl 5.x. No additional CPAN Perl modules are required to run the profiling script. On the Web-server system the only requirement is a sendmail-like MDA that allows the processing of messages through a .forward mechanism and Perl 5.x. The Web server must support handling CGI scripts written in Perl.

How Does the Software Work?

First, the system administrator sets up the Web-server software, creating a new account on the system where the Web server exists with a userid of "syssumm". The output of syssumm.pl would be emailed to this account.

Next, in the home directory of the "syssumm" account, the system administrator creates a .forward file which delivers each incoming message to the incoming.pl script that processes the messages and places them in the appropriate directory under the Web server.

The system administrator loads the "remote" software onto a system to be profiled and either manually runs the syssumm.pl script or sets up cron to run periodically. The "-m" command-line option specifies the email address to which to mail the output. (This should be "[email protected]".)

Next, a pointy-haired user brings up the form (generated by sy squery.pl) and enters the name of a system, which is sent back to the same sysquery.pl script to interpret. If the name equates to an existing system for which there is profile information, then that information is formatted as HTML and returned to the requestor's browser. Figure 1 shows what the query form looks like.

Figure 1. Query form

A format file is used to control the appearance of the Web page so that you can choose what fields to display in your system summaries and in what order they appear. Figure 2 shows what a system profile would look like.

Figure 2. System summary

The existing sysquery.pl script is intended to be an example of how to query and retrieve system profiles; obviously, your local Web pages will have a different look.

Appearance of the Generated Data

The information generated by syssumm.pl and processed by incoming.pl is a basic ASCII flat file. Each line is self-sufficient and is composed of:

category:sub-category:value

The following categories have been "hard-coded" into the script that creates the output:

General
Hardware
Network
Software
Comment

Within each category are subcategories. For example, within the General category, you'll find:

NodeName
Organization
Vendor
Model
...

Some of the subcategories are "indexed." For example, the subcategories for disks and tapes look like:

Disks-0
Disks-1
Disks-2
...
Tapes-0
Tapes-1
...

Each line represents an individual device.

Entries in the Comment category are optional. They would be remarks such as "Run as root to get more information."

This is what the actual lines would look like for a hypothetical system:

General:NodeName:ornithomimus
General:Organization:PROTECTED
General:Vendor:Sun
General:Model:Geewhiz
General:HostId:12345678
...

Note that certain lines contain the value "PROTECTED". These fields signify that this information cannot really be figured out by a stupid Perl script and needs the omniscience of a human system administrator.

If a system has been profiled in the past and an updated profile is sent to the Web server system, PROTECTED fields will not overwrite prior contents. This is especially effective when the system administrator has gone in and provided the information that the Perl script couldn't figure out. This feature protects the work that they've done so that they don't need to fill in those fields again.

Note that when the incoming.pl script processes PROTECTED fields (especially for the first time), it changes "PROTECTED" to "To be provided" (to make the report more readable).

Fields Collected by the Profiling Scripts

The following list summarizes the categories and subcategories collected.

 General

NodeName
Organization	(PROTECTED)
Vendor
Model
HostId
SystemHandle	(PROTECTED)
Location	(PROTECTED)
DateSystemInstalled	(PROTECTED)
LargerPicture	(PROTECTED)

 Hardware

Processors
Memory
Disks	(Indexed)
Tapes	(Indexed)
Console	(PROTECTED)
OtherPeripherals	(PROTECTED)

 Network

DomainName
DefaultRouter
NameServer
NetworkInterfaces	(Indexed)
HardwareNetworkAddress

 Software

OsName
OsVersion
NbrOfLicenses
Patches	(Indexed)
RunningLpsched
Printers
RunningSendmail
RunningNfs
RunningAutomount
RunningNis
RunningNameserver
RunningSamba
GraphicalUserInterface
FontServer
LocalFileSystems
RemoteFileSystems
RunningAccounting
ApplicationsInstalled	(PROTECTED)
InstallationProcedure	(PROTECTED)
DataFlowsTo	(PROTECTED)
DataFlowsFrom	(PROTECTED)

 Comments

The "LargerPicture," "DataFlowTo," and "DataFlowFrom" PROTECTED fields are intended to provide the context that the system resides in within your local data center.

Note that while items such as patches are summarized, tunable parameters are not. This may be an area of interest for some system administrators to profile.

Multi-line Values and Extensibility

The system is extensible in the sense that you can add fields to the profile. When incoming.pl merges a new profile into an old one, non-PROTECTED fields "overwrite" existing fields, existing PROTECTED fields are unchanged, and new PROTECTED fields are installed in the profile as "To be provided." As long as you add them to the format file that controls the CGI/HTML output, they will appear in any system profiles requested after that point.

PROTECTED fields are allowed to point to one or more files that may contain text or (Web-compatible) graphics. In fact, if the value begins with a "." (period) or "/" (forward slash) -- indicating a relative or an absolute path, respectively -- and if that file exists, then it will include the contents of the file in place of the value.

Summary

These scripts are the humble beginnings of a system that can profile general, hardware, network, and software configuration information about UNIX systems and display the output on any Web browser.

Since I'm probably not the only person to get these kinds of questions, I am wondering if there is any interest within the SAGE community to collaborate on this software to extend it, refine it, port it to other "species" of UNIX, and make it available to the SAGE community as a whole. I've gotten the permission of my manager to release the Perl source if there's sufficient interest. If you're interested in collaborating on this software, or if you'd just like to keep track of the progress of the software and use it once it's more functional, please contact me at <[email protected]>.