This chapter is from the book
This chapter is from the book
This chapter is from the book
All-Purpose Tools
Unix provides many tools for the system administrator to monitor systems. However, it is worth noting at this point that what you read here about these utilities might be different, depending on your Linux distribution.
The top Utility
Arguably one of the most popular tools for performance monitoring on Unix systems in general is the top utility. As its name might suggest, this tool displays the top CPU-intensive processes in close to real time. The top display is refreshed every 5 seconds by default, but it can be modified with the -s option for longer or shorter intervals. The top utility also provides an interactive mode for modifying process behavior. The following is the syntax for starting top:
$ top
Figure 3.1 shows an example of top output to a standard terminal screen.
){kind=link}
Figure 3.1 An example of the top process monitor.
Table 3.1 explains the monitor fields in top.
Table 3.1 top Monitor Fields
|
Field |
Description |
|---|---|
|
up |
How long the system has been running since the last time it was rebooted. The load averages are displayed as well. |
|
processes |
Total number of processes running since the last time the top monitor checked. This measurement includes all processes on the system. |
|
CPU States |
An average percent of CPU time. This field examines all aspects of CPU time, including user, idle, and niced tasks. Because niced tasks are included, the total can go over 100%, so do not be alarmed if this is the case (unless it is an outrageous value, such as 160%). |
|
Mem |
All the memory statistics, such as total memory, available memory for nonkernel processes, memory in use, and memory that is shared and buffered. |
|
Swap |
Swap statistics, including total allocated swap space, available swap space, and used swap space. |
Table 3.2 explains the process fields in top.
Table 3.2 top Process Fields
|
Field |
Description |
|---|---|
|
PID |
Process ID of each task. |
|
USER |
Username of each task. |
|
PRI |
Priority of each task. |
|
NI |
Nice value of each task. |
|
SIZE |
Total size of a task, including code and data, plus the stack space in kilobytes. |
|
RSS |
Amount of physical memory used by the task. |
|
SHARE |
Amount of shared memory used by a task. |
|
STATE |
Current CPU state of a task. The states can be S for sleeping, D for uninterrupted, R for running, T for stopped/traced, and Z for zombied. |
|
TIME |
The CPU time that a task has used since it started. |
|
%CPU |
The CPU time that a task has used since the last update. |
|
%MEM |
A task's share of physical memory. |
|
COMMAND |
The task's command name. |
NOTE
A task's command name is truncated if the tasks have only the program name in parentheses.
In addition to just watching the top display, you can manipulate the top display in interactive mode and modify running processes. The interactive mode is invoked by pressing H while the display is running.
What about understanding what top tells you and knowing how to filter? A good example is simple filtering. In this example, a process is intermittently jumping to the top of the monitor. It might help to stop the monitor and start it with a few filtering options to get rid of information that you do not want:
$ top -i
Still, you cannot catch the culprit in action. Next you can disable some of the information that you want by toggling on or off displays such as the memory summary.
The great thing about top is that it provides a lot of different information quickly, and it updates periodically. For example, a process might be at the top, but why? Is it because it requires more processor time? Or is it eating memory? With the additional fields shown, more information is displayed in one location.
vmstat
The name vmstat comes from "report virtual memory statistics." The vmstat utility does a bit more than this, though. In addition to reporting virtual memory, vmstat reports certain kernel statistics about processes, disk, trap, and CPU activity.
The syntax for vmstat is as follows:
$ vmstat interval [count]
A sample syntax with an interval of 5 seconds and five counts would look like this:
$ vmstat 5 5 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 380 3760 71616 944176 0 0 24 13 40 45 0 0 32 0 0 0 380 3760 71616 944176 0 0 0 4 105 5 0 1 99 0 0 0 380 3760 71616 944176 0 0 0 1 105 4 0 0 100 0 0 0 380 3760 71616 944176 0 0 0 0 103 4 0 0 100 0 0 0 380 3760 71616 944176 0 0 0 0 103 5 0 0 100
The very first line of output by vmstat is the average values for statistics since boot time, so do not be alarmed by high values.
The vmstat output is actually broken up into five sections: procs, memory, swap, io, and cpu.
Each section is outlined in the following table.
Table 3.3 The procs Section
|
Field |
Description |
|---|---|
|
r |
Number of processes that are in a wait state and basically not doing anything but waiting to run |
|
b |
Number of processes that were in sleep mode and were interrupted since the last update |
|
w |
Number of processes that have been swapped out by mm and vm subsystems and have yet to run |
Table 3.4 The Memory Section
|
Field |
Description |
|---|---|
|
swpd |
The total amount of physical virtual memory in use |
|
free |
The amount of physical memory that is free or available |
|
buff |
Memory that was being buffered when the measurement was taken |
|
cache |
Cache that is in use |
Table 3.5 The Swap Section
|
Field |
Description |
|---|---|
|
si |
Amount of memory transferred from swap space back into memory |
|
so |
Amount of memory swapped to disk |
Table 3.6 The IO Section
|
Field |
Description |
|---|---|
|
bi |
Disk blocks sent to disk devices in blocks per second |
Table 3.7 The System Section
|
Field |
Description |
|---|---|
|
in |
Interrupts per second, including the CPU clocks |
|
cs |
Context switches per second within the kernel |
Table 3.8 The CPU Section
|
Field |
Description |
|---|---|
|
us |
Percentage of CPU cycles spent on user processes |
|
sy |
Percentage of CPU cycles spent on system processes |
|
id |
Percentage of unused CPU cycles or idle time when the CPU is basically doing nothing |
Interpreting the output of these measurements tells you that there is some outbound disk activity (as shown by the bo field). Also note the increase in the in field of the system section once vmstat was started. This implies that vmstat increases the number of interrupts. Here you return to the idea of properly reading output. It is a good idea to look at the tools covered in this chapter when the system is idling so that you know what effect a performance-monitoring tool can have on the system.
xload and xosview
Along with many command-line-driven tools for Unix systems are X11-based tools. This section covers two of them, xload and xosview. Both distinctly different graphical monitoring tools, they provide a very quick glance method of watching your systems run.
xload
The first of the two is xload. In a nutshell, xload displays the system load average. It also has an alarm bar that shows when the load average is high. Figure 3.2 shows what xload typically looks like.
){kind=link}
Figure 3.2 An example of xload.
To start xload, simply open an xterminal on the system and type the following:
$ xload &
The system knows to which display to go, so it automatically pops up. If you are running xload from a remote system to your X Window System, you can do one of two things:
-
Set a permanent DISPLAY variable in your .profile by entering this:
-
Simply direct it to your system with this syntax:
DISPLAY=<your_ip_address>:0 export DISPLAY
$ xload -display <your_ip_address> &
The xload utility displays a histogram (chart) of the system load average. It updates this information periodically. The default is every 10 seconds, but this can be changed with the -update option.
xosview
The xosview utility can display a much more detailed collection of histograms about the system's performance in close to real time.
Of course, you easily can filter what you want to see in xosview using command-line options by specifying the subsystem with + or - (true and false, respectively). For example, to not see the CPU histogram, the syntax would be this:
$ xosview -cpu &
The xosview utility can be started from remote systems to a local X server in the same manner as xload. The colors, font, and other properties can be controlled by Xresource settings.
Figure 3.3 xosview, with all subsystems being monitored.
){kind=link}
One interesting aspect of xosview is that it combines certain histograms into one horizontal bar. This requires some explanation.
-
LOAD—This field has two colors. The first color is the load average of processes, and the second color (the background color) is idle processes relative to the load average. When the load average goes above 1.0, the bar will change colors.
-
CPU—The CPU field has four colors related to process type usage: usr, nice, sys, and free.
-
MEM—In this field, the full amount of real memory in use is shown. There are four colors in this field: user-allocated memory, shared memory, buffers, and free memory.
-
SWAP—Swap has two colors; the first indicates the swap in use, and the second indicates what is free.
-
PAGE—This field has three colors: in, for paging in; out, for paging out; and idle.
-
DISK—This field has three colors: in, for transfers to disk; out for transfers from disk; and idle.
-
INTS—This field is a set of indicators that correspond to IRQ usage. They are numbered starting at 0 from the left.
uptime
The uptime command displays the current time, the length of time that the system has been up, the number of users, and the load average of the system over the last 1, 5, and 15 minutes. It looks something like this:
6:51PM up 2 days, 22:50, 6 users, load averages: 0.18, 0.30, 0.34