5.2 Observing Physical I/O
The traditional method of observing file system activity is to induce information from the bottom end of the file system, for example, physical I/O. This can be done easily with iostat or DTrace, as shown in the following iostat example and further in Chapter 4.
$ iostat -xnczpm 3 cpu us sy wt id 7 2 8 83 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.6 3.8 8.0 30.3 0.1 0.2 20.4 37.7 0 3 c0t0d0 0.6 3.8 8.0 30.3 0.1 0.2 20.4 37.7 0 3 c0t0d0s0 (/) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 48.7 0 0 c0t0d0s1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0s2 0.0 0.0 0.0 0.0 0.0 0.0 405.2 1328.5 0 0 c0t1d0 0.0 0.0 0.0 0.0 0.0 0.0 405.9 1330.8 0 0 c0t1d0s1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0s2 14.7 4.8 330.8 6.8 0.0 0.3 0.0 13.9 0 8 c4t16d1 14.7 4.8 330.8 6.8 0.0 0.3 0.0 13.9 0 8 c4t16d1s7 (/export/home) 1.4 0.4 70.4 4.3 0.0 0.0 0.0 21.8 0 2 c4t16d2 1.4 0.4 70.4 4.3 0.0 0.0 0.0 21.8 0 2 c4t16d2s7 (/export/home2) 12.8 12.4 73.5 7.4 0.0 0.1 0.0 2.5 0 3 c4t17d0 10.8 10.8 0.4 0.4 0.0 0.0 0.0 0.0 0 0 c4t17d0s2 2.0 1.6 73.1 7.0 0.0 0.1 0.0 17.8 0 3 c4t17d0s7 (/www) 0.0 2.9 0.0 370.4 0.0 0.1 0.0 19.1 0 6 rmt/1
Using iostat, we can observe I/O counts, bandwidth, and latency at the device level, and optionally per-mount, by using the -m option (note that this only works for file systems like UFS that mount only one device). In the above example, we can see that /export/home is mounted on c4t16d1s7. It is generating 14.7 reads per second and 4.8 writes per second, with a response time of 13.9 milliseconds. But that's all we know—far too often we deduce too much by simply looking at the physical I/O characteristics. For example, in this case we could easily assume that the upper-level application is experiencing good response times, when in fact substantial latency is being added in the file system layer, which is masked by these statistics. We talk more about common scenarios in which latency is added in the file system layer in Section 5.4.
By using the DTrace I/O provider, we can easily connect physical I/O events with some file-system-level information; for example, file names. The script from Section 5.4.3 shows a simple example of how DTrace can display per-operation information with combined file-system-level and physical I/O information.
# ./iotrace.d DEVICE FILE RW SIZE cmdk0 /export/home/rmc/.sh_history W 4096 cmdk0 /opt/Acrobat4/bin/acroread R 8192 cmdk0 /opt/Acrobat4/bin/acroread R 1024 cmdk0 /var/tmp/wscon-:0.0-gLaW9a W 3072 cmdk0 /opt/Acrobat4/Reader/AcroVersion R 1024 cmdk0 /opt/Acrobat4/Reader/intelsolaris/bin/acroread R 8192 cmdk0 /opt/Acrobat4/Reader/intelsolaris/bin/acroread R 8192 cmdk0 /opt/Acrobat4/Reader/intelsolaris/bin/acroread R 4096 cmdk0 /opt/Acrobat4/Reader/intelsolaris/bin/acroread R 8192 cmdk0 /opt/Acrobat4/Reader/intelsolaris/bin/acroread R 8192