- Layers of File System and I/O
- Observing Physical I/O
- File System Latency
- Causes of Read/Write File System Latency
- Observing File System "Top End" Activity
- File System Caches
- NFS Statistics
This chapter is from the book
This chapter is from the book
This chapter is from the book
5.5 Observing File System "Top End" Activity
Applications typically access their data from a file system through the POSIX I/O library and system calls. These accesses are passed into the kernel and into the underlying file system through the VOP layer (see Section 5.1).
Using DTrace function boundary probes, we can trace the VOP layer and monitor file system activity. Probes fired at the entry and exit of each VOP method can record event counts, latency, and physical I/O counts. We can obtain information about the methods by casting the arguments of the VOP methods to the appropriate structures; for example, we can harvest the file name, file system name, I/O size, and the like from these entry points.
The DTrace vopstat command instruments and reports on the VOP layer activity. By default, it summarizes each VOP in the system and reports a physical I/O count, a VOP method count, and the total latency incurred for each VOP during the sample period. This utility provides a useful first-pass method of understanding where and to what degree latency is occurring in the file system layer.
The following example shows vopstat output for a system running ZFS. In this example, the majority of the latency is being incurred in the VOP_FSYNC method (see Table 14.3 in Solaris ™ Internals).
# ./vopstat VOP Physical IO Count fop_fsync 236 VOP Count Count fop_create 1 fop_fid 1 fop_lookup 2 fop_access 3 fop_read 3 fop_poll 11 fop_fsync 31 fop_putpage 32 fop_ioctl 115 fop_write 517 fop_rwlock 520 fop_rwunlock 520 fop_inactive 529 fop_getattr 1057 VOP Wall Time mSeconds fop_fid 0 fop_access 0 fop_read 0 fop_poll 0 fop_lookup 0 fop_create 0 fop_ioctl 0 fop_putpage 1 fop_rwunlock 1 fop_rwlock 1 fop_inactive 1 fop_getattr 2 fop_write 22 fop_fsync 504