DTrace and Application Analysis
This chapter is from the book
This chapter is from the book
This chapter is from the book
DTrace has the ability to follow the operation of applications from within the application source code, through system libraries, through system calls, and into the kernel. This visibility allows the root cause of issues (including performance issues) to be found and quantified, even if it is internal to a kernel device driver or something else outside the boundaries of the application code. Using DTrace, questions such as the following can be answered.
- What transactions are occurring? With what latency?
- What disk I/O is the application performing? What network I/O?
- Why is the application on-CPU?
As an example, the following one-liner frequency counts application stack traces when the Apache Web server (httpd) performs the read() system call:
#dtrace -n 'syscall::read:entry /execname == "httpd"/ { @[ustack()] = count(); }'
dtrace: description 'syscall::read:entry ' matched 1 probe
[...]
libc.so.1`__read+0x7
libapr-1.so.0.3.9`apr_socket_recv+0xb0
libaprutil-1.so.0.3.9`socket_bucket_read+0x5b
httpd`ap_core_input_filter+0x294
mod_ssl.so`bio_filter_in_read+0xbc
libcrypto.so.0.9.8`BIO_read+0xaf
libssl.so.0.9.8`ssl3_get_record+0xb5
libssl.so.0.9.8`ssl3_read_n+0x144
libssl.so.0.9.8`ssl3_read_bytes+0x161
libssl.so.0.9.8`ssl3_read_internal+0x66
libssl.so.0.9.8`ssl3_read+0x16
libssl.so.0.9.8`SSL_read+0x42
mod_ssl.so`ssl_io_input_read+0xf0
mod_ssl.so`ssl_io_filter_input+0xd0
httpd`ap_rgetline_core+0x66
httpd`ap_read_request+0x1d1
httpd`ap_process_http_connection+0xe4
httpd`ap_run_process_connection+0x28
httpd`child_main+0x3d8
httpd`make_child+0x86
httpd`ap_mpm_run+0x410
httpd`main+0x812
httpd`_start+0x7d
31
|
The output has been truncated to show only the last stack trace. This stack trace was responsible for calling read() 31 times and shows the application code path through libssl (the Secure Sockets Layer library, because this was an HTTPS read). Each of the functions shown by the stack trace can be traced separately using DTrace, including function arguments, return value, and time.
The previous chapter focused on the programming languages of application software, particularly for developers who have access to the source code. This chapter focuses on application analysis for end users, regardless of language or layer in the software stack.
Capabilities
DTrace is capable of tracing every layer of the software stack, including examining the interactions of the various layers (see Figure 9-1).
)
Figure 9-1 Software stack