- 12.1. Background
- 12.2. Benchmarking Types
- 12.3. Methodology
- 12.4. Benchmark Questions
- 12.5. Exercises
This chapter is from the book
This chapter is from the book
This chapter is from the book
12.4. Benchmark Questions
If a vendor gives you a benchmark result, there are a number of questions you can ask to better understand and apply it to your environment. The goal is to determine what is really being measured and how realistic or repeatable the result is.
The hardest question may be: Can I reproduce the result myself?
The benchmark result may be from an extreme hardware configuration (e.g., DRAM disks), special-case tuning (e.g., striped disks), a stroke of luck (not repeatable), or a measurement error. Any of these can be determined if you are able to run it in your own data center and perform your own analysis: active benchmarking. This does, however, consume a lot of your time.
Here are some other questions that may be asked:
In general:
- What was the configuration of the system under test?
- Was a single system tested, or is this the result of a cluster of systems?
- What is the cost of the system under test?
- What was the configuration of the benchmark clients?
- What was the duration of the test?
- Is the result an average or a peak? What is the average?
- What are other distribution details (standard deviation, percentiles, or full distribution details)?
- What was the limiting factor of the benchmark?
- What was the operation success/fail ratio?
- What were the operation attributes?
- Were the operation attributes chosen to simulate a workload? How were they selected?
- Does the benchmark simulate variance, or an average workload?
- Was the benchmark result confirmed using other analysis tools? (Provide screen shots.)
- Can an error margin be expressed with the benchmark result?
- Is the benchmark result reproducible?
For CPU/memory-related benchmarks:
- What processors were used?
- Were any CPUs disabled?
- How much main memory was installed? Of what type?
- Were any custom BIOS settings used?
For storage-related benchmarks:
- What is the storage device configuration (how many were used, their type, RAID configuration)?
- What is the file system configuration (how many were used, and their tuning)?
- What is the working set size?
- To what degree did the working set cache? Where did it cache?
- How many files were accessed?
For network-related benchmarks:
- What was the network configuration (how many interfaces were used, their type and configuration) ?
- What TCP settings were tuned?
When studying industry benchmarks, many of these questions may be answered from the disclosure details.