- Goals
- Commitment
- Practicality
- Define the Rules
- Software
- Hardware
- Types of Runs
- Reporting
- Questions
- Summary
- Checklist
Define the Rules
Everyone will know what the benchmark is when you define in writing the benchmark and its goals and rules. If you do not state them clearly and publicly, you will be asked lots of questions or maybe none, with each computer vendor creating their own rules. In the absence of information people will invent their own to fill the gaps.
Clearly state what the main parts of the benchmark are, why these are included and their respective objectives. What are the goals for each part of the benchmark? Is it functionality? Performance? Portability?
How is each goal measured? Here are some ideas. For functionality, is the failover within your allowable time? For performance, is 2x faster than your current compute environment acceptable? State the current performance and what you expect from your new system. For portability, what is your tolerance level?
What if your requirements are not met? Do not be overly restrictive in your specifications, such as avoiding exact specifications of number of CPUs to be used in fat SMP nodes for throughput tests. Keep your goal in mind, not a specific system from a specific vendor you happen to know.
Everyone involved including your team and the computer vendor's team must keep in mind the definition and goals throughout the benchmark. It is easy to get sidetracked by "It would be nice if we could..." Finish the benchmark and, if there is time remaining, explore further.
If there are any confidentiality issues, state them in the rules. Computer vendors may have a collection of benchmark reports available for internal use. If you do not want a report of your benchmark as part of that collection, state that.