Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Commitment
Running a benchmark to simulate some subset of your compute environment requires an investment on your part and on the part of the computer vendor who wants your business. There is the time commitment of your staff, computing resources, and, possibly, travel and per diem. The computer vendor will have similar commitments.
There may be some legal requirements. Talk to your legal department to establish what, if any, rules may be imposed on your benchmark and selection process by your government, state laws, and so forth; for example, European Union rules or the license to use a program that does not give you the right to redistribute it for a benchmark. The presence of such rules may impact your planning and execution of the benchmark process substantially.
You need to determine the part of your compute environment that you want to benchmark. This is not something that can just be thrown together. Planning is necessary.
Are any independent software vendor (ISV) codes required to run the benchmark? Due to business partnerships, the computer vendors most likely have access to the major database software, like Oracle™ and DB2, and key HPTC applications, such as MSC.Nastran, FLUENT, and STAR-CD. The computer vendor may request your help in obtaining other licensed software necessary for your benchmark.
Do all of your ISV codes run on the computers built by the computer vendors? For Commercial benchmarks load generators written by ISVs are available for a fee. In general, benchmark centers do not have licenses for these. There are times when the database ISV will actively participate in the planning and execution of benchmarks.
Can the program or set of programs be extracted and run independently from the rest of the environment? The best way to test that is to stage and test it on its own in your own data center or lab. It does not really matter which computer or OS you use . Set aside a computer or two and disk space on an isolated network. Load a new OS. Load your benchmark. Does it run successfully? Does it measure what you intend? Write down what you did, every step. You will be doing this again at each computer vendor visit or you may have to write down the instructions carefully for someone else to follow. Measure the performance. Verify the results and write down how the results should be verified.
You have just run your own benchmark. How long did it take you to set up? How long is the run? Can you do that again in a reasonable amount of time?
This staging and testing step produces several key items:
Validates that you are benchmarking the right thing
Allows you to verify that you have all the components necessary to complete the benchmark successfully.
Provides you with a reference to compare the computer vendors
Provides a verification reference for the computer vendor
Provides a performance reference for the computer vendor
Defines your requirements for the computer vendor
Plan to hold several conference calls with the computer vendor's sales team and the benchmark team prior to the benchmark itself. These calls give everyone an opportunity to learn about the benchmark; what is required, what computer configurations are available, and what the goals are.
For commercial benchmarks, less often for HPTC benchmarks, you should plan to travel to the benchmark centers of each computer vendor. You know your benchmark and your applications best and can speedily install and run them. The computer vendor can set up the computer environment before you arrive based on the conference call planning sessions. The computer vendor will host you with experts who can tune the computer and perhaps even your application to perform optimally. Assemble a team of individuals whose skills are complementary, are familiar with the benchmark and are available to travel. Consider which days during the benchmark you need the skills of each individual on the team. There is no point to having your Oracle application expert there while the database administrator is setting up the compute environment. You will need to pay your own travel and per diem expenses. Have individuals on call who are not traveling to consult with those who are to resolve any issues that arise.
If you are not traveling to the benchmark center to perform the benchmark yourself with assistance from the experts, the experts will run the benchmark based on your directions. They will have questions as the benchmark progresses. How do you want to answer them? You may want to have several people answer questions about parts of the benchmark that are in their area of expertise or you could have one person funnel the questions to the others. Supply names, phone numbers, email addresses and area of expertise for each person with the benchmark. You might also consider creating a website with benchmark information, downloads, FAQs, and so forth, as a way to communicate with the benchmark experts.