Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Practicality
While you are deciding the composition of the benchmark, use common sense. How practical it is to assemble and run? Are you willing to do it yourself?
It is not practical to expect that the computer vendor has 20 PCs to act as clients. It is not practical to expect the computer vendor to download three gigabytes of files from a website. It is practical to work with your computer vendors to use the resources that are available effectively.
Do not forget about confidentiality. Do you want to use real, sensitive, and proprietary data as part of the benchmark? If you will be using sensitive, proprietary data, have the computer vendor sign a non-disclosure agreement, unless there is one already in place. As part of the benchmark rules include a statement that all programs, applications, databases, data, and media should be removed, destroyed, or returned at the conclusion of the benchmark.
If you plan to travel to the benchmark center and run the test yourself, you will probably want to limit the benchmark test. Do you really want to ship off a benchmark team for a month to work on this when they need to do their usual jobs, too? Determine how long the tests will take. This is where that pre-staging in your own computer lab will tell you a lot.
How long does the benchmark take? Do runs that last just a few minutes or days make sense? When benchmarks complete too quickly (seconds) or take too long (a day), the opportunities for the computer vendor to tune the computer configuration and the application are limited due to the turnaround time. If you really have to have long runs, include a shorter run that exhibits the same characteristics as the longer run, so that the vendors can tune and showcase their computers quickly. Then the long run can be made with the optimizations of the shorter one.
If you want to benchmark using a three terabyte database, where is the data coming from? Tapes are not practical. Can you write a program to generate the data? When making tapes of data, compress the files first. Writing and reading the tapes will be much faster.