Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Software
What software is required to run this benchmark successfully?
Which operating system should be used? UNIX™? What tools do you want to use to monitor the compute environment? If you want to use more than the common UNIX tools like sar and vmstat, plan on providing them.
For an ISV code, if it is not a commonly available product like DB2 or ANSYS, identify how the computer vendor can get it or get a copy and license for them. Also indicate the version of the ISV product that should be used. Sometimes there are compatibility issues, particularly between major releases, so knowing the version saves time.
For database benchmarks, table layouts, queries, and middle ware must be identified and documented. To ensure that the tables are loaded correctly, can you provide a query that verifies this along with the correct results of that query?
Does your benchmark include programs that your staff has written? In addition to source code for all the programs, provide some documentation about the program, such as flowcharts and overall descriptions of the program and algorithms used. For every input data set include the corresponding output file(s). The computer vendor will use them to verify that the answers are correct on his computer. If the run time is on the order of hours, is there a smaller data set that shows the same characteristics as the longer one? The computer vendor's optimization activity will be more successful if the runtimes are not excessive.
Once the computer vendor has your program, what is the vendor allowed to do with it? Can the vendor only make changes necessary to run it successfully (porting)? Can the benchmark expert rewrite parts of the program to improve performance? Can optimized libraries be used? Can directives (like OpenMP) be used? What about assembly code? Is that acceptable as an optimization technique? You probably do not want optimizations that are beyond the expertise of your users.