Benchmarking interactive systems: Producing the software

The author has recently developed a new methodology of benchmarking, which is being applied to a procurement in which (a) a single integrated interactive application is to span a distributed configuration of computing hardware, (b) the configuration is unknown when the benchmark is being developed, and (c) the application software will be written after the benchmark has been run. The buyer prepares a simulation model of the intended application in the form of programs that will run on the hardware being benchmarked. Each competing vendor is expected to tune the performance of this model to the hardware configuration that he has proposed, so he will require several versions of the model. This presents the buyer with a formidable software-production problem, which is further complicated by a requirement for extreme flexibility and reliability. The paper addresses the software-production problem and describes its solution. The solution was to develop an automated code-production system based on two principal design features. First, the model and its translator are both written in the same language; secondly, the common language is selected on the basis of readability and extensibility. The paper examines why this approach to the code-production problem was successful. Though the code-production system was developed to support a particular benchmarking approach, it should also be useful in other modeling situations. Indeed it might be of interest in any field where readability, reliability, ease of maintenance, and economy of programming effort are considered important.