10.4 Performance
A question managers need to ask themselves is whether they can buy a product that satisfies their requirements. This is especially a consideration for real-time, fault-tolerant, secure, or high-availability systems. If the product's performance is not going to fulfill the requirements, this may be an instance when you should not buy a COTS product but instead should either build an implementation or relax a requirement. There is a broad spectrum between build and buy, and the use of open systems does not limit that spectrum. With open, COTS-based systems, you may not see performance benefits immediately, but if you are moving with the marketplace, performance may tend to improve over time because of technology insertion and product maturation.
Standards play a role in performance, too, although this is not often recognized. You must determine whether COTS products based on standards meet performance requirements. It may be advantageous for an organization to define a preferred set of standards to limit choices and thereby achieve greater interoperability.
Standards-based COTS products provide opportunities for adopting technological advances almost as they occur. But you must resist the temptation to chase the latest gadgets and performance capabilities. Stick to your standards, and make sure that changes make sense in the face of all trade-offs.
When assessing the performance of a COTS product, it is important to identify those components that are performance critical and to understand those components as part of the evaluation process. Refer back to the discussion of quality characteristics, especially Table 3.2. Consider the environment in which the COTS product needs to perform, and then build a test suite to evaluate candidate COTS products. Components can be rated by the importance of the purpose they serve within the system. If a system fails to perform in, for example, a hard real-time situation, the system's mission may fail.