Discussion of An Analysis of the Market for Systems Reliability Assurance Services

Abstract

It must be pretty clear to most people by now that as the market for and resultant dependency on purchased vendor-developed systems intensifies, a corresponding growth in interest for assurance about the “reliability” of such systems is occurring. Indeed, “reliability reports” and “reliability seals” can act as a form of proactive diligence for buyers in providing degree of up-front assurance that key reliability-risk issues have been dealt with. Arguably, with competition among system vendors inevitably growing fiercer, this can help differentiate market participants in the eyes of buyers and thus provide a level of advantage for vendors whose systems qualify to carry the seals vs. those that do not. However, all of this motivates questions regarding the most appropriate form of credible reliability-assurance reporting, the strength and competence of processes and standards leading to the development of such reports, and the effect on, or the reaction by, the market to the reports. In discussing the research in “An Analysis of the Market for System Reliability Assurance Services” (Arnold et. al. 2000) (hereafter, the research), which has looked at the issues around assurance reporting, I propose to focus on two subjects: the analysis itself and the market to which the analysis relates. My comments will generally follow the flow of the research report. Based on the application of an “Experimental Markets Methodology,” the research concludes that the market for assurance services for vendor systems best supports high-quality services, which should follow reporting models that distinguish between quality levels of systems reliability, i.e., graded-assurance reporting that differentiates between different levels of quality vs. traditional, or binary assurance reporting that communicates whether predetermined criteria are being met. It also concludes that as the report of choice with buyers and high-quality vendors, use of graded reports can allow high-quality vendors to drive lower quality vendors out of the market. When a traditional, binary-form report is available, buyers are more confused with less available information and vendors have less motivation to purchase the assurance reports. The thought and rigor behind the research leading to these conclusions is clearly competent and generally sound. The tightness of the logic and methodical interpretation of the results of the “Experimental Markets Methodology” that leads to the conclusions reflects this. It is hard therefore to not accept the conclusions as they are described. This said, the context for systems reliability assurance services is in practice, I believe, somewhat broader than that used in the research to develop the conclusions. A number of questions come to mind. One of the premises of the research is the perception based on the belief that the stature and reputation of the CA or CPA provides the accounting profession with a strategic advantage in competing with other service providers for systems reliability assurance services. Any major accounting-firm practitioner knows the reality of real-world competing and that they do not always win against the consulting arms of software development firms and the IBMs, Hewlett Packards, and other consulting groups. Further, as the introductory pages to the research report point out, numerous PC and other Information Technology journals routinely publish their own graded multidimensional reliability-assurance reports increasingly using standardized criteria. Having graded issues such as functionality, security, and user