Participatory consensual assessment technique: End-users assessing outcomes of participatory design

Abstract

The use of participatory methods in the design of innovative software systems is increasing. However, when users design, the maturity of their design skills can result in nonoptimal solutions. Their produced designs may sometimes extend well beyond the intended scope of the designed system, are infeasible to implement, or meaningless to the target audience. An accepted solution to the problem is to let design experts assess and refine the artefacts designed in participation. However, including experts into the design process brings expert-centeredness into a participatory project, which can violate the basic philosophy of user-centeredness. Another solution is using end-users as judges of the design solutions. However, the quality and the validity of assessment in this case is often questioned. The authors postulate that if an appropriate evaluation technique is used, the users can produce valid and reliable assessments. In this paper, the authors report on the conception and evaluation of such an end-user based participatory assessment technique, namely Participatory Consensual Assessment Technique (PCAT). The authors stage a series of experiments in which end-users of a case tool (Jeliot Mobile) assess design ideas of the said tool by employing PCAT. The statistical analysis of the assessment data establishes the evidence that a subset of end-users when employing PCAT produces valid, reliable and consistent assessments.