An eye tracking experiment investigating synonymy in conceptual model validation

Abstract

A key advantage of conceptual models is that their quality can be evaluated and validated before beginning the costlier stages of information system development. Few research studies investigate the validation process for such models, particularly regarding multiplicities, even though multiplicity mistakes can be very costly. We investigated the validation of conceptual model multiplicities, varying how closely natural language statements of business rules match the models that purport to represent those rules. Participants in an eye tracking experiment completed validation tasks in which they viewed a statement and an accompanying UML class diagram in which a specified multiplicity was consistent with the statement (valid) or inconsistent with the statement (invalid). We varied whether the focal multiplicity was a minimum or a maximum and varied the class diagram’s semantics and order compared to that of the statement. Logistic regression was used to analyze the relationship between accuracy and the experimental manipulations and controls. The results show that the odds of accuracy in validating class diagrams that used synonyms instead of the exact statement terminology were only 0.46 times the odds of accuracy when the class diagram and statement words matched, showing a costly effect of synonymy. Interestingly, independent of the three levels of relative semantics, the odds of accuracy were 0.48 times when class diagrams were consistent with business rules as they were when class diagrams were inconsistent with business rules. To gain insight into cognition under correct task performance, we conducted additional linear regression analysis on various eye tracking metrics for only the accurate responses. Again, synonymy was observed to be costly, with a cognitive burden of increased integrative transitions between statement and model in the range of 39 to 66%.