Evaluation of MT systems: A programmatic view

Abstract

In this paper, I have tried to make several points. First, I have argued that the Human Translator Metaphor does not provide the most fruitful approach to evaluation of MT systems or perhaps even to the design of MT systems in general — one should rather go for specialized systems, whose properties are negotiated between designers and users. Second, I have tried to emphasise the dangers which the gear box evaluation syndrome raises in evaluating MT systems. Third, I have suggested an alternative basis for evaluation: rather than MT systems being seen as simply devices, they should be seen as pairs 〈Specs,Device〉, where the Specs constitute the fixed point for evaluation, and evaluation is seen in terms of both conformity checking between Device and Specs, and usefulness checking via comparison of user requirements and Specs. Though this raises some questions, I have tried to answer the most pressing worries one might have with the proposal, however, it is clear that this approach requires some effort to be put into specification methods and languages for specialized MT systems.