Courseware Evaluation Activities in Europe

Abstract

In the ECOSET-project data on evaluation activities of nineteen organizations in ten member states of the EG are gathered by questionnaire. Four of the organizations are interviewed for more and detailed information about these activities. Ten out of nineteen organizations do as well summative as formative evaluations and four organizations only perform summative evaluation activities. Five organizations are involved in formative evaluation activities only. The summative evaluation activities in the member states vary from a critical appraisal of a piece of courseware to extensive in-depth testing. Three groups of organizations are distinguished. The first group does not use formal instruments. The data indicate that this group are more or less on the line of product reviews of educational software. A second group (5 organizations) are using formal instruments, but the evaluation in real classroom situations is not a standard procedure. The third group of organizations use formal instruments and the educational software is tested in real classroom-situations. There are indications of differences between group 1 and 2 in the relation between development activities and the target group for the results. A remarkable aspect is the difference in dissemination of results between group 2 and group 3. Teachers in both groups are an important target group for the results, but in group 2 there is only one organization that use educational or other magazines for the dissemination of the results. In group 3 all the organizations make use of magazines. Fifteen organizations perform formative evaluation activities in the decision as well as in the design and development phase. Nine of them are also involved in evaluation activities in the dissemination phase. In the decision phase thirteen organizations do comparisons with other products and eleven of them are looking at the technical feasibility of products. Ten organizations do needs analysis studies. As evaluation activities in the design & development phase the evaluations of pre-releases of the software and release of the documentation are most often mentioned. Other activities are evaluations of prototypes and continuing consultancy during the design and technical implementation stages. Products most often mentioned as being used as input for activities in this phase are the descriptions of the ideas behind the product, global prototypes and product definitions. In the dissemination phase three organizations assess results as one of their evaluation activities. In the group of organizations performing formative evaluation activities two elements are remarkable. Organizations with few (<6) evaluation activities hardly train their evaluators. Almost all other organizations give training to in-house evaluators, subject matter specialists and/or teachers. If the number of evaluation activities gives an indication of the role of evaluation in the development process, you might say that those organizations also value training for evaluators as a necessity for good evaluations. The organizations with a lot of evaluation activities all use data of own or other observers. The aspect of training may also be connected with this fact. For observations the inter subjectivity is an important factor. Inter subjectivity you will get by training your observers.