Summative Purpose Research Articles

Competency-Based Assessment Practices in Higher Education: Lessons from the Pandemics

This article reports on a research project on university teaching and learning in the context of pandemics. Sixteen university professors and fifteen bachelor’s degree students were interviewed regarding “emergency e-assessment practices” during the first lockdown semester at a Spanish institution. The research aimed to understand their perception of how generic competencies were being assessed. Data were generated in semi-structured individual interviews. The main findings are: (a) generic competencies are not explicitly considered in e-assessment practices; (b) online assessment practices follow mainly a summative purpose; (c) digital technologies are not considered for the instructional design; (d) both instructors and students lack assessment literacy. Furthermore, there are difficulties in reaching a shared understanding regarding what competency-based assessment means and its implications for daily praxis. The results underline the challenge of using digital technologies for fostering and assessing generic competencies, as well as the need for assessment literacy on both sides, teachers and students.

Reflexões sobre a utilização do Teste de Progresso na avaliação programática do estudante

Abstract: Introduction: The Progress Test (PT) is a well-established and mostly successful modality of student knowledge assessment in the health professions, mainly those in the medical area, with the potential to contribute substantially to the formative and informative purposes (quality control and indication of improvement in the teaching-learning processes). Additionally, the PT has characteristics that are adequate for its inclusion in institutional evaluation systems that facilitate the formative purpose, such as programmatic assessment (PA), but that also meet the summative purpose. In schools that have defined actions aimed at introducing PA in their undergraduate courses, it is necessary to reflect on the strengths and limitations of using PT in the evaluation system. Development: based on the considerations of a working group representative of the entire institution, tasked with proposing means of introducing PA in a new curriculum for the medical course, with international advice with experience in both PT and PA, we generated a reflection on this topic, based on the authors’ experience and data from the literature. It is proposed that, within the longitudinal perspective of the PA, the PT constitutes one of the pillars in the assessment of knowledge. The PT can be used as a basis for monitoring the students, in the context of their class (cohort), and its results should be discussed with the mentors who accompanies and supports them. The PT must also play a central role in management, as a source of information for eventual review and qualification of the curriculum and its teaching-learning activities. It is predictable that the use of the PT in PA will bring different challenges and barriers, which will be more easily overcome if the institution has already consolidated experiences in the application of institutional exams and in faculty development for the production of good quality objective questions. Conclusion: the effectiveness of the PT within the institutional PA system will depend on measures aimed at increasing its effectiveness in the assessment and that encourage the student’s active participation, reflecting on their performance in the PT, with the support of their mentor, aiming to engage in actions that encourage learning self-regulation.

Biology Undergraduate Students’ Views of Assessment:

This paper draws on data from research focusing on assessment methods, its purposes and effects, effectiveness and fairness, as well as student participation and feedback. In total, 124 Biology undergraduate students participated in the study. Data were collected through questionnaires and focus groups. Findings reveal written test as the most used method and that students associate assessment mainly with a summative purpose. Although students report they negotiate some assessment-related aspects with their teachers, in general they do not participate in self and peer assessment. Feedback is clearly recognized an important feature in the assessment process. Implications of the present findings are discussed.

Uses of Authentic Assessment Tools: Implications towards Competence Acquisition among Undergraduate Prospective Science Teachers in Tanzania

The study was concerned with the uses of authentic assessment tools in higher education institutions, particularly portfolios and practical work. The study employed the mixed method research approach with the explanatory sequential research design. The sample for the study involved 231 third year undergraduate prospective science teachers who were selected randomly and 6 instructors who were purposively selected from two higher learning institutions. Data was collected by using a questionnaire and a semi-structured interview. The findings indicated that authentic assessment tools namely portfolios and practical works were used both formatively and summatively though summative use was dominant. Portfolios were found to be minimally used both for formative and summative purpose. Findings from interview indicated that awareness on the formative uses of authentic assessment tools among instructors was minimal since summative use of such tools was observed to be dominant. It is concluded that the use of authentic assessment tools ranged from formative to summative, though summative was more dominant. Therefore it is recommended that there should be policy on the authentic assessment in higher education institutions to guide the effective use in both summative and summative aspects.

Supervisors’ untrained postgraduate rubric use for formative and summative purposes

Using rubrics can benefit the quality of assessment and learning. However, the conditions that stimulate or obstruct these benefits have been insufficiently studied. One underinvestigated claim is that rubrics are no substitution for good instruction and assessment and that teachers need training in utilising them. This is relevant since teachers in daily practice often use rubrics without training. In this study, we investigated data from a rubric filled out by supervisors who were not specifically trained in its use and used the rubric voluntarily. The rubric was designed for assessment moments with a formative and summative purpose. Results of quantitative analyses of 313 rubric forms indicated that the rubric was used flexibly: supervisors vary in using the rubric for the formative and/or summative purpose and in the criteria they assess. More criteria were omitted during formative use than during summative use. Some of these omitted criteria were most predictive for the final grade. This raises serious concerns with respect to the tension between flexible rubric use and constructive alignment. To understand the quality of rubric use in education, future research is needed on supervisors’ perceptions toward rubrics use. Supplemental data for this article is available online at https://doi.org/10.1080/02602938.2021.2021390 .

Validating a problem-based learning process assessment tool in a Nepalese medical school

Introductions: The newly established Patan Academy of Health Sciences (PAHS) has incorporated the measurement of non-cognitive skills and behaviors into the summative assessment in the setting of problem based learning (PBL). This study was conducted to validate a PBL process assessment tool for PAHS. 
 Methods: A list of 72 items of student behaviors observable in PBL tutorials was compiled from literature review. They were categorized under ten broad dimensions consistent with predefined PAHS Graduate Attributes. A series of PBL project committee meetings and expert inputs refined the list of 72 items to 47 and categorized them under eight dimensions. These 47 items, each with a 4-point rating scale, formed the Tutor Assessment of Student Tool (TAS-Tool). Twenty-four trained faculty members used the TAS-Tool to evaluate the performance of 41 senior high school students in PBL tutorials. 
 Results: The internal-consistency of the TAS-Tool was very high (Cronbach’s α = 0.954). Removal of two inconsistent items further increased it to 0.975. Principal components analysis with varimax rotation applied to the remaining 45 items gave seven components and explained 69.47% of the variation between the components. These seven components (% variation) were: Immersed in the Tutorial Process (20.16%); Professional (12.71%); Communicator and Team Leader (11.25%); Critical Thinker (8.77%); Reflector (6.22%); Creative (5.95%), and Sensitive (4.41%). 
 Conclusions: TAS-Tool was found to be reliable and valid instrument deemed applicable in formative PBL process assessment at PAHS starting with the pioneer cohort of medical students. Further validation of TAS-Tool through longitudinal study with PAHS students is required for summative purpose.

Validating a Problem-Based Learning Process Assessment Tool in a Nepalese Medical School

Introduction: The newly established Patan Academy of Health Sciences (PAHS) has incorporated measurement of non-cognitive skills and behaviours into the summative assessment in the setting of problem based learning (PBL). This study aims to validate a PBL process assessment tool for PAHS. Methods: A list of 72 items of student behaviours observable in PBL tutorials was compiled from literature review. They were categorized under ten broad dimensions consistent with predefined PAHS Graduate Attributes. A series of PBL project committee meetings and expert inputs refined the list of 72 items to 47 and categorized them under eight dimensions. These 47 items, each with a 4-point rating scale, formed the Tutor Assessment of Student Tool (TAS-Tool). Twenty-four trained faculty members used the TAS-Tool to evaluate the performance of 41 senior high school students in PBL tutorials. Results: The internal-consistency of the TAS-Tool was very high (Cronbach’s α = 0.954). Removal of two inconsistent items further increased it to 0.975. Principal components analysis with varimax rotation applied to the remaining 45 items gave seven components and explained 69.47% of the variation between the components. These seven components (% variation) were: Immersed in the Tutorial Process (20.16%); Professional (12.71%); Communicator and Team Leader (11.25%); Critical Thinker (8.77%); Reflector (6.22%); Creative (5.95%), and Sensitive (4.41%). Conclusion: TAS-Tool was found to be a reliable and valid instrument and applicable in formative PBL process assessment at PAHS starting with the pioneer cohort of medical students. Further validation of TAS-Tool through longitudinal study with PAHS students is required for summative purpose.

A methodology for applying students' interactive task performance scores from a multimedia-based performance assessment in a Bayesian Network

Computer-based simulations are increasingly being used in educational assessment. In most cases, the simulation-based assessment (SBA) is used for formative assessment, which can be defined as assessment for learning, but as research on the topic continues to grow, possibilities for summative assessment, which can be defined as assessment of learning, are also emerging. The current study contributes to research on the latter category of assessment. In this article, we present a methodology for scoring the interactive and complex behavior of students in a specific type of SBA, namely, a Multimedia-based Performance Assessment (MBPA), which is used for a summative assessment purpose. The MBPA is used to assess the knowledge, skills, and abilities of confined space guard (CSG) students. A CSG supervises operations that are carried out in a confined space (e.g., a tank or silo). We address two specific challenges in this article: the evidence identification challenge (i.e., scoring interactive task performance), and the evidence accumulation challenge (i.e., accumulating scores in a psychometric model). Using expert ratings on the essence and difficulty of actions in the MBPA, we answer the first challenge by demonstrating that interactive task performance in MBPA can be scored. Furthermore, we answer the second challenge by recoding the expert ratings in conditional probability tables that can be used in a Bayesian Network (a psychometric model for reasoning under uncertainty and complexity). Finally, we validate and illustrate the presented methodology through the analysis of the response data of 57 confined space guard students who performed in the MBPA.

Examining the Internal Structure Evidence for the Performance Assessment for California Teachers

Interpretations for licensure tests involve a series of inferences or a validity argument, leading from the test score to decisions about who is accepted or denied entry into a profession. Utilizing an argument-based framework for validation based on the Standards for Educational and Psychological Testing, we explore the evidence for the ongoing use of the Performance Assessment for California Teachers (PACT) for Tier I licensure decisions. The evidence for a unidimensional and a multidimensional structure based on the instrument’s content are examined with an item response model. Examining operational data ( n = 1,711) from seven California teacher education institutions, we found sufficient internal structure validity evidence to support the continued, but limited, use of this instrument for its intended summative purpose. Evidence for a three-dimensional structure of model fit better explains overall teacher candidate performance on the PACT instrument as it is currently designed.

¿Para qué sirve la evaluación de la docencia? Un estudio exploratorio de las creencias de los estudiantes

El propósito de este trabajo fue explorar las creencias de los estudiantes acerca de la evaluación de la docencia, sus fines, utilidad y validez. Participaron 163 estudiantes de carreras de las Ciencias Sociales y Humanidades de una universidad pública mexicana. Los resultados del estudio indican que los estudiantes consideran que la evaluación de la docencia tiene un fin sumativo, de control y supervisión del trabajo del docente, que tiene poca utilidad y los cuestionarios de evaluación no son válidos. Asimismo, de manera secundaria consideran que los fines de la evaluación docente son de tipo formativo. Se concluye que las políticas de evaluación de la docencia vistas a través de los estudiantes, no han tenido el efecto de mejorar la calidad de la docencia. Asimismo que las creencias de los estudiantes en torno a la evaluación docente son semejantes a los hallazgos producidos por la investigación reciente que critica el uso de los cuestionarios para evaluar la docencia con el fin de elevar la calidad de la enseñanza. Se discuten los resultados.

Power of the policy: how the announcement of high-stakes clinical examination altered OSCE implementation at institutional level

BackgroundThe Objective Structured Clinical Examination (OSCE) has been widely applied as a high-stakes examination for assessing physicians’ clinical competency. In 1992, OSCE was first introduced in Taiwan, and the authorities announced that passing the OSCE would be a prerequisite for step-2 medical licensure examination in 2013. This study aimed to investigate the impacts of the announced national OSCE policy on implementation of OSCE at the institutional level. Further, the readiness and the recognition of barriers toward a high-stakes examination were explored.MethodsIn 2007 and 2010, the year before and after the announcement of high-stakes OSCE policy in 2008, respectively, questionnaires on the status of OSCE implementation were distributed to all hospitals with active OSCE programs in Taiwan. Information on OSCE facilities, equipment, station length, number of administrations per year, and the recognition of barriers to the success of implementing an OSCE were collected. The missing data were completed by telephone interviews. The OSCE format, administration, and facilities before and after the announcement of the nationwide OSCE policy were compared.ResultsThe data were collected from 17 hospitals in 2007 and 21 in 2010. Comparing the OSCE formats between 2007 and 2010, the number of stations increased and the station length decreased. The designated space and the equipment for OSCE were also found to have been improved. As for the awareness of OSCE implementation barriers, the hospital representatives concerned mostly about the availability and quality of standardized patients in 2007, as well as space and facilities in 2010.ConclusionsThe results of this study underscored an overall increase in the number of OSCE hospitals and changes in facilities and formats. While recruitment and training of standardized patients were the major concerns before the official disclosure of the policy, space and facilities became the focus of attention after the announcement. The study results highlighted the influence of government policy on different aspects of OSCE implementation in Taiwanese training institutes that showed high level of support as reflected in the improved hardware and the change in OSCE format to serve the summative purpose.

E-Assessment as a Self-Test Quiz Tool: The Setting Features and Formative Use

The setting features of an e-assessment can impact the adoption of the e-assessment in general and impact a learner's learning process in particular. The setting features of an e-assessment which is used as a self-test quiz tool are reviewed and examined in this paper. A survey questionnaire was administered to 324 students to gather their perceptions on the setting features of the e-assessment system. It was found that time duration is an important factor. The instructor should be more careful when setting the duration for a self-test online quiz tool to serve the formative purpose. An online assessment can serve the formative as well as the summative purpose of assessment depending on how the setting features are customised. Proper settings should be used to facilitate learning if online assessment is to be used as a self-test quiz tool to serve the formative purposes.

Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology

Objectives To compare user satisfaction and acceptability, reliability and validity of three different methods of assessing the surgical skills of trainees by direct observation in the operating theatre across a range of different surgical specialties and index procedures. Design and setting A 2-year prospective, observational study in the operating theatres of three teaching hospitals in Sheffield. Methods The assessment methods were procedure-based assessment (PBA), Objective Structured Assessment of Technical Skills (OSATS) and Non-technical Skills for Surgeons (NOTSS). The specialties were obstetrics and gynaecology (O&G) and upper gastrointestinal, colorectal, cardiac, vascular and orthopaedic surgery. Two to four typical index procedures were selected from each specialty. Surgical trainees were directly observed performing typical index procedures and assessed using a combination of two of the three methods (OSATS or PBA and NOTSS for O&G, PBA and NOTSS for the other specialties) by the consultant clinical supervisor for the case and the anaesthetist and/or scrub nurse, as well as one or more independent assessors from the research team. Outcome measures Information on user satisfaction and acceptability of each assessment method from both assessor and trainee perspectives was obtained from structured questionnaires. The reliability of each method was measured using generalisability theory. Aspects of validity included the internal structure of each tool and correlation between tools, construct validity, predictive validity, interprocedural differences, the effect of assessor designation and the effect of assessment on performance. Results Of the 558 patients who were consented, a total of 437 (78%) cases were included in the study: 51 consultant clinical supervisors, 56 anaesthetists, 39 nurses, 2 surgical care practitioners and 4 independent assessors provided 1635 assessments on 85 trainees undertaking the 437 cases. A total of 749 PBAs, 695 NOTSS and 191 OSATSs were performed. Non-O&G clinical supervisors and trainees provided mixed, but predominantly positive, responses about a range of applications of PBA. Most felt that PBA was important in surgical education, and would use it again in the future and did not feel that it added time to the operating list. The overall satisfaction of O&G clinical supervisors and trainees with OSATS was not as high, and a majority of those who used both preferred PBA. A majority of anaesthetists and nurses felt that NOTSS allowed them to rate interpersonal skills (communication, teamwork and leadership) more easily than cognitive skills (situation awareness and decision-making), that it had formative value and that it was a valuable adjunct to the assessment of technical skills. PBA demonstrated high reliability (G > 0.8 for only three assessor judgements on the same index procedure). OSATS had lower reliability (G > 0.8 for five assessor judgements on the same index procedure). Both were less reliable on a mix of procedures because of strong procedure-specific factors. A direct comparison of PBA between O&G and non-O&G cases showed a striking difference in reliability. Within O&G, a good level of reliability (G > 0.8) could not be obtained using a feasible number of assessments. Conversely, the reliability within non-O&G cases was exceptionally high, with only two assessor judgements being required. The reasons for this difference probably include the more summative purpose of assessment in O&G and the much higher proportion of O&G trainees in this study with training concerns (42% vs 4%). The reliability of NOTSS was lower than that for PBA. Reliability for the same procedure (G > 0.8) required six assessor judgements. However, as procedure-specific factors exerted a lesser influence on NOTSS, reliability on a mix of procedures could be achieved using only eight assessor judgements. NOTSS also demonstrated a valid internal structure. The strongest correlations between NOTSS and PBA or OSATS were in the ‘decision-making’ domain. PBA and NOTSS showed better construct validity than OSATS, the year of training and the number of recent index procedures performed being significant independent predictors of performance. There was little variation in scoring between different procedures or different designations of assessor. Conclusions The results suggest that PBA is a reliable and acceptable method of assessing surgical skills, with good construct validity. Specialties that use OSATS may wish to consider changing the design or switching to PBA. Whatever workplace-based assessment method is used, the purpose, timing and frequency of assessment require detailed guidance. NOTSS is a promising tool for the assessment of non-technical skills, and surgical specialties may wish to consider its inclusion in their assessment framework. Further research is required into the use of health-care professionals other than consultant surgeons to assess trainees, the relationship between performance and experience, the educational impact of assessment and the additional value of video recording. Funding The National Institute for Health Research Health Technology Assessment programme.

How do Dutch secondary teachers and students conceive the purpose of assessment?

How teachers and students deal with assessment practices is largely influenced by the conceptions they hold about the purpose of assessment. This means that, given the current plea for a paradigm shift from Assessment of Learning to Assessment for Leaning with a balance between the formative and summative purposes of assessment, the study of teachers’ and students’ conceptions of the purpose of assessment is of the utmost relevance. In this study Dutch secondary school teachers’ ( n = 351) and students’ ( n = 712) conceptions of the purpose of assessment were measured using a translation of the teachers’ conceptions of assessment questionnaire and a translated and shortened version of the students’ conceptions of assessment questionnaire. Maximum Likelihood factor analyses resulted in four teacher and five student conceptions of the purpose of assessment. The results indicate that the teachers in our sample did not distinguish between formative and summative purposes of assessment. They did discern classroom assessment (formative as well as summative) from school accountability. In addition, they conceive assessment as being of bad (in terms of measurement errors) or good quality. In contrast with the teachers, students do make a distinction between the formative and summative purpose of assessment. They also refer to school accountability and the affects that assessment triggers in them as further purposes of assessment.

Teaching effectiveness as measured by student evaluation of teaching: an empirical study

This study focuses on the formative aspect of student evaluation of teachings (SETs) as one method of measuring teaching effectiveness from the student perspective. This study does not focus on the summative purpose of SETs, where the primary intent is forming a basis for tenure/promotion and/or pay for performance (raises). A longitudinal study of SET in an undergraduate operations management course required by all business majors, along with the grade point averages (GPAs) for the corresponding semester, provided the basis for predicting which questions (variables) on the SET predicted the students perception of overall teaching effectiveness and the impact of the SET on student GPA. Triangulation consisted of descriptive statistics analysis, variable selection regression, multiple regression with different dependent variables and exploratory principal components analysis. The results support that SETs can be used to measure overall teaching effectiveness for formative evaluation.

Clinician's Commentary.

Clinician's Commentary.

Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology

To compare user satisfaction and acceptability, reliability and validity of three different methods of assessing the surgical skills of trainees by direct observation in the operating theatre across a range of different surgical specialties and index procedures. A 2-year prospective, observational study in the operating theatres of three teaching hospitals in Sheffield. The assessment methods were procedure-based assessment (PBA), Objective Structured Assessment of Technical Skills (OSATS) and Non-technical Skills for Surgeons (NOTSS). The specialties were obstetrics and gynaecology (O&G) and upper gastrointestinal, colorectal, cardiac, vascular and orthopaedic surgery. Two to four typical index procedures were selected from each specialty. Surgical trainees were directly observed performing typical index procedures and assessed using a combination of two of the three methods (OSATS or PBA and NOTSS for O&G, PBA and NOTSS for the other specialties) by the consultant clinical supervisor for the case and the anaesthetist and/or scrub nurse, as well as one or more independent assessors from the research team. Information on user satisfaction and acceptability of each assessment method from both assessor and trainee perspectives was obtained from structured questionnaires. The reliability of each method was measured using generalisability theory. Aspects of validity included the internal structure of each tool and correlation between tools, construct validity, predictive validity, interprocedural differences, the effect of assessor designation and the effect of assessment on performance. Of the 558 patients who were consented, a total of 437 (78%) cases were included in the study: 51 consultant clinical supervisors, 56 anaesthetists, 39 nurses, 2 surgical care practitioners and 4 independent assessors provided 1635 assessments on 85 trainees undertaking the 437 cases. A total of 749 PBAs, 695 NOTSS and 191 OSATSs were performed. Non-O&G clinical supervisors and trainees provided mixed, but predominantly positive, responses about a range of applications of PBA. Most felt that PBA was important in surgical education, and would use it again in the future and did not feel that it added time to the operating list. The overall satisfaction of O&G clinical supervisors and trainees with OSATS was not as high, and a majority of those who used both preferred PBA. A majority of anaesthetists and nurses felt that NOTSS allowed them to rate interpersonal skills (communication, teamwork and leadership) more easily than cognitive skills (situation awareness and decision-making), that it had formative value and that it was a valuable adjunct to the assessment of technical skills. PBA demonstrated high reliability (G > 0.8 for only three assessor judgements on the same index procedure). OSATS had lower reliability (G > 0.8 for five assessor judgements on the same index procedure). Both were less reliable on a mix of procedures because of strong procedure-specific factors. A direct comparison of PBA between O&G and non-O&G cases showed a striking difference in reliability. Within O&G, a good level of reliability (G > 0.8) could not be obtained using a feasible number of assessments. Conversely, the reliability within non-O&G cases was exceptionally high, with only two assessor judgements being required. The reasons for this difference probably include the more summative purpose of assessment in O&G and the much higher proportion of O&G trainees in this study with training concerns (42% vs 4%). The reliability of NOTSS was lower than that for PBA. Reliability for the same procedure (G > 0.8) required six assessor judgements. However, as procedure-specific factors exerted a lesser influence on NOTSS, reliability on a mix of procedures could be achieved using only eight assessor judgements. NOTSS also demonstrated a valid internal structure. The strongest correlations between NOTSS and PBA or OSATS were in the 'decision-making' domain. PBA and NOTSS showed better construct validity than OSATS, the year of training and the number of recent index procedures performed being significant independent predictors of performance. There was little variation in scoring between different procedures or different designations of assessor. The results suggest that PBA is a reliable and acceptable method of assessing surgical skills, with good construct validity. Specialties that use OSATS may wish to consider changing the design or switching to PBA. Whatever workplace-based assessment method is used, the purpose, timing and frequency of assessment require detailed guidance. NOTSS is a promising tool for the assessment of non-technical skills, and surgical specialties may wish to consider its inclusion in their assessment framework. Further research is required into the use of health-care professionals other than consultant surgeons to assess trainees, the relationship between performance and experience, the educational impact of assessment and the additional value of video recording.

Avaliação do Desempenho Docente: Um novo enquadramento para o exercício do ensino

Partindo da experiência de formação desenvolvida junto dos educadores e professores no âmbito da implementação do novo modelo de avaliação do desempenho docente, apresentamos algumas contribuições para a compreensão dos seus pressupostos que implicam transformações não só ao nível do exercício do ensino, mas também ao nível das diversas estruturas que o enquadram. A complexidade de que se reveste qualquer modelo de avaliação de desempenho docente não deve obstar a que as escolas desenvolvam um espírito crítico, no sentido de identificarem e operacionalizarem aquilo que é realmente importante considerar. O desafio consiste em combinar o propósito formativo e de supervisão, com vista ao desenvolvimento profissional, com o propósito sumativo, da responsabilização pelo sucesso académico dos alunos. As propostas de melhoria do sistema de avaliação de professores que, no final, enunciamos afiguram-se pertinentes, devendo contribuir para a adequada concretização de uma avaliação com cujos objectivos os professores se identifiquem.

Evaluating the quality of teaching in medical education: are we using the evidence for both formative and summative purposes?

Heightened focus on the quality of teaching in medical schools has led to increased use of student surveys as a means of evaluating teaching. In some institutions, however, the summative purpose is emphasized over the developmental or self-improvement objective. The aim of this study was to examine the utility of a student evaluation of teaching questionnaire for use in administrative decision making (summative purpose) and in identification of faculty whose instruction needs improvement (formative purpose). Results of this analysis are discussed in addition to related issues regarding faculty development and the importance of adopting multiple methods of evaluating and documenting teaching and other educational activity.

A contingent approach to strategy and tactics in formative and summative evaluations

A contingent approach to strategic and tactical choices in formative and summative evaluations is presented. Choices regarding evaluation phases and tactics were found to be dependent on evaluation goals (e.g., whether the evaluation has a formative or summative purpose) and contextual variables. A simulation was designed to pool the judgments of evaluators and users of evaluations to see how they felt evaluation strategy and tactics should change as the situation changes so as to increase the likelihood of evaluation success. Eight panels were constructed to examine eight different situations. The next step in this research is to determine whether following the pooled judgments does lead to greater evaluation success.