Content-related Evidence Research Articles

Explaining Neural News Recommendation with Attributions onto Reading Histories

An important aspect of responsible recommendation systems is the transparency of the prediction mechanisms. This is a general challenge for deep-learning-based systems such as the currently predominant neural news recommender architectures which are optimized to predict clicks by matching candidate news items against users’ reading histories. Such systems achieve state-of-the-art click-prediction performance, but the rationale for their decisions is difficult to assess. At the same time, the economic and societal impact of these systems makes such insights very much desirable. In this paper, we ask the question to what extent the recommendations of current news recommender systems are actually based on content-related evidence from reading histories. We approach this question from an explainability perspective. Building on the concept of integrated gradients, we present a neural news recommender that can accurately attribute individual recommendations to news items and words in input reading histories while maintaining a top scoring click-prediction performance. Using our method as a diagnostic tool, we find that: (a), a substantial number of users’ clicks on news are not explainable from reading histories, and many history-explainable items are actually skipped; (b), while many recommendations are based on content-related evidence in histories, for others the model does not attend to reasonable evidence, and recommendations stem from a spurious bias in user representations. Our code is publicly available 1 .

A Validation Study on PEPEC Based on Content-related Evidence

For the purpose of improving the overall flight English proficiency of Chinese pilots to ensure civil aviation flight safety, PEPEC expert group has conducted a series of researches in 2020 to optimize PEPEC test system. This study, employing ICAO rating scale and open-ended interview, validated PEPEC 900 Sentence Recitation optimized tasks administered in August 2020 on the basis of content-related evidence theory. The results indicate that:1) test tasks adequately represented the proposed constructs; 2) the objectives defined in each combination task were generally consistent with the test task objectives. Considering the evidence listed above, the conclusion of the study is that PEPEC 900 Sentence Recitation optimized tasks are of high validity in terms of content-related evidence.

THE LEVEL OF AWARENESS OF GUIDANCE AND COUNSELLING AT KOMENDA COLLEGE OF EDUCATION

Purpose: This study explored the nature of guidance and counselling services available for the students in Komenda College of Education.
 Methodology: The descriptive survey design was used for the study. A sample of 185 students consisting of 95 level 200 students and 90 level 300 students were sampled from the population. The major instrument used for data collection was questionnaire. Questionnaire was used because it is the most appropriate instrument that could be used to collect data on the variables for the study. Content-related evidence of validity was established for the instrument. The researchers personally administered the questionnaire. The entry of quantitative data and analysis was done using Statistical Package for the Social Sciences (SPSS).
 Findings: The results from the study showed that guidance and counselling services were vibrant and visible in Komenda College of Education. The study revealed that Seventy-five per cent (75%) of the respondents claimed that attitudes of Guidance and Counselling Coordinators and Tutors were very appropriate for counselling services. All types of guidance and counselling services were rendered in the college. The study further revealed that some students had negative attitudes towards accessing guidance and counselling services.
 Unique Contribution to Practice and Policy: The researchers therefore recommend that, school authorities should be provided with needed logistics for effective guidance and counselling services in schools and colleges. Also, there should be in-service trainings and refresher courses and seminars for guidance and counselling coordinators to up-date their skills in guidance and counselling. Finally, the Ghana Education Service should organise periodic educational programmes for both teachers and students on the role of guidance and counselling in schools and colleges.

Developing the Experience Impact Scale: A qualitative study using a study abroad and international internship program

Recent work has identified different experience types, including experience classifications of ordinary, memorable, meaningful, and transformative. This study further clarifies the experience typology through qualitative data analysis of each experience type. Data analysis identified that within each extraordinary experience type, additional subtypes were present. Experience types and subtypes were organized according to the experience typology continuum into a 10-item Experience Impact Scale (EIS), as no instrument to measure experience types or experience impact was available. An expert panel was used to determine the representativeness of the items in the EIS and to provide content-related evidence of validity. Discussion of the findings and future recommendations regarding the scale are provided.

Opening the black box of selection

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the ‘black box of selection’). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing’s validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants’ performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants’ results sheds light into the ‘black box of selection’ and can be used to support the construct validity of selection procedures.

Development and Initial Validation of the Computerized Evaluation Protocol of Interactions in Physical Education

The purpose of this study was to delineate the development of the Computerized Evaluation Protocol of Interactions in Physical Education, a data collection program for multiple interaction behavior measures in elementary inclusive physical education settings. The theoretical and empirical base for the Computerized Evaluation Protocol of Interactions in Physical Education is described in a three-phase study presenting (1) development of 25 multi-component interaction behavior items from presented literature, (2) content related evidence of validity evaluated by international adapted physical education experts, and (3) strong evidence of convergent validity of the Computerized Evaluation Protocol of Interactions in Physical Education. Overall, the outcomes of multiple studies indicated that the Computerized Evaluation Protocol of Interactions in Physical Education demonstrates promise for the adapted physical education research and assessment by documenting interaction behaviors in elementary physical education environments.

Is Content-Related Evidence Useful in Validating Selection Tests?

The 12 papers commenting on K. R. Murphy (2009a) raise a number of important issues, most of which can be subsumed in one of four themes. First, papers examining content-oriented validation strategies are still necessary and useful, in part because of the frequent use of these strategies in the practice of industrial–organizational (I–O) psychology. Second, the term “content validity” means many different things both within and beyond the field of I–O psychology, and it is useful to understand what sorts of inferences examinations of test content do and do not support. Third, these 12 papers present very little evidence that content validation, as typically carried out by I–O psychologists, actually provides information about the likelihood that people who do well on the test will do well on the job. Finally, I believe that the best use of content-related evidence in validating selection tests is in developing hypotheses about relationships between test scores and criteria rather than in testing these hypotheses.

Development of an Instrument to Measure Student Disruptive Behavior

The study was designed to develop an instrument to examine students' reports of the type and frequency of various student behaviors in physical education classes that might disrupt classroom management. This study had multiple phases, including instrument development, preliminary studies, content-related evidence study, and an exploratory factor analysis and internal consistency reliability study. Participants for the content-related evidence study were 27 experts in physical education pedagogy. They were sent items from the student disruptive behavior instrument and asked to organize them into 3 management severity categories. Participants for the factor analysis and internal consistency reliability study were 2,309 middle and high school students from 2 states. The following 6 factors emerged from the exploratory factor analysis: (a) aggressive, (b) low engagement or irresponsibility, (c) fails to follow directions, (d) illegal or harmful, (e) distracts or disturbs others, and (f) poor self-management. Alpha reliability coefficients for the factors were mostly above .70 and the rotated factor loadings for the items ranged from .26 to .73. Investigating students' reports of the type and frequency of potentially troublesome behaviors may lead to a better understanding of student disruptive behaviors in physical education contexts.

Importance of Expert Judgment in Content-Related Validity Evidence

The unified conceptualization of validity with regard to content-related evidence has been expressed succinctly by the authors of the Standards for Educational and Psychological Testing (AERA et al., 1985): Content-related evidence of validity is a central concern during [instrument] development, whether such development occurs in a research setting, in a publishing house, or in the context of professional practice. Expert professional judgment should play an integral part in developing the definition of what is to be measured, such as describing the universe of content, generating or selecting the content sample, and specifying the item format and scoring system. Thus, inferences about content are linked to [instrument] construction as well as to establishing evidence of validity after [an instrument] has been developed and chosen for use. (p. 11) This article has demonstrated the process of collecting content-related validity evidence in terms of the specific requirements of the Standards. Five standards were identified and interpreted according to the initial stages of instrument construction: domain specification, item development, and item, subscale, and scale content validation. The role of expert judgment during these stages and the variety of evidence that can be gathered were described. For most instruments, the review process would necessitate two meetings of 1 to 2 hours each to review the domain specifications and another two meetings to determine the match between the items and the specifications. The importance of these 8 hours or whatever additional time is needed to obtain the validity evidence was emphasized. Finally, an application of the Standards was provided to illustrate step-by-step how the judgmental review process can be planned and executed.

Development of a Diagnostic Instrument for Evaluating Tennis Serving Performance

A learning-sequence rating form for the slice tennis serve was developed and validated. Six skill components were identified and served as the basis for evaluation. Movement characteristics for observable learning stages were also described. The videotaped serving actions of 81 tennis players of different ability were then evaluated independently by trained raters using the rating form developed. A measure of the rating form's reliability was obtained using the intraclass correlation method (R) described by Baumgartner and Jackson in 1987. Content-related evidence for validity was also obtained. Intraclass reliability coefficients ranged from .70 to .96 across the six skill components identified in the final rating form. The tennis serve rating form provides researchers and coaches a reliable and valid instrument for evaluating tennis serving performance at various stages of learning.

DEVELOPMENT OF A DIAGNOSTIC INSTRUMENT FOR EVALUATING TENNIS SERVING PERFORMANCE

A learning-sequence rating form for the slice tennis serve was developed and validated. Six skill components were identified and served as the basis for evaluation. Movement characteristics for observable learning stages were also described. The videotaped serving actions of 81 tennis players of different ability were then evaluated independently by trained raters using the rating form developed. A measure of the rating form's reliability was obtained using the intraclass correlation method (R) described by Baumgartner and Jackson in 1987. Content-related evidence for validity was also obtained. Intraclass reliability coefficients ranged from .70 to .96 across the six skill components identified in the final rating form. The tennis serve rating form provides researchers and coaches a reliable and valid instrument for evaluating tennis serving performance at various stages of learning.