Judgment Test Research Articles

Clinical Assessment of Judgment in Adults and the Elderly: Development and Validation of the Three Domains of Judgment Test-Clinical Version (3DJT-CV).

(1) Background: This article discusses the first two phases of development and validation of the Three Domains of Judgment Test (3DJT). This computer-based tool, co-constructed with users and capable of being administered remotely, aims to assess the three main domains of judgment (practical, moral, and social) and learn from the psychometric weaknesses of tests currently used in clinical practice. (2) Method: First, we presented the 3DJT to experts in cognition, who evaluated the tool as a whole as well as the content validity, relevance, and acceptability of 72 scenarios. Second, an improved version was administered to 70 subjects without cognitive impairment to select scenarios with the best psychometric properties in order to build a future clinically short version of the test. (3) Results: Fifty-six scenarios were retained following expert evaluation. Results support the idea that the improved version has good internal consistency, and the concurrent validity primer shows that 3DJT is a good measure of judgment. Furthermore, the improved version was found to have a significant number of scenarios with good psychometric properties to prepare a clinical version of the test. (4) Conclusion: The 3DJT is an interesting alternative tool for assessing judgment. However, more studies are needed for its implementation in a clinical context.

Code-switching costs from Chinese-English relative clauses processing.

The source of costs is a primary concern in code-switching, yet a consensus has not yet been reached. This study investigates whether code-switching during syntactic processing in Chinese-English dual languages results in a cost. We use Chinese and English relative clauses in either object (Experiment 1) or subject (Experiment 2, which has a more complex structure) positions to test the costs in syntactic processing. Forty-seven Chinese-English bilinguals and 17 English-Chinese bilinguals participated in acceptability judgment tests and self-paced reading experiments. The statistical findings indicate that syntactic processing is a source of the costs incurred in code-switching, as evidenced by the code-switching costs observed in the head movement during relative clause comprehension. The outcomes are consistent with the implications of the 4-Morpheme Model and the Matrix Language Framework. Additionally, the experiment shows that the processing of relative clauses depends on the underlying structures, which is consistent with the Dependency Locality Theory.

From face-to-face to remote evaluation of teacher-education candidates during the COVID-19 pandemic.

Although teacher's personality is an essential component of successful classroom learning, most teacher education programs accept students solely on the basis of scholastic ability scores such as school grades, national test scores (SAT, GRE) or undergraduate college transcripts. To ensure suitability to teaching, a personality-evaluation system was developed in Israel for teacher education candidates. This evaluation system includes non-cognitive measures, such as group dynamic exercises, simulations, a teaching exercise, situational judgement tests, personality tests and an inter-personal interview, all performed face-to-face (FTF) at a testing center. The outbreak of COVID-19 in 2020 brought about lockdowns and social distancing, precluding the administration of this FTF evaluation system. Therefore, the development team adapted the system to online remote testing, using Zoom technology. The present study examined the effect of this transition to remote evaluation on the quality of selection for teaching, looking at suitability-to-teaching scores and the subjective views of applicants and evaluators. A comparison of the 2020 remote scores with the 2019 FTF scores revealed that scores on remote evaluation were slightly lower than FTF scores, and were more centralized. While the candidates found that remote evaluation provided fewer opportunities to express themselves, both candidates and evaluators were satisfied with the administration and convenience of the evaluation day. The Discussion chapter summarizes the unique affordances and constraints of remote evaluations and presents suggestions for changes which might be made when moving an assessment online that could take advantage of this new environment.

Optimal decision for the satellite navigation system under navigation countermeasures based on spoofing effect evaluation results

AbstractThis paper realizes the evaluation and optimal decision of the spoofing effect on the global navigation satellite system (GNSS) under navigation countermeasures based on the game perspective, to overcome the limitations of current relevant research findings and maintain the secure application of the GNSS and maximize its defensive benefit during countering. An evaluation system based on the perspective of the game between spoofing strategies and defense measures is created to build the benefit matrix, which can solve the limitation in building a benefit matrix (the vital element in a game) that conforms to the confrontation‐oriented GNSS spoofing properties. Based on the built benefit matrix, a method integrating the grey expert‐based system with the hybrid evolutionary method is developed to realize quantifying the matrix and further make the optimal decision stably, which can solve the uncertainties of qualitative indices during quantifying and no global optimal solution for benefit equations deduced from the quantified matrix. Finally, based on various judgment principles, simulation and testing results verify the effectiveness of the obtained results and the superiority of the proposed method by comparing the results solved by the method in this paper with the results obtained by other decision‐making processing methods.

The role of automatised and non-automatised explicit knowledge in general L2 proficiency

ABSTRACT This study examines the knowledge representation of Japanese university students assessed using grammaticality judgement tests (GJTs) and a metalinguistic knowledge test (MKT). The study also investigates the role of automatised and non-automatised explicit knowledge in general L2 language proficiency. Participants were 87 late learners of English as a foreign language (EFL) who completed the timed and untimed GJTs and MKT, modified Group Embedded Figures Test (GEFT), and the Test of English for International Communication (TOEIC). The principal component factor analysis results indicated that ungrammatical sentences in the GJTs loaded on non-automatised explicit knowledge, whereas grammatical sentences loaded on automatised explicit knowledge. The score for ungrammatical sections on the timed GJT was the most significant predictor of all components of the TOEIC. The results also indicated that the time pressure applied in the timed GJT could not sufficiently limit participants’ access to explicit knowledge when they processed grammatical sentences. The findings suggest that non-automatised explicit knowledge may play a more significant role than automatised explicit knowledge in L2 proficiency in the case of Japanese EFL learners.

Evolution of a Cocurricular Program to Support Student Professional Identity Formation

ObjectiveDescribe the development, implementation, and assessment of a cocurricular program intentionally designed to foster professional identity formation at a private institution. MethodsA cocurricular committee was established which developed a cocurriculum program in 3 phases. The committee utilized a gap analysis to inform the development of a continuing-education-based elective program (phase I), broadened program components and enhanced assessment (phase II), and continued to strengthen select areas of the affective domains through a second gap analysis and implement a summative assessment (phase III). ResultsOver 2 academic years, the completion rates for reflections, continuing education programming, and community outreach in the most recent academic year were consistently above 80% by the final due date. Rates of the mentor-mentee meetings fell below 50%; however, this component is tracked by the faculty member and not the students. Community outreach monitoring was managed by the committee for the first time in the 2021–2022 academic year and completion rates increased from 64% to 82%. Student reflections indicated consistent progression toward practice readiness from first- to third-year pharmacy students. For the Pharmacy Affective Domain Situational Judgment test, 22% and 16% of first-year pharmacy students were flagged during the first and second year implemented, respectively, and only 8% of the third-year pharmacy students were flagged in both years. ConclusionUtilization of a cocurricular committee has been vital for the development, progression, and assessment of the cocurriculum at a single private institution.

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

ObjectiveThe primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness. MethodsAn SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics. ResultsThirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics. ConclusionAn SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.

학부 번역교육에서 기계번역 메타언어 학습 활동에 대한 연구

The purpose of this study is to present Machine Translation (MT) metalanguage learning activities and to examine whether the degree of metalanguage utilization shown by undergraduate students majoring in translation is related to their explicit knowledge, and their ability to identify and solve translation problems. Metalanguage helps learners to learn foreign languages by activating the metalanguage knowledge they have about their native language and their ability to explain specific language phenomena or translation problems. Unlike the previous undergraduate machine translation education research focused on post-editing, this study attempts to use machine translation as a tool for translation learning with the purpose of promoting understanding and knowledge of translation and explaining theoretical approaches to translation problems. Using the machine translation result of the novel Pachinko, students conducted the following learning activities with “Translation Problem Judgment Tests”: 1) Understand the translation problems, 2) Indicate the translation problems, and 3) Explain and correct the translation problems. The analysis found that although the use of metalanguage was active, there was a case where the translation problem was not accurately identified and analyzed. This meant that the two sub-categories of explicit knowledge, metalanguage and analyzed knowledge, were not highly correlated. On the other hand, the higher the frequency of using the metalanguage, the higher the explicit knowledge needed to explain the translation problem. Taken together, these results suggest that meta-linguistic education enhances students' awareness of translation and ultimately helps expand explicit knowledge. The findings of this research have significant implications because they show that metalanguage learning activities using machine translation in undergraduate translation education can help students increase their explicit knowledge of translation and extend their analytical skills.

422 Utilizing Situational Judgment Test in Neurosurgery Residency Online Interviews

INTRODUCTION: Neurosurgery residency selection is heavily reliant on pre-residency factors and unstructured interviews. These metrics, however, have not been shown to reliably predict residency performance, and have potential to introduce bias and discrimination. Outside of medicine, the science of selection is much better developed, and methods such as structured interviews, situational judgment tests (SJT), and personality assessments are routinely used with robust evidence for efficacy and reliability at predicting job performance. METHODS: We designed a set of 4 SJT questions mimicking common situations a neurosurgery resident would encounter during residency. Each question had individualized grading rubrics scored on a scale of 1 to 4. For the 2021 residency application cycle, our program incorporated these SJT questions as part of the online interview process. Four volunteer faculty interview pairs delivered one question each for every candidate, and recorded the scores. Other traditional evaluation metrics from the candidate’s application and interview were also collected post interview. RESULTS: All 44 interview candidates applying to our program during the 2021 cycle were assessed. Due to time constraints, the number of questions each candidate responded to varied from 2 to 3. The average score was 3.6 (SD 0.44, range 2-4). The SJT score was not associated with USMLE Step 1 score (p = 0.31), number of publications (p = 0.42), or letter of recommendation score (p = 0.21). The SJT scores had a moderate positive correlation with personality on interview (r = 0.42, p < 0.01), and overall interview score (r = 0.30, p = 0.04). CONCLUSIONS: It is feasible to incorporate SJT questions into the neurosurgery resident online interview process. They may provide objective metrics independent of academic and research abilities. Further investigations with long term follow up is needed to see if SJT can be validated and correlated with residency performance.

Examining Validity for the Pharmacy Affective Domain Situational Judgment Test

Objective. With the integration of the affective domain in the pharmacy accreditation standards, it is important for programs to have methods for formatively assessing student competency in these areas. The objective of this study was to examine the validity evidence for a recently developed situational judgment test to assess the affective domain contained in the Accreditation Council for Pharmacy Education (ACPE) Standards 2016.Methods. After pilot testing items in multiple pharmacy schools, a revised 15-item situational judgment test instrument was developed measuring the affective domain as it relates to the pharmacy profession. The resulting Pharmacy Affective Domain Situational Judgment Test (PAD-S) was administered to students at three institutions and internal structure validity evidence was examined using item descriptives, Cronbach alpha, and results from a Rasch measurement model.Results. There were 559 valid responses included in the study. The items were of varying difficulty, with Rasch results confirming the hierarchy of items and suggesting that items were contributing to the measure. The Cronbach alpha was 0.70, suggesting acceptable reliability. However, the reliability items from the Rasch model were lower (0.65 and 0.66), indicating that the ranking of ability was limited, which may be due to fewer items of high difficulty.Conclusion. Overall, the PAD-S performed well as a measure of the affective domain. The PAD-S may be a useful formative instrument as part of a comprehensive assessment plan and may be less resource intensive than other measures.

The Relationship Between a Multiple Mini-interview and Situational Judgment Test for Admissions

Objective. To explore the relationship between a multiple mini-interview (MMI) and situational judgment test (SJT) designed to evaluate nonacademic constructs.Methods. A 30-question ranked-item SJT was developed to test three constructs also measured by MMIs during a pharmacy school's admissions process. First-year pharmacy students were invited to complete the SJT in fall 2020. One hundred four students took the SJT (82.5% response rate), with 97 (77% of possible participants) having MMI scores from the admissions process. Descriptive statistics and other statistical analyses were used to explore the psychometric properties of the SJT and its relationship to MMI scores.Results. Seventy-four percent of students identified as female (n=72), and 11.3% identified with an underrepresented racial identity (n=11). The average age, in mean (SD), was 21.8 (2.1) years. Students' mean (SD) scores were 85.5 (3.1) (out of 100 points) on the SJT and 6.1 (1.0) (out of 10 points) on the MMI. Principal components analysis indicated that the SJT lacked construct validity and internal reliability. However, reliability of the entire SJT instrument provided support for using the total SJT score for analysis (α=.63). Correlations between total SJT and MMI scores were weak (r p<0.29).Conclusion. Results of this study suggest that an SJT may not be a good replacement for the MMI to measure distinct constructs during the admissions process. However, the SJT may provide useful supplemental information during admissions or as part of formative feedback once students are enrolled in a program.

High school students’ teamwork ability in project-based learning of organic waste decomposition

Teamwork is an important skill to develop through school learning. This ability affects how a learner can learn independently and with a group to obtain academic achievement. This study applies project-based learning on decomposing organic waste in the biology subject to help develop learners' teamwork skills. This study was used by one group pretest-posttest design. The data in this study were collected through the administration of Situational judgment test sheets on pretest and posttest and self-report sheet of teamwork ability. The data were analyzed descriptively quantitatively to produce a better understanding. As a result, the ability to work in teams measured has increased although the increase has not been significant (n Gain = 0.2). The implementation of project-based learning on waste decomposition had an advantageous effect on teamwork ability (d = 0.5). The results in this study indicate that fostering teamwork through Project Based Learning can be optimized in the public product stage. The suggestion for the future research is to conduct pre-treatment to the research participants in the form of introduction to collaborative learning.

The importance of situation evaluation and the ability to identify criteria in a construct-driven situational judgment test

Situational judgment tests (SJTs) typically focus on candidates' judgment of various response options and ignore the process of arriving at that judgment. In three studies, we developed two situational construal measures (i.e., situation evaluation and the ability to identify criteria [ATIC]) based on a construct-driven SJT measuring cooperativeness. We tested their role in the response judgment process and examined their incremental validity in predicting work performance. Study 1 (N = 30) provided qualitative evidence that situation evaluation and ATIC shape response judgment. Study 2 (Nstudent = 136, Nemployee = 154) showed that situation evaluation predicts response judgment. Moreover, situation evaluation had incremental validity in predicting supervisor-rated task performance and peer-rated interpersonal performance over and above response judgment and ATIC. Furthermore, ATIC predicted response judgment and had incremental validity in predicting interpersonal performance but not task performance over and above response judgment. Finally, Study 3 (N = 266) replicated the findings of Study 2 using a new sample and yielded similar results regardless of the order in which situation evaluation and response judgment were measured.

Scrambling and Binding in Korean: An Experimental Investigation

This article provides an experiment-based study on scrambling involved in binding. There have been a large number of attempts to analyze scrambling in terms of A/A’-distinctions. These analyses are dependent upon the canonical Binding Theory(BT). Binding relations, however, do not always determine which copy of a scrambled phrase is interpreted. Besides, different acceptability judgments on the same sentence among researchers have been one of the impediments to proposing a far-reaching theory regarding scrambling. For this reason, this paper conducted an acceptability judgment testing to scrutinize the mixed nature of scrambling and binding. In the experiment, ‘scrambling effects’ are observed in all but one of the conditions; it suggests that scrambling-involved data can be analyzed by means of the inactiveness of the base copy. The parallels of the pronoun ku and the anaphor caki observed in scrambled contexts support the idea that the dependence on the BT for understanding the nature of scrambling is problematic.

Brain network mechanism on cognitive control task in the elderly with brain aging: A functional near infrared spectroscopy study.

To study the brain network mechanism of cognitive control in the elderly with brain aging. 21 normal young people and 20 elderly people were included in this study. Mini-mental State Examination and functional near-infrared spectroscopy (fNIRS) synchronous judgment test (including forward tests and reverse judgment tests) were performed on all subjects. To observe and compare differences in brain region activation and brain functional connectivity between subjects and forward and reverse trials by recording functional connectivity (FC) in different task paradigms and calculating bilateral prefrontal and primary motor cortical (PMC) areas. In the forward and reverse judgment tests, the reaction time of the elderly group was significantly longer than the young group (P < 0.05), and there was no significant difference in the correct rate. In the homologous regions of interest (ROI) data, the FC of PMC and prefrontal cortex (PFC) in the elderly group was significantly decreased (P < 0.05). In the heterologous ROI data, except for left primary motor cortex (LPMC)-left prefrontal cortex (LPFC), the other PMC and PFC of the elderly group were significantly lower than the young group (P < 0.05) while processing the forward judgment test. However, the heterologous ROI data of LPMC-right prefrontal cortex (RPFC), LPMC-LPFC and RPFC-LPFC in the elderly group were significantly lower than the young group (P < 0.05) while processing the reverse judgment test. The results suggest that brain aging affected degeneration of whole brain function, which reduce the speed of information processing and form a brain network functional connection mode different from that of young people.

The acquisition of L3 French present simple and present progressive by adult L1 Chinese speakers of L2 English

AbstractThe study examines the acquisition of L3 French present form by advanced L2 English learners of L1 Chinese within the framework of theTypological Primacy Model(TPM) of Rothman (2011) and theInterpretability Hypothesisof Tsimpli and Dimitrakopoulou (Tsimpli, Ianthi-maria &amp; Maria Dimitrakopoulou. 2007. The Interpretability Hypothesis: Evidence from wh-interrogatives in second language acquisition.Second Language Research23. 215–242. The meaning-form distinction of the present form in relation to habitual and event-in-progress interpretations varies between French, English, and Chinese, but is constrained by the presence/absence of tense and agreement features. Thirty-two adult L1 Chinese speakers of high proficiency L2 English/low proficiency L3 French and thirty adult L1 Chinese speakers of low proficiency L2 English together with a native English control group and a French control group were invited to take part in an acceptability judgement test. The results provide partial support to the TPM, which posits that the perceived typological similarity between L3 and the previously acquired languages conditions transfer in L3 initial state. The findings are, instead, more consistent with theInterpretability Hypothesis, which assumes the inaccessibility of uninterpretable syntactic features in subsequent language acquisition. What remain available in adult multilingual acquisition are interpretable syntactic features, computational devices, and other aspects of Universal Grammar (UG).

The relationships between individual differences in working memory and language analytical ability and the effectiveness of different types of corrective feedback

Abstract This study examined the relationships between individual differences in the executive component of working memory, and language analytical ability and the effectiveness of recasts, explicit correction and, metalinguistic prompts on the second-language (L2) development of the English past progressive. Participants were 155 Japanese learners of English, who were divided into recast, explicit correction, metalinguistic prompt, and task-only groups. The treatment groups completed tasks that required past progressive usage and received feedback. The executive component of working memory and language analytical ability were measured by the reading span test and LLAMA F. The outcome measures were an untimed grammaticality judgment test (UGJT) and elicited imitation test (EIT), which measured the participants’ declarative and procedural knowledge, respectively. The participants performed the pretest, immediate posttest, and delayed posttest. Multiple regression analyses showed that the executive component of working memory was significantly associated with the gain scores of the UGJT for the recast group and with the gain scores of the EIT for the metalinguistic prompt group. Language analytical ability was not associated with gain scores regardless of the test type for all groups.

THE EFFECTS OF AGE, PRESENTATION MODE (ONLINE, OFFLINE), AND SEGMENTATION OF AMBIGUOUS SENTENCES ON ATTACHMENT PREFERENCES OF FEMALE EFL LEARNERS

One kind of structural ambiguity is in sentences including two determiner phrases (DP) that are followed by a relative clause (RC). One example of this type of ambiguity is "Somebody shot the servant of the actress who was on the balcony". Speakers of different languages attribute RC either to DP1 or DP2. The goal of this study was to investigate the relationship between participants' age and participants' attachment preferences, as well as different ways of presenting the material (i.e., offline vs. online) on the attachment preferences. This study also aims to see if the segmentation of experimental sentences plays any role in participant attachment preferences. To this end, a sample of 50 female native speakers of Persian ranging between 15 to 25 years participated in this study. The instruments used in the present research include an offline sentence acceptability judgment test and main tests that included the first main test, offline test, online complete presentation (timed), and an online segment-by-segment sentence. For the analysis of data t-tests and ANOVA were used. The statistics showed that participants' age affects attachment preferences, and adolescents have a tendency toward DP1 selection. Different modes of presenting ambiguous sentences also affected the results; presenting materials in a self-paced online method leads to a significant difference other two modes. Finally, segmentation plays a role in the attachment preference of RC and in resolving ambiguous sentences. The results showed that students had a preference for segmented presentation rather than a holistic mode of presentation.&lt;p&gt; &lt;/p&gt;&lt;p&gt;&lt;strong&gt; Article visualizations:&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;&lt;img src="/-counters-/soc/0029/a.php" alt="Hit counter" /&gt;&lt;/p&gt;

The relationship between moral judgment ability, parenting style, and perfectionism in obsessive-compulsive disorder patients: A mediating analysis.

Guilt is an important part of obsessive-compulsive disorder. The abnormal moral cognition of obsessive-compulsive disorder patients may be closely related to their high level of guilt. The purpose of this study was to explore the development level of moral judgment in patients with obsessive-compulsive disorder and the role of parenting style and perfectionism in moral judgment development. A cross-sectional study was conducted in the clinical psychology department of a Class III hospital in Beijing. The patients with obsessive-compulsive disorder were recruited, and the healthy control subjects were recruited at the same time. Questionnaires were used to collect data, including the Yale-Brown Compulsion Scale, the Moral Judgment Test, the Parenting Style Evaluation Scale, and the Frost Multidimensional Perfectionism Scale. A total of 231 patients with obsessive-compulsive disorder and 246 healthy controls were included. The results showed that, first, the obsessive-compulsive group scored significantly lower on moral judgment than the healthy control group. Second, the tendency of non-adaptive perfectionism was significantly higher in the obsessive-compulsive group than in the healthy control group. Third, parents' excessive control, denial, punishment, and other parenting styles and non-adaptive perfectionism are higher than those of healthy people. Fourthly, the mother of obsessive-compulsive disorder patients is overly interference and protective. Rejection, denial, punishment, harshness, and father's rejection and denial play a partial mediating role in moral judgment ability through the degree of non-adaptive perfectionism. The development level of moral judgment ability of patients with obsessive-compulsive disorder was significantly lower than that of the normal group, and the level of non-adaptive perfectionism was significantly higher than that of the normal group. Parents of obsessive-compulsive patients use more high-pressure control education. Parenting style partially affects the moral judgment of obsessive-compulsive patients through the degree of non-adaptive perfectionism.

Development of cross-service training and job performance criterion measures for the U.S. Military

ABSTRACT As part of an effort to provide standardized criterion measures across the Armed Services, the current research effort developed a set of service-wide criterion measures for first-term enlisted personnel using a recently developed model of cross-service job performance. Two concurrent work streams developed criterion measures which provide complete construct coverage of the cross-service performance model. Using existing service-specific instruments, methods for development of the Cross-Service Situational judgment Test (CSSJT), the Cross-Service Job Performance Ratings Scales (CSPRS), and two cross-service self-report measures (end of training and in unit surveys) are described and discussed.