Validity Issues Research Articles

Introduction of English-Speaking Observation Method and Process

The evaluation and monitoring of English speaking skills has gained great prominence in today’s world which has been globalized to such an extent that English is used both at the academic and occupational level. This research paper attempts to shed light on the strategies and the practices of the assessment of spoken English ability in different settings. After a literature review, original research, and case studies, we discuss the available methodologies for the observation, their efficacy, and what can be done, if anything, to enhance them. The results of this study suggest that the best way to assess speaking ability is to use both quantitative and qualitative measuring components. Research literature and practice dealing with the traditional interviews, computer based testing (CBT), e-assessments, and recent developments in the area of peer assessments are considered. The paper discusses the concerns of instructors and evaluators, such as those related to issues of reliability and validity and on the other hand, practicality. Ethnocultural aspects, as well as non verbal aspects of speaking, and even more, enhancing speaking with artificial intelligence are discussed as well. These days, new technologies facilitate solving many problems. However, these findings imply that considering all the human aspects of communication is also very important. The paper concludes with practical recommendations for teachers, researchers, and language testers that relate to improvement of techniques of performance observation and to the enhancement of English language teaching and learning in the changing world.

Revisiting Validity in Research With Multilingual Learners: Moving From a Monolingual to a Multilingual Lens on Research Quality

A critical revisiting of research methods with multilingual learners raises issues of validity that are often unaddressed in current discussions of research quality. Though validity is conceptualized and assessed differently across research traditions, validity broadly describes the production of findings that are credible and trustworthy. The historically monolingual lens of research methodologies has led to conceptions of validity that are inadequate for ensuring that reported findings “fit the data” in research with multilingual learners. In this article, I describe a retrospective inquiry to critically reassess how current paradigms of validity might be reconceptualized in research with multilingual learners. Through a reexamination of research practices from two of my own studies, this article considers three ways that current monolingual paradigms of validity should be expanded in research with multilingual learners to work toward multilingual conceptualizations of research quality. This article then offers guidance to aid other researchers in taking up issues of validity in the design and reporting of research with multilingual learners.

Impact of time window on estimating representativeness error in sea surface salinity data

ABSTRACT Validating satellite data using buoys presents a significant challenge due to the differing spatial and temporal representativeness of these data sources. This discrepancy introduces two critical issues in satellite data validation: the selection of the matchup window during the collocation process and the estimation of representativeness error. While numerous studies have addressed these issues independently, research on their interaction remains limited. Utilizing quadruple Sea Surface Salinity (SSS) matchup datasets, this study investigates the impact of the time window on the estimation of SSS representativeness error between two satellite observations: SMOS and SMAP. Our results reveal a clear dependence of representativeness error on the selected time windows. In the global ocean, the variance of representativeness errors within a small time window is 0.040 psu2, which is substantially greater than that of the entire dataset (0.016 psu2). In contrast, the representativeness errors within larger time windows are systematically lower than those within smaller time windows. This research also finds that this dependency is not influenced by the randomness of the matchup data, climatological Sea Surface Salinity, or geographic locations. Instead, our findings suggest that the impact of the time window on representativeness error may reflect the characteristics of satellite data across different time intervals. Satellites are more likely to capture SSS signals within smaller time windows, resulting in larger representativeness errors when comparing satellite SSS with buoy measurements. Conversely, larger time windows lead to significant mismatches between satellite observations and hinder the ability of satellites to detect SSS signals, which can result in smaller or even reversed representativeness errors.

Development and Preliminary Testing of the Staffordshire Questionnaire for Adolescent Idiopathic Scoliosis (SQ-AIS): Content and Face Validity.

Adolescent Idiopathic Scoliosis (AIS) is a structural spinal deformity with implications for health-related quality of life (HR-QoL). The Scoliosis Research Society-22 revised (SRS-22r) questionnaire is the standard for HR-QoL assessment. However, studies have identified limitations with the SRS-22r, including content and face validity issues, reliability concerns, and language appropriateness. This study aimed to develop and validate a patient-reported questionnaire, the Staffordshire Questionnaire for Adolescent Idiopathic Scoliosis (SQ-AIS), to assess the impact of AIS on HR-QoL. The SQ-AIS comprises six domains: general health, pain, function/activity, self-image/appearance, mental health, and intervention. Individuals with AIS aged 10-19 years and clinicians from a range of countries with expertise in AIS contributed to the testing process. Face validity and clinical applicability were assessed using Likert scales, while content validity was evaluated through a categorical binary variable (yes/no). Involving 8 AIS patients and 43 clinicians, face validity scores demonstrated an acceptable level of understanding (≥ 4/5) for both individuals with AIS and clinicians. Most individuals with AIS (85.71%) and clinicians (80.95%) affirmed that the questionnaire sufficiently covers various aspects of scoliosis, indicating a satisfactory level of content validity. Ratings for applicability to clinical practice indicated an acceptable level of practical relevance (≥ 4/5). The SQ-AIS emerges as a valid and promising tool to overcome existing challenges in AIS-related outcome assessment. Pending further validation studies, the favorable reception from the international community of clinicians suggests its potential as a new benchmark for evaluating AIS impact on HR-QoL and monitoring scoliosis management.

A comprehensive item bank of internal validity issues of relevance to in vitro toxicology studies

A comprehensive item bank of internal validity issues of relevance to in vitro toxicology studies

Principle-oriented versus Information-oriented Approach Towards Neuroanatomy: A Selective Profile of Question Papers of Four Public University of Bangladesh

Most of the students considered neuroanatomy to be a difficult subject that contains a large volume of information. Therefore, many of them have the tendency to memorise the subject rather than understanding. The learning will be permanent if learning and its assessment is structured around generalisable rules or ‘principle’ rather than trying to memorise plenty of apparently disconnected ‘information’. Materials and methods: The Neuroanatomy portion of the Anatomy question papers of the First Professional MBBS Examinations of four public universities were analysed. It was a cross sectional, observational study carried out in the Department of Anatomy, Bangabandhu Sheikh Mujib Medical University, Dhaka from January 2012 to December 2012. Results: The analyses of written question papers revealed that a vast majority (85.40 ± 7.32%) of the question segments in the question papers of four public universities were ‘Information-based segments’. The mean proportion (±SD) of ‘principle-based segments’ was 14.60 ± 7.32%. Conclusion: The results of present study provide a baseline data on the relative proportion of ‘principle-based questions’ and ‘information-based questions’ in the Neuroanatomy portion of Anatomy question papers of four public universities in Bangladesh. The study also suggests that the issue of principle-oriented approach in the teach-learning and assessment of neuroanatomy is a valid issue to be addressed in the undergraduate curriculum planning and implementation. EWMCJ Vol. 12, No. 1&2, January-July 2024: 88-93

Strategic philanthropy of athletes after transgression: examining the congruence effect of pre-transgression and transgression issues

PurposeThis study comprehensively explains how sports consumers evaluate athletes’ post-transgression philanthropic activities. We specifically focus on the congruence effect between the pre-transgression philanthropic endeavors and the transgression issue, shedding light on the effectiveness of these strategies in reinstating the positive image of athletes and associated entities.Design/methodology/approachThis study employed a rigorous research design, replicating two studies using sports-related (n = 409) and non-sports-related (n = 404) transgression cases. Data were collected by the Amazon Mechanical Turk platform. A series of experimental studies aimed to investigate the congruence mechanism underlying athletes’ post-transgression philanthropic efforts.FindingsWhen the post-transgression philanthropic initiative is related to the transgression, sport consumers are less likely to view it skeptically and are more inclined to positively evaluate the brand attitude and purchase intention, especially when the transgression is unrelated to the pre-transgression philanthropic efforts.Research limitations/implicationsAs is the case with most research, this study has a limitation. This study used a fictitious athlete name to prevent any prior biases or preconceived notions about the athletes and to avoid any unforeseen influences of personal attitudes toward the athlete. However, designing this study around a fictitious athlete may pose construct validity issues because it may not reflect real-life interactions with the athletes. To increase the validity of findings, future research should aim to replicate the current findings using the names of actual athletes.Originality/valueThis unique approach provides valuable insights and equips sports marketers and brand managers with effective strategies to restore the positive image of athletes and associated entities after a transgression has been made public, empowering them to make informed decisions in challenging situations.

Improving the Quality of Individual-Level Web Tracking: Challenges of Existing Approaches and Introduction of a New Content and Long-Tail Sensitive Academic Solution

This article evaluates the quality of data collection in individual-level desktop web tracking used in the social sciences and shows that the existing approaches face sampling issues, validity issues due to the lack of content-level data and their disregard for the variety of devices and long-tail consumption patterns as well as transparency and privacy issues. To overcome some of these problems, the article introduces a new academic web tracking solution, WebTrack, an open-source tracking tool maintained by a major European research institution, GESIS. The design logic, the interfaces, and the backend requirements for WebTrack are discussed, followed by a detailed examination of the strengths and weaknesses of the tool. Finally, using data from 1,185 participants, the article empirically illustrates how an improvement in data collection through WebTrack leads to innovative shifts in the use of tracking data. As WebTrack allows for collecting the content people are exposed to beyond the classical news platforms, it can greatly improve the detection of politics-related information consumption in tracking data through automated content analysis compared to traditional approaches that rely on the source-level analysis.

Vitality structures in ‘addictive’ game design

For decades, research on technology use and mental health has been based on the assumption that identifiable structures of ‘design’ are psychologically relevant for their users. This has been central especially for the nosological emergence of ‘behavioural addictions’, which currently include two formal diagnoses on technology use: problems related to playing gambling games (gambling disorder) and videogames (gaming disorder). Alas, the research on identifying ‘addictive’ design structures has suffered from major construct validity issues. To make progress over those issues, I draw from the history of ‘vitality affects’ in psychiatry and introduce vitality structures as a design-phenomenological framework that can help researchers conceptualise psychologically relevant ‘bonds’ between entities of game design and corresponding player phenomenology. Vitality structures are not natural kinds to be discovered but pragmatic constructs to be used—and useful as long as they communicate what is both identifiable and empirically prevalent. As a demonstration of practice, I propose working conceptualisations of three vitality structures, which surface in videogames played by gaming treatment-seekers. Systematic research programs for identifying relevant vitality structures can lead to construct-valid and replicable design effects.

Fuzzy Linear Temporal Logic with Quality Constraints

As an extension of quantitative temporal logic, uncertain temporal logic essentially describes the temporal behavior of uncertain and incomplete systems, thus better solving search and decision-making problems in such systems. Fuzzy linear temporal logic (FLTL) is a focal point in uncertain temporal logic research. However, there are evident shortcomings in the current research outcomes. First, in previous FLTL studies, the practice of obtaining path reachability and formula satisfaction values independently and subsequently selecting the smaller of the two as the satisfaction value metric led to information loss. Furthermore, this simplistic information fusion approach fails to reflect the varying importance of these two types of information to the requirements. Second, computing path reachability and temporal logic formula satisfaction values separately may result in a mismatch between the two pieces of information with respect to the same path segment. Thus, the primary challenge lies in accurately integrating the satisfaction values of temporal logic formulas with the path reachability of the segments that yields these satisfaction values, utilizing various reasonable information synthesis methods, to ensure synchronization between path reachability and formula satisfaction values without incurring information loss. Additionally, it is crucial to reflect the different preference requirements for these two types of information. Moreover, the temporal logic formula characterizes system properties, with its sub-formulas delineating distinct sub-properties. Consequently, considering the varying importance preferences of sub-formulas is also significant. To address these deficiencies, we introduced quality constraint operators into FLTL, resulting in quality-constrained fuzzy linear temporal logic (QFLTL). This incorporation enables the synchronization and comprehensive fusion of path-reachability information and formula satisfaction values within the final semantic metric, thereby resolving the issues related to information synchronization and loss. Furthermore, it can accommodate the differing preference requirements between the two types of information and sub-properties during the information synthesis process. We defined the syntax and semantics of QFLTL and examined its expressive power and properties. Notably, we investigated the decidability of logical decision problems in QFLTL, encompassing validity, satisfiability, and model-checking issues. We proposed corresponding solution algorithms and analyzed their complexities.

Construct Clarity in Physician-Hospital Alignment: The Need for Precision in Definition, Measurement, and Management Comment on "Alignment in the Hospital-Physician Relationship: A Qualitative Multiple Case Study of Medical Specialist Enterprises in the Netherlands".

The study by Ubels and van Raaij highlights the importance of alignment in hospital-physician relationships and the challenges in understanding, measuring, and managing it. Despite extensive research on alignment, drawing precise conclusions about its nature, drivers, and outcomes is difficult due to construct clarity and construct validity issues. This commentary focuses on clarifying these issues and the problems they create for hospitals attempting to manage alignment, as well as for scientific inquiry in this area. These issues involve the need to specify more clearly the essential nature of alignment and how it is distinct from other constructs such as engagement. It also involves demarcating alignment from the structures, arrangements, or processes intended to foster it, as well as from its outcomes. Improved precision in these areas will enable the development of more reliable and valid measures, thereby supporting hypothesis testing, theory building, and the identification of best practices.

Improving the DSM-5 approach to cognitive impairment: Developmental prosopagnosia reveals the need for tailored diagnoses.

The Diagnostic Statistical Manual of Mental Disorders (DSM-5) recommends diagnosing neurocognitive disorders (i.e., cognitive impairment) when a patient scores beyond -1 SD below neurotypical norms on two tests. I review how this approach will fail due to cognitive tests' power limitations, validity issues, imperfect reliabilities, and biases, before summarizing their resulting negative consequences. As a proof of concept, I use developmental prosopagnosia, a condition characterized by difficulties recognizing faces, to show the DSM-5 only diagnoses 62-70% (n1 = 61, n2 = 165) versus 100% (n1 = 61) through symptoms alone. Pooling the DSM-5 missed cases confirmed the presence of group-level impairments on objective tests, which were further evidenced through meta-analyses, thus validating their highly atypical symptoms. These findings support a paradigm shift towards bespoke diagnostic approaches for distinct cognitive impairments, including a symptom-based method when validated effective. I reject dogmatic adherence to the DSM-5 approach to neurocognitive disorders, and underscore the importance of a data driven, transdiagnostic approach to understanding patients' subjective cognitive impairments. This will ultimately benefit patients, their families, clinicians, and scientific progress.

METHODS AND TOOLS FOR ASSESSING STUDENT COMPETENCIES

The article deals with the detailed information about Methods and tools for assessing student competencies. The accurate assessment of student competencies is crucial in modern education, as it allows educators to measure not only knowledge retention but also skills, attitudes, and critical thinking abilities. This paper explores various methods and tools used for assessing student competencies, including traditional assessments, performance-based assessments, and technology-enhanced assessments. Traditional methods, such as multiple-choice tests and written exams, remain popular but are often limited in their ability to evaluate higher-order thinking skills. In contrast, performance-based assessments—such as projects, presentations, and portfolios—offer a more comprehensive evaluation by focusing on the application of knowledge in real-world contexts. Technology-enhanced tools, such as digital simulations, e-portfolios, and automated feedback systems, provide innovative ways to measure competencies while increasing engagement and personalization. The paper also discusses the importance of aligning assessment tools with learning objectives and provides insights into the challenges associated with each method, including issues of reliability, validity, and scalability. By combining multiple assessment methods, educators can obtain a holistic view of student competencies, enabling targeted support and improved learning outcomes.

Fuzzy Petri Nets for Traffic Node Reliability.

Self-driving cars are one of the main areas of research today, but it has to be acknowledged that the information from the sensors (the perceptron) is a huge amount of data, which is now unmanageable even when projected onto a single traffic junction. In the case of self-driving, the nodes have to be sequenced and organized according to the planned route. A self-driving car in Hungary would have to be able to interpret more than 70,000 traffic junctions to be able to drive all over the country. Besides the huge amount of data, another problem is the issue of validation and verification. For self-driving cars, this implies a level of complexity using traditional methods that calls into question the economics of the already existing system. Fuzzy Petri nets provide an alternative solution to both problems. They allow us to obtain a model that accurately describes the reliability of a node through its dynamics, which is essential in perception since the more reliable a node is, the smaller the deep learning mesh required. In this paper, we outline the analysis of a traffic node's safety using Petri nets and fuzzy analysis to gain information on the reliability of the node, which is essential for the modeling of self-driving cars, due to the deep learning model of perception. The reliability of the dynamics of the node is determined by using the modified fuzzy Petri net procedure. The need for a fuzzy extension of the Petri net was developed by knowledge of real traffic databases.

Fetus descent simulation with the active uterine contraction during the vaginal delivery: MRI-based evaluation and uncertainty quantification

Finite element models ranging from single to multiscale models have been widely used to gain valuable insights into the physiological delivery process and associated complication scenarios. However, the fetus descent simulation with the active uterine contraction is still challenging for validation and uncertainty quantification issues. The present study performed a fetus descent simulation using the active uterine contraction. Then, simulation outcomes were evaluated using theoretical and in vivo MRI childbirth data. Moreover, parameter uncertainty and propagation were also performed. A maternal pelvis model was developed. The active uterine contraction was modeled using a transversely isotropic Mooney–Rivlin material. Displacement trajectories were compared between simulation, theoretical and in vivo MRI childbirth data. Monte Carlo (M.C) and Polynomial Chaos Expansion (PCE) methods were applied to quantify uncertain parameters and their propagations. Obtained results showed that fetal descent behavior is consistent with the MRI-based observation as well as the theoretical trajectory (curve of Carus). The head downward vertical displacement ranges from 0 to approximately 47 mm. A reduction of 50% in uterine size was observed during the simulation. Three high-sensitive parameters ( C 1 , C 2 , Ca 0 ) were also identified. Our study suggested that the use of the active uterine contraction is essential for simulating vaginal delivery but the global parameter sensitivity, parameter uncertainty, and outcome evaluation should be carefully performed. As a perspective, the developed approach could be extrapolated for patient-specific modeling and associated delivery complication simulations to identify risks and potential therapeutic solutions.

Landslide susceptibility assessment using novel hybridized methods based on the support vector regression

Landslide susceptibility assessment is a complex task due to the multitude of causative factors, spatial variability, data availability, modeling uncertainty, and validation issues. This study addresses these challenges by proposing two predictive models that hybridize support vector regression (SVR) with two evolutionary algorithms: grey wolf optimizer (GWO) and cuckoo search algorithm (CSA). These models were developed using an extensive geospatial database from northern Iran. Over the training phase, the basic predictive model, developed using SVR, was enhanced by incorporating the GWO and CSA algorithms, resulting in the development of two hybridized models: SVR-GWO and SVR-CSA. Over the validation phase, the performance and effectiveness of each hybridized model were compared to the standalone SVR using several metrics. Compared to the standalone SVR model, the hybridized models demonstrated significant improvement in generalization and predictive abilities. Despite non-significant difference between the performances of the SVR-GWO and SVR-CSA models, the SVR-GWO model demonstrated superior performance. This could be attributed to the GWO's capabilities, which included generating a variety of solutions, demonstrating robustness against noise and outliers, achieving faster convergence speed, and effectively interacting with SVR. This study highlighted that utilizing intelligence hybridized models can significantly enhance the balance between accuracy, robustness, and objectives compared to single models. This finding holds significant implications for ecological engineers tasked with designing and implementing solutions to mitigate the impact of shallow landslides on the environment and human communities. The predictive models developed in this study serve as valuable tools for these engineers, enabling them to identify high-risk areas and implement preventative measures. This interdisciplinary approach, which combines machine learning, optimization algorithms, and ecological engineering, highlights the potential for pioneering solutions in tackling complex environmental challenges, thereby standing as a testament to the power of innovation in driving progress in landslide susceptibility assessment.

PASS Theory of Intelligence and Its Measurement Using the Cognitive Assessment System, 2nd Edition.

The goal of this paper was to describe the context within which the PASS theory of intelligence was conceived and the reasons why this theory was used to guide the construction of the Cognitive Assessment System and the several versions of the Cognitive Assessment System, 2nd Edition. We also discuss validity issues such as equitable assessment of intelligence, using PASS scores to examine a pattern of strengths and weaknesses related to academic variability and diagnosis, and the utility of PASS scores for intervention. We provide summaries of the research that informs our suggestions that intelligence testing should be theory-based, not constrained by the seminal work of test developers in the early 1900s, and neurocognitive processes should be measured based on brain function.

Can items derived from international literature be used in national quality of life instruments? A qualitative study conceptualising the EQ-HWB in China

IntroductionThe EQ Health and Wellbeing (EQ-HWB) is a new questionnaire for measuring quality of life (QoL) from a broad perspective. The items of the EQ-HWB were derived based on a ‘qualitative review’ of literature, which reported primarily on Western studies. It can be argued that the QoL is a cultural-related concept and therefore people from China have a different understanding of the QoL. This study aimed to explore whether Chinese citizens could understand the EQ-HWB’s candidate items and what they thought of those items. In doing so, we wanted to examine the face validity of the candidate items and explore if further cultural adaptation is necessary.MethodsThis research was part of the E-QALY project, in which 36 candidate items were selected for the EQ-HWB from a 97-item pool. In China, three interviewers investigated the face validity of these EQ-HWB candidate items in semi-structured qualitative face-to-face interviews. Respondents were invited to report ‘problems’ with regard to the interpretation of the items and these problems were grouped into themes. We explored to what extent those themes related to specific cultural aspects in China. We also classified the rates of reported problems for each item into three groups: 1) less than 20%, 2) from 20–50%, and 3) over 50%.ResultsFor 17 items the rate of reported problems was less than 20%, 15 items fell into the second group (with 20 − 50%) and for 4 items the rate of problems reported was more than 50%. The thematic analysis revealed eight themes: ambiguous problems in the interpretation of 16 items; difficult to understand (11); contained a complex negative expression (10); examples used seemed inappropriate (7); misleading connotation in Chinese (2); long and complex (2); complex response options (1); and use of non-colloquial language (1).DiscussionOur research shows that EQ-HWB candidate items require careful examination to make them more comprehensible. Most of the reported problem themes were generic problems related to the items, and only a few face validity issues appeared to relate to specific cultural aspects in China, even though most of the items were based on Western studies. Our findings are reassuring for the instrument’s international application, especially in China.

A review of main issues regarding validity, reliability, generalizability, transferability, and applicability of log-based measurement of computer-assisted learning

Log analysis has become a common methodology in the research of computer-assisted learning. Using this method, variables to measure various aspects of learning are computed from the data that is stored in computer-assisted learning environments’ log files; these files document fine-grained data on student interaction with the learning system, and are updated automatically, continuously, and unobtrusively. However, besides challenges that any empirical investigation faces, log-based studies face some other, unique challenges. Despite their methodological importance, these distinctive challenges have not yet discussed in a comprehensive manner. In this review paper, we critically examine issues of validity, reliability, generalizability and transferability, and applicability of log-based analysis. We do so by covering relevant theoretical aspects, and demonstrating them via past research. We conclude with practical recommendations for researchers in the fields of Learning Analytics and Educational Data Mining.

Toward a functional classification for autism in adulthood.

Autism spectrum disorder (ASD) is a heterogeneous condition that affects development and functioning from infancy through adulthood. Efforts to parse the heterogeneity of the autism spectrum through subgroups such as Asperger's and Profound Autism have been controversial, and have consistently struggled with issues of reliability, validity, and interpretability. Nonetheless, methods for successfully identifying clinically meaningful subgroups within autism are needed to ensure that research, interventions, and services address the range of needs experienced by autistic individuals. The purpose of this study was to generate and test whether a simple set of questions, organized in a flowchart, could be used in clinical practice and research to differentiate meaningful subgroups based on individuals' level of functioning. Once generated, subgroups could also be compared to the recently proposed administrative category of Profound Autism and to groupings based on standardized adaptive measures. Ninety-seven adults with autism or related neurodevelopmental disorders participating in a longstanding longitudinal study, or their caregivers if they could not answer for themselves, completed phone interviews when the participants were ~30 years old. Information from these phone interviews was used to generate vignettes summarizing characteristics and aspects of the daily lives of each participant (e.g., language level, vocational activities, and social relationships). Three expert clinicians then used these vignettes to classify each participant based on their level of support needs. Meaningfully distinct subgroups within the sample were identified which could be reliably distinguished from one another. Implications of such categorizations and future directions are discussed.