Validity Issues Research Articles

Fifty years for Plog’s psychographic theory: lessons of replication and validity for tourist behaviour research

ABSTRACT Plog’s psychographic theory was developed to analyse tourist behaviour. The present study examines this theory as Plog’s psychographic theory reaches its 50th anniversary. Replication is one of the most acceptable ways to assess scientific quality and reliability. Using categories of replications, the study systematically screened the citations of Plog (1974, 2001, 2002) from the Scopus database (n = 1185) of referred academic works and analysed its 34 replications (1991–2023). The findings of this consecutive replication analysis showed 23.5% support for Plog’s theory alongside various validity and reliability issues in replicating papers. The segmentation of people was found to be relatively more replicable. Based on these findings, four lessons for tourist behaviour scholars were set: to adopt updated theories and perspectives, maintain and report reliability measures, conduct replication studies, and borrow data but not models from practitioners.

Governance of Generative AI

Abstract The rapid and widespread diffusion of generative artificial intelligence (AI) has unlocked new capabilities and changed how content and services are created, shared, and consumed. This special issue builds on the 2021 Policy and Society special issue on the governance of AI by focusing on the legal, organizational, political, regulatory, and social challenges of governing generative AI. This introductory article lays the foundation for understanding generative AI and underscores its key risks, including hallucination, jailbreaking, data training and validation issues, sensitive information leakage, opacity, control challenges, and design and implementation risks. It then examines the governance challenges of generative AI, such as data governance, intellectual property concerns, bias amplification, privacy violations, misinformation, fraud, societal impacts, power imbalances, limited public engagement, public sector challenges, and the need for international cooperation. The article then highlights a comprehensive framework to govern generative AI, emphasizing the need for adaptive, participatory, and proactive approaches. The articles in this special issue stress the urgency of developing innovative and inclusive approaches to ensure that generative AI development is aligned with societal values. They explore the need for adaptation of data governance and intellectual property laws, propose a complexity-based approach for responsible governance, analyze how the dominance of Big Tech is exacerbated by generative AI developments and how this affects policy processes, highlight the shortcomings of technocratic governance and the need for broader stakeholder participation, propose new regulatory frameworks informed by AI safety research and learning from other industries, and highlight the societal impacts of generative AI.

The State of ESG Disclosure: An Exploration of Stakeholders and Sustainability Materiality

The hospitality and tourism industry is known to be resource dependent and wasteful (e.g., water, energy, food, etc.). Given growing interest in the environmental, social, and governance (ESG) framework, hospitality and tourism companies’ civic duties are highlighted, and they are required to show their due diligence. From a risk management perspective, accounting, financial, and consulting institutions have introduced ESG disclosure and reporting systems, but these tools vary enormously depending on the industry, continents, and scope of the business. Retrieving the top 50 hospitality and tourism businesses’ ESG reports (by market capital), this study investigates the status of ESG disclosure and compliance. The findings of this study make important theoretical and practical contributions such that this study provides an important theoretical avenue for ESG conceptualization and addresses measurement validity and reliability issues. Practically, industry practitioners can detect the inequity in ESG reporting systems and identify lacking areas of ESG input.

Autoethnography in medical education: Ethical and legal considerations

AbstractFor medical educators, autoethnography as a research methodology provides a means of active engagement in reflective practice, ranging from teaching and educational innovation to interactions with patients and colleagues. In this way, they may benefit from the systematic reflexivity required, improve their interactions with the people around them, as well as demonstrate ongoing personal development and improve their skillset as an educator. On a wider level, autoethnography has the potential to study a culture or a phenomenon as it is directly and personally experienced by the author, thereby providing unique insight into medical education theories and/or the patient‐physician relationship. As autoethnography often examines sensitive issues, hidden aspects of our lives or untold stories, it is clear that autoethnography poses several ethical and moral challenges which must be considered when the relevant research is planned. In this essay, I will discuss some of the ethical considerations of autoethnography which are relevant in the medical education context, and highlight potential legal considerations of these published personal reflections. To do this, I will discuss issues of validity with autoethnography, ownership of the stories, the tension between authenticity and anonymity, ethics of representation of study participants and relevant legal issues which may impact the use of autoethnography in the healthcare setting. Context and implicationsRationale for this study: Autoethnography as a research methodology provides a means of active engagement in reflective practice, and should lend itself to research in medical education and healthcare research.Why the new findings matter: As this is a relatively new methodology in the context of medical education, there are several challenges to adoption and researchers should be aware of some of the ethical and legal considerations.Practical implications: This article aims to highlight some of the issues for researchers who are considering autoethnography as a research methodology in the healthcare setting.

Quantitative Research in Education: Philosophy, Uses and Limitations

Conventionally, the quantitative research approach is believed to be used in natural science to experiment with phenomena and test theories and hypothesis. However, it has been practiced by researchers in social science and education at present. The study aims to critically analyze the philosophical stances of quantitative research and to explore the uses and limitations of quantitative research methods in education. I used document analysis as a research method where I collected previously published books, journal articles, and book chapters in the related field, read them, explored the required information from the materials, analyzed them critically, and made the arguments in different themes. The study revealed that quantitative research is based on the positivist paradigm which believes in objectivity, and experimental reality. It further showed that quantitative research is significant in the field of education to test the methods, and teaching-learning theories, to see the cause-and-effect relationship, and to see the causal relationship between the variables. Moreover, it shows that quantitative research is significant to explore the results that can be generalized in a large population. Additionally, the study explored that quantitative research has several issues such as negligence on educational complexities, sampling bias, issues of validity, and generalizability, and ethical issues related to consent, confidentiality, anonymity, sampling, data collection, interpretation, and reporting. It suggests that researchers in education should incorporate the qualitative method and mixed-method research to address diversity, emotion, experience, and multiple realities.

Quality Control and Validation Issues in LC-MS-Based Metabolomics.

Metabolic profiling performed using untargeted metabolomics of different, complex biological samples aims to apply agnostic/holistic, hypothesis-free, analysis of the small molecules that are present in the analyzed sample. This approach has been the center of major investments and dedicated efforts from the research community for many years. However, limitations and challenges remain, particularly with regard to the validation and the quality of the obtained results. This has led to increasing community engagement, with the formation of think tanks, the establishment of working groups, and the many seminars on quality control (QC) in metabolomics. Here we describe a quality control (QC) protocol used to monitor LC-MS-based metabolomics analysis. A key target is the monitoring of analytical precision. This methodology is described for the analysis of urine but can be applied to different biological matrices, such as various biofluids, cell, and tissue extracts.

Apples and oranges: Conceptual review as task analysis method.

Conceptual review is a method to address issues of task comparability and task validity in cognitive neuroscience. Meta-analyses within cognitive neuroscience (CNS) as well as integration of neuroscientific findings with findings from adjacent disciplines both involve gathering studies that have purportedly investigated the same mental concept. After all, it is no use comparing apples and oranges. Tasks, and in particular the experimental contrasts implemented through tasks, determine whether studies are in fact comparable. Yet studies tend to be grouped together or kept apart based on the mental label researchers have applied and unfortunately, labels are an unreliable proxy for experimental contrasts. Different contrasts may receive the same label: 'working memory' studies rely on a variety of contrasts, derived from a variety of tasks. Vice versa, the same contrast may receive different labels: 'task switching' and 'working memory' studies can be exactly the same in terms of their experimental contrast. Label use thus obscures comparability problems. What is more, even when experimental contrasts are comparable, they may be invalid operationalizations of the mental label attached to them. In this paper, I introduce conceptual review as a method for task analysis. It can stand on its own or be combined with a cognitive ontology. Conceptual review applies philosophical strategies for analysing concepts to methodological choices in CNS studies, to uncover their conceptual implications. Conceptual review thus sheds light on the precise concept that was studied and thereby, on the comparability of CNS studies and the validity of tasks.

Prediction of treatment response and outcome of transarterial chemoembolization in patients with hepatocellular carcinoma using artificial intelligence: A systematic review of efficacy.

To perform a systematic literature review of the efficacy of different AI models to predict HCC treatment response to transarterial chemoembolization (TACE), including overall survival (OS) and time to progression (TTP). This systematic review was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines until May 2, 2024. The systematic review included 23 studies with 4,486 HCC patients. The AI algorithm receiver operator characteristic (ROC) area under the curve (AUC) for predicting HCC response to TACE based on mRECIST criteria ranged from 0.55 to 0.97. Radiomics-models outperformed non-radiomics models (AUCs: 0.79, 95%CI: 0.75-0.82 vs. 0.73, 0.61-0.77, respectively). The best ML methods used for the prediction of TACE response for HCC patients were CNN, GB, SVM, and RF with AUCs of 0.88 (0.79-0.97), 0.82 (0.71-0.89), 0.8 (0.60-0.87) and 0.8 (0.55-0.96), respectively. Of all predictive feature models, those combining clinic-radiologic features (ALBI grade, BCLC stage, AFP level, tumor diameter, distribution, and peritumoral arterial enhancement) had higher AUCs compared with models based on clinical characteristics alone (0.79, 0.73-0.89; p=0.04 for CT+clinical, 0.81, 0.75-0.88; p=0.017 for MRI+clinical versus 0.6, 0.55-0.75 in clinical characteristics alone). Integrating clinic-radiologic features enhances AI models' predictive performance for HCC patient response to TACE, with CNN, GB, SVM, and RF methods outperforming others. Key predictive clinic-radiologic features include ALBI grade, BCLC stage, AFP level, tumor diameter, distribution, and peritumoral arterial enhancement. Multi-institutional studies are needed to improve AI model accuracy, address heterogeneity, and resolve validation issues.

THE IMPACT OF SOCIAL MEDIA USE ON DESIGN OFFICES

This study aims to elucidate the effects of Instagram usage on the design processes, client relations, project management, and corporate identity of Interior Design and Architecture offices. It seeks to provide a comprehensive perspective on how social media platforms induce transformations within design offices and the extent to which these changes are reflected in sectoral dynamics. In this context, the "cross-sectional research model," one of the general survey models, was chosen to test the hypotheses regarding the "operations" and "developments" of Interior Design and Architecture firms through Instagram usage. The study's population consists of Interior Architecture and Architecture offices operating in Istanbul, with the sample including 21 offices that actively use Instagram and have at least 30,000 followers. To efficiently gather data from the larger population, the "convenience sampling method," a type of random sampling, was employed. A 23-question survey form, developed by the researchers, was used in the study. The findings indicate that Instagram positively impacts the operations and development of design offices, supporting the study's hypotheses. However, the limited scope of the sample weakens the generalizability of the findings and negatively affects the external validity of the research. To address the issue of low external validity and ensure that the data obtained represents a broader audience, it is recommended that future research includes other popular social media platforms such as Facebook, Twitter, and Pinterest.

Does Public Opinion on Foreign Policy Affect Elite Preferences? Evidence from the 2022 US Sanctions against Russia

Abstract Does public opinion on international affairs affect elites’ policy preferences? Most research assumes that it does, but this key assumption is difficult to test empirically given limited research access to elite decision-makers. We examine elite responsiveness to public opinion on sanctioning Russia during the 2022 Russian invasion of Ukraine. We fielded a preregistered experiment within the 2022 TRIP survey of US foreign policy practitioners, offering a rare opportunity for a fairly large elite survey experiment ($N = 253$). We used contemporary public polling highly supportive of increasing sanctions as an information treatment. Our research design, involving a salient issue and real-world treatment, substantially expands on previous work. Exposure to the treatment raises elite support for increasing sanctions from 68.0 percent to 76.3 percent (+8.3 pp.). While meaningful, this effect is smaller than those identified elsewhere. We argue that this difference is driven by pretreatment dynamics related to issue salience and ceiling effects and is therefore all the more notable. We provide evidence for substantial treatment effect heterogeneity depending on subject-matter expertise, degree of involvement in political decision-making, and gender, but not party identification. While our results support previous research, they highlight issues of external validity and the context-dependence of elite responsiveness.

Conceptualizing self-reported adverse childhood experiences: From epidemiologic exposure to psychometric latent variable.

In this paper, we apply a measurement science perspective to explore both the epidemiologic and psychometric frameworks for the conceptualization, operationalization and assessment of self-reported adverse childhood experiences (srACEs). The epidemiologic paradigm suggests that srACEs are 'exposures', while the psychometric paradigm views responses on srACEs instrumentation as 'indicators'. It is the central premise of this paper that srACEs cannot be both exposures and indicators of scales. We review issues of reliability and validity from both perspectives, examine the degree of agreement between objective and subjective reports of childhood maltreatment and the implications of poor agreement, and conclude that the for the assessment of ACEs via self-report, the epidemiologic paradigm is not a good fit. We then review a number of reflexive and formative latent variable models that might be usefully fit to srACEs data for purposes of modeling structural properties of assessments, and/or to model ACE-health relationships. We highlight the mismatch and limitations of the reflexive measurement model for srACEs and conclude by endorsing either a formative latent variable model or application of latent class analyses. We emphasize the importance of considering and potentially formally testing competing measurement models and conducting both rationale analysis, conceptualization and hypothesis-testing to assess the fit of these models.

Introduction of English-Speaking Observation Method and Process

The evaluation and monitoring of English speaking skills has gained great prominence in today’s world which has been globalized to such an extent that English is used both at the academic and occupational level. This research paper attempts to shed light on the strategies and the practices of the assessment of spoken English ability in different settings. After a literature review, original research, and case studies, we discuss the available methodologies for the observation, their efficacy, and what can be done, if anything, to enhance them. The results of this study suggest that the best way to assess speaking ability is to use both quantitative and qualitative measuring components. Research literature and practice dealing with the traditional interviews, computer based testing (CBT), e-assessments, and recent developments in the area of peer assessments are considered. The paper discusses the concerns of instructors and evaluators, such as those related to issues of reliability and validity and on the other hand, practicality. Ethnocultural aspects, as well as non verbal aspects of speaking, and even more, enhancing speaking with artificial intelligence are discussed as well. These days, new technologies facilitate solving many problems. However, these findings imply that considering all the human aspects of communication is also very important. The paper concludes with practical recommendations for teachers, researchers, and language testers that relate to improvement of techniques of performance observation and to the enhancement of English language teaching and learning in the changing world.

Revisiting Validity in Research With Multilingual Learners: Moving From a Monolingual to a Multilingual Lens on Research Quality

A critical revisiting of research methods with multilingual learners raises issues of validity that are often unaddressed in current discussions of research quality. Though validity is conceptualized and assessed differently across research traditions, validity broadly describes the production of findings that are credible and trustworthy. The historically monolingual lens of research methodologies has led to conceptions of validity that are inadequate for ensuring that reported findings “fit the data” in research with multilingual learners. In this article, I describe a retrospective inquiry to critically reassess how current paradigms of validity might be reconceptualized in research with multilingual learners. Through a reexamination of research practices from two of my own studies, this article considers three ways that current monolingual paradigms of validity should be expanded in research with multilingual learners to work toward multilingual conceptualizations of research quality. This article then offers guidance to aid other researchers in taking up issues of validity in the design and reporting of research with multilingual learners.

Impact of time window on estimating representativeness error in sea surface salinity data

ABSTRACT Validating satellite data using buoys presents a significant challenge due to the differing spatial and temporal representativeness of these data sources. This discrepancy introduces two critical issues in satellite data validation: the selection of the matchup window during the collocation process and the estimation of representativeness error. While numerous studies have addressed these issues independently, research on their interaction remains limited. Utilizing quadruple Sea Surface Salinity (SSS) matchup datasets, this study investigates the impact of the time window on the estimation of SSS representativeness error between two satellite observations: SMOS and SMAP. Our results reveal a clear dependence of representativeness error on the selected time windows. In the global ocean, the variance of representativeness errors within a small time window is 0.040 psu2, which is substantially greater than that of the entire dataset (0.016 psu2). In contrast, the representativeness errors within larger time windows are systematically lower than those within smaller time windows. This research also finds that this dependency is not influenced by the randomness of the matchup data, climatological Sea Surface Salinity, or geographic locations. Instead, our findings suggest that the impact of the time window on representativeness error may reflect the characteristics of satellite data across different time intervals. Satellites are more likely to capture SSS signals within smaller time windows, resulting in larger representativeness errors when comparing satellite SSS with buoy measurements. Conversely, larger time windows lead to significant mismatches between satellite observations and hinder the ability of satellites to detect SSS signals, which can result in smaller or even reversed representativeness errors.

Development and Preliminary Testing of the Staffordshire Questionnaire for Adolescent Idiopathic Scoliosis (SQ-AIS): Content and Face Validity.

Adolescent Idiopathic Scoliosis (AIS) is a structural spinal deformity with implications for health-related quality of life (HR-QoL). The Scoliosis Research Society-22 revised (SRS-22r) questionnaire is the standard for HR-QoL assessment. However, studies have identified limitations with the SRS-22r, including content and face validity issues, reliability concerns, and language appropriateness. This study aimed to develop and validate a patient-reported questionnaire, the Staffordshire Questionnaire for Adolescent Idiopathic Scoliosis (SQ-AIS), to assess the impact of AIS on HR-QoL. The SQ-AIS comprises six domains: general health, pain, function/activity, self-image/appearance, mental health, and intervention. Individuals with AIS aged 10-19 years and clinicians from a range of countries with expertise in AIS contributed to the testing process. Face validity and clinical applicability were assessed using Likert scales, while content validity was evaluated through a categorical binary variable (yes/no). Involving 8 AIS patients and 43 clinicians, face validity scores demonstrated an acceptable level of understanding (≥ 4/5) for both individuals with AIS and clinicians. Most individuals with AIS (85.71%) and clinicians (80.95%) affirmed that the questionnaire sufficiently covers various aspects of scoliosis, indicating a satisfactory level of content validity. Ratings for applicability to clinical practice indicated an acceptable level of practical relevance (≥ 4/5). The SQ-AIS emerges as a valid and promising tool to overcome existing challenges in AIS-related outcome assessment. Pending further validation studies, the favorable reception from the international community of clinicians suggests its potential as a new benchmark for evaluating AIS impact on HR-QoL and monitoring scoliosis management.

A comprehensive item bank of internal validity issues of relevance to in vitro toxicology studies

A comprehensive item bank of internal validity issues of relevance to in vitro toxicology studies

Principle-oriented versus Information-oriented Approach Towards Neuroanatomy: A Selective Profile of Question Papers of Four Public University of Bangladesh

Most of the students considered neuroanatomy to be a difficult subject that contains a large volume of information. Therefore, many of them have the tendency to memorise the subject rather than understanding. The learning will be permanent if learning and its assessment is structured around generalisable rules or ‘principle’ rather than trying to memorise plenty of apparently disconnected ‘information’. Materials and methods: The Neuroanatomy portion of the Anatomy question papers of the First Professional MBBS Examinations of four public universities were analysed. It was a cross sectional, observational study carried out in the Department of Anatomy, Bangabandhu Sheikh Mujib Medical University, Dhaka from January 2012 to December 2012. Results: The analyses of written question papers revealed that a vast majority (85.40 ± 7.32%) of the question segments in the question papers of four public universities were ‘Information-based segments’. The mean proportion (±SD) of ‘principle-based segments’ was 14.60 ± 7.32%. Conclusion: The results of present study provide a baseline data on the relative proportion of ‘principle-based questions’ and ‘information-based questions’ in the Neuroanatomy portion of Anatomy question papers of four public universities in Bangladesh. The study also suggests that the issue of principle-oriented approach in the teach-learning and assessment of neuroanatomy is a valid issue to be addressed in the undergraduate curriculum planning and implementation. EWMCJ Vol. 12, No. 1&2, January-July 2024: 88-93

Strategic philanthropy of athletes after transgression: examining the congruence effect of pre-transgression and transgression issues

PurposeThis study comprehensively explains how sports consumers evaluate athletes’ post-transgression philanthropic activities. We specifically focus on the congruence effect between the pre-transgression philanthropic endeavors and the transgression issue, shedding light on the effectiveness of these strategies in reinstating the positive image of athletes and associated entities.Design/methodology/approachThis study employed a rigorous research design, replicating two studies using sports-related (n = 409) and non-sports-related (n = 404) transgression cases. Data were collected by the Amazon Mechanical Turk platform. A series of experimental studies aimed to investigate the congruence mechanism underlying athletes’ post-transgression philanthropic efforts.FindingsWhen the post-transgression philanthropic initiative is related to the transgression, sport consumers are less likely to view it skeptically and are more inclined to positively evaluate the brand attitude and purchase intention, especially when the transgression is unrelated to the pre-transgression philanthropic efforts.Research limitations/implicationsAs is the case with most research, this study has a limitation. This study used a fictitious athlete name to prevent any prior biases or preconceived notions about the athletes and to avoid any unforeseen influences of personal attitudes toward the athlete. However, designing this study around a fictitious athlete may pose construct validity issues because it may not reflect real-life interactions with the athletes. To increase the validity of findings, future research should aim to replicate the current findings using the names of actual athletes.Originality/valueThis unique approach provides valuable insights and equips sports marketers and brand managers with effective strategies to restore the positive image of athletes and associated entities after a transgression has been made public, empowering them to make informed decisions in challenging situations.

Improving the Quality of Individual-Level Web Tracking: Challenges of Existing Approaches and Introduction of a New Content and Long-Tail Sensitive Academic Solution

This article evaluates the quality of data collection in individual-level desktop web tracking used in the social sciences and shows that the existing approaches face sampling issues, validity issues due to the lack of content-level data and their disregard for the variety of devices and long-tail consumption patterns as well as transparency and privacy issues. To overcome some of these problems, the article introduces a new academic web tracking solution, WebTrack, an open-source tracking tool maintained by a major European research institution, GESIS. The design logic, the interfaces, and the backend requirements for WebTrack are discussed, followed by a detailed examination of the strengths and weaknesses of the tool. Finally, using data from 1,185 participants, the article empirically illustrates how an improvement in data collection through WebTrack leads to innovative shifts in the use of tracking data. As WebTrack allows for collecting the content people are exposed to beyond the classical news platforms, it can greatly improve the detection of politics-related information consumption in tracking data through automated content analysis compared to traditional approaches that rely on the source-level analysis.

Vitality structures in ‘addictive’ game design

For decades, research on technology use and mental health has been based on the assumption that identifiable structures of ‘design’ are psychologically relevant for their users. This has been central especially for the nosological emergence of ‘behavioural addictions’, which currently include two formal diagnoses on technology use: problems related to playing gambling games (gambling disorder) and videogames (gaming disorder). Alas, the research on identifying ‘addictive’ design structures has suffered from major construct validity issues. To make progress over those issues, I draw from the history of ‘vitality affects’ in psychiatry and introduce vitality structures as a design-phenomenological framework that can help researchers conceptualise psychologically relevant ‘bonds’ between entities of game design and corresponding player phenomenology. Vitality structures are not natural kinds to be discovered but pragmatic constructs to be used—and useful as long as they communicate what is both identifiable and empirically prevalent. As a demonstration of practice, I propose working conceptualisations of three vitality structures, which surface in videogames played by gaming treatment-seekers. Systematic research programs for identifying relevant vitality structures can lead to construct-valid and replicable design effects.